[meta-swupdate,RFC] Fix use of config fragments

commit 009a36128ffd4553f47f75c114df4a3662fdf536
Author: Alexander Vickberg <wickbergster@gmail.com>
Date:   Mon Apr 27 10:49:18 2020 +0200

    Fix use of config fragments
    
    This is a Python port of the regular expressions used by
    merge_config.sh to parse config fragments. The assembled config is
    stored temporary in memory.
    
    Signed-off-by: Alexander Vickberg <wickbergster@gmail.com>

+    elif 'CONFIG_SSL_IMPL_OPENSSL=y\n' in features or 
'CONFIG_ENCRYPTED_IMAGES=y\n' in features or 'CONFIG_MONGOOSESSL=y\n' in 
features or 'CONFIG_HASH_VERIFY=y\n' in features or 
'CONFIG_SURICATTA_SSL=y\n' in features:
         depends = d.getVar('DEPENDS', False)
         d.setVar('DEPENDS', depends + ' openssl')

Hi Alexander,

On 27.04.20 10:51, Alexander Vickberg wrote:
> commit 009a36128ffd4553f47f75c114df4a3662fdf536
> Author: Alexander Vickberg <wickbergster@gmail.com>
> Date:   Mon Apr 27 10:49:18 2020 +0200
> 
>     Fix use of config fragments
>     
>     This is a Python port of the regular expressions used by
>     merge_config.sh to parse config fragments. The assembled config is
>     stored temporary in memory.
>     
>     Signed-off-by: Alexander Vickberg <wickbergster@gmail.com>
> 
> diff --git a/recipes-support/swupdate/swupdate.inc
> b/recipes-support/swupdate/swupdate.inc
> index 21de272..941c89a 100644
> --- a/recipes-support/swupdate/swupdate.inc
> +++ b/recipes-support/swupdate/swupdate.inc
> @@ -79,6 +79,8 @@ def find_cfgs(d):
>      return [s for s in src_patches(d, True) if s.endswith('.cfg')]
>  
>  python () {
> +    import re
> +
>      try:
>          defconfig = bb.fetch2.localpath('file://defconfig', d)
>      except bb.fetch2.FetchError:
> @@ -89,14 +91,35 @@ python () {
>      except IOError:
>          return
>  
> -    features = configfile.readlines()
> +    features = configfile.read()

Not sure about this, as later you get back to a list with splitlines().
Which is the intention here ?

>      configfile.close()
>  
> +    for current_fragment in find_cfgs(d):

Who defines this function ? Where is it ?

> +        try:
> +            fragment_fd = open(current_fragment)
> +        except IOError:
> +            continue
> +
> +        fragment = fragment_fd.read()
> +        fragment_fd.close()
> +
> +        fragment_search = re.findall('^(?:# )?(CONFIG_[a-zA-Z0-9_]*)[=
> ].*\n?', fragment, re.MULTILINE)
> +
> +        for feature in fragment_search:
> +            features = re.sub("^(?:# )?" + feature + "[= ].*\n?", "",
> features, flags=re.MULTILINE)
> +
> +        features = features + fragment
> +
> +    features = features.splitlines(True)
> +
>      if 'CONFIG_REMOTE_HANDLER=y\n' in features:
>          depends = d.getVar('DEPENDS', False)
>          d.setVar('DEPENDS', depends + ' zeromq')
>  
> -    if 'CONFIG_ENCRYPTED_IMAGES=y\n' in features or
> 'CONFIG_MONGOOSESSL=y\n' in features or 'CONFIG_HASH_VERIFY=y\n' in
> features or 'CONFIG_SURICATTA_SSL=y\n' in features:
> +    if 'CONFIG_SSL_IMPL_MBEDTLS=y\n' in features:
> +        depends = d.getVar('DEPENDS', False)
> +        d.setVar('DEPENDS', depends + ' mbedtls')
> +    elif 'CONFIG_SSL_IMPL_OPENSSL=y\n' in features or

This has nothing to do with the topic and should be addressed by a
separate patch.

> 'CONFIG_ENCRYPTED_IMAGES=y\n' in features or 'CONFIG_MONGOOSESSL=y\n' in
> features or 'CONFIG_HASH_VERIFY=y\n' in features or
> 'CONFIG_SURICATTA_SSL=y\n' in features:
>          depends = d.getVar('DEPENDS', False)
>          d.setVar('DEPENDS', depends + ' openssl')
>  
> 

Best regards,
Stefano Babic

Hi Stefano,

First email didn't go to the list so sending again. Sorry about that.

Thank you for commenting on the proposed patch.

Den mån 27 apr. 2020 kl 14:41 skrev Stefano Babic <sbabic@denx.de>:

> Hi Alexander,
>
> On 27.04.20 10:51, Alexander Vickberg wrote:
> > commit 009a36128ffd4553f47f75c114df4a3662fdf536
> > Author: Alexander Vickberg <wickbergster@gmail.com>
> > Date:   Mon Apr 27 10:49:18 2020 +0200
> >
> >     Fix use of config fragments
> >
> >     This is a Python port of the regular expressions used by
> >     merge_config.sh to parse config fragments. The assembled config is
> >     stored temporary in memory.
> >
> >     Signed-off-by: Alexander Vickberg <wickbergster@gmail.com>
> >
> > diff --git a/recipes-support/swupdate/swupdate.inc
> > b/recipes-support/swupdate/swupdate.inc
> > index 21de272..941c89a 100644
> > --- a/recipes-support/swupdate/swupdate.inc
> > +++ b/recipes-support/swupdate/swupdate.inc
> > @@ -79,6 +79,8 @@ def find_cfgs(d):
> >      return [s for s in src_patches(d, True) if s.endswith('.cfg')]
> >
> >  python () {
> > +    import re
> > +
> >      try:
> >          defconfig = bb.fetch2.localpath('file://defconfig', d)
> >      except bb.fetch2.FetchError:
> > @@ -89,14 +91,35 @@ python () {
> >      except IOError:
> >          return
> >
> > -    features = configfile.readlines()
> > +    features = configfile.read()
>
> Not sure about this, as later you get back to a list with splitlines().
> Which is the intention here ?
>
>
The regular expressions work on a (multiline) string. All the if checks for
building the DEPENDS variable works on list of lines. I liked that the
checks were made on a list and I think the regular expression code is
cleaner if not needed to be done on a list of lines. Hence the splitting.


> >      configfile.close()
> >
> > +    for current_fragment in find_cfgs(d):
>
> Who defines this function ? Where is it ?
>
>
It's in swupdate.inc. Other packages using Kconfig with support for config
fragments uses something similar. It was added in
commit 5cfdbc7da55b6b0f2ab877823ea7073de076f467.


> > +        try:
> > +            fragment_fd = open(current_fragment)
> > +        except IOError:
> > +            continue
> > +
> > +        fragment = fragment_fd.read()
> > +        fragment_fd.close()
> > +
> > +        fragment_search = re.findall('^(?:# )?(CONFIG_[a-zA-Z0-9_]*)[=
> > ].*\n?', fragment, re.MULTILINE)
> > +
> > +        for feature in fragment_search:
> > +            features = re.sub("^(?:# )?" + feature + "[= ].*\n?", "",
> > features, flags=re.MULTILINE)
> > +
> > +        features = features + fragment
> > +
> > +    features = features.splitlines(True)
> > +
> >      if 'CONFIG_REMOTE_HANDLER=y\n' in features:
> >          depends = d.getVar('DEPENDS', False)
> >          d.setVar('DEPENDS', depends + ' zeromq')
> >
> > -    if 'CONFIG_ENCRYPTED_IMAGES=y\n' in features or
> > 'CONFIG_MONGOOSESSL=y\n' in features or 'CONFIG_HASH_VERIFY=y\n' in
> > features or 'CONFIG_SURICATTA_SSL=y\n' in features:
> > +    if 'CONFIG_SSL_IMPL_MBEDTLS=y\n' in features:
> > +        depends = d.getVar('DEPENDS', False)
> > +        d.setVar('DEPENDS', depends + ' mbedtls')
> > +    elif 'CONFIG_SSL_IMPL_OPENSSL=y\n' in features or
>
> This has nothing to do with the topic and should be addressed by a
> separate patch.
>
>
No problem.


> > 'CONFIG_ENCRYPTED_IMAGES=y\n' in features or 'CONFIG_MONGOOSESSL=y\n' in
> > features or 'CONFIG_HASH_VERIFY=y\n' in features or
> > 'CONFIG_SURICATTA_SSL=y\n' in features:
> >          depends = d.getVar('DEPENDS', False)
> >          d.setVar('DEPENDS', depends + ' openssl')
> >
> >
>
> Best regards,
> Stefano Babic
>
>
> --
> =====================================================================
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de
> =====================================================================
>

Alexander Vickberg

This is a Python port of the regular expressions used by
merge_config.sh to parse config fragments. The assembled config is
stored temporary in memory.

Signed-off-by: Alexander Vickberg <wickbergster@gmail.com>
---
Changes in v2: Don't touch the DEPENDS

 recipes-support/swupdate/swupdate.inc | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/recipes-support/swupdate/swupdate.inc 
b/recipes-support/swupdate/swupdate.inc
index 21de272..d717eea 100644
--- a/recipes-support/swupdate/swupdate.inc
+++ b/recipes-support/swupdate/swupdate.inc
@@ -89,9 +89,27 @@ python () {
     except IOError:
         return
 
-    features = configfile.readlines()
+    features = configfile.read()
     configfile.close()
 
+    for current_fragment in find_cfgs(d):
+        try:
+            fragment_fd = open(current_fragment)
+        except IOError:
+            continue
+
+        fragment = fragment_fd.read()
+        fragment_fd.close()
+
+        fragment_search = re.findall('^(?:# )?(CONFIG_[a-zA-Z0-9_]*)[= 
].*\n?', fragment, re.MULTILINE)
+
+        for feature in fragment_search:
+            features = re.sub("^(?:# )?" + feature + "[= ].*\n?", "", 
features, flags=re.MULTILINE)
+
+        features = features + fragment
+
+    features = features.splitlines(True)
+
     if 'CONFIG_REMOTE_HANDLER=y\n' in features:
         depends = d.getVar('DEPENDS', False)
         d.setVar('DEPENDS', depends + ' zeromq')

This is a Python port of the regular expressions used by
merge_config.sh to parse config fragments. The assembled config is
stored temporary in memory.

Signed-off-by: Alexander Vickberg <wickbergster@gmail.com>
---
Changes in v2: Don't touch the DEPENDS
Changes in v3: Readd import that got missing in v2.

 recipes-support/swupdate/swupdate.inc | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/recipes-support/swupdate/swupdate.inc 
b/recipes-support/swupdate/swupdate.inc
index 21de272..4894bc9 100644
--- a/recipes-support/swupdate/swupdate.inc
+++ b/recipes-support/swupdate/swupdate.inc
@@ -79,6 +79,8 @@ def find_cfgs(d):
     return [s for s in src_patches(d, True) if s.endswith('.cfg')]
 
 python () {
+    import re
+
     try:
         defconfig = bb.fetch2.localpath('file://defconfig', d)
     except bb.fetch2.FetchError:
@@ -89,9 +91,27 @@ python () {
     except IOError:
         return
 
-    features = configfile.readlines()
+    features = configfile.read()
     configfile.close()
 
+    for current_fragment in find_cfgs(d):
+        try:
+            fragment_fd = open(current_fragment)
+        except IOError:
+            continue
+
+        fragment = fragment_fd.read()
+        fragment_fd.close()
+
+        fragment_search = re.findall('^(?:# )?(CONFIG_[a-zA-Z0-9_]*)[= 
].*\n?', fragment, re.MULTILINE)
+
+        for feature in fragment_search:
+            features = re.sub("^(?:# )?" + feature + "[= ].*\n?", "", 
features, flags=re.MULTILINE)
+
+        features = features + fragment
+
+    features = features.splitlines(True)
+
     if 'CONFIG_REMOTE_HANDLER=y\n' in features:
         depends = d.getVar('DEPENDS', False)
         d.setVar('DEPENDS', depends + ' zeromq')

Hi Alexander,

I cannot apply the patches, it looks like that your e-mailer has
malformed them. I get:

patching file recipes-support/swupdate/swupdate.inc
patch: **** malformed patch at line 150: 'CONFIG_MONGOOSESSL=y\n' in
features or 'CONFIG_HASH_VERIFY=y\n' in

Could you repost them and use git send-email to avoid issues ? Thanks.

Best regards,
Stefano Babic

On 28.04.20 11:00, Alexander Vickberg wrote:
> This is a Python port of the regular expressions used by
> merge_config.sh to parse config fragments. The assembled config is
> stored temporary in memory.
> 
> Signed-off-by: Alexander Vickberg <wickbergster@gmail.com>
> ---
> Changes in v2: Don't touch the DEPENDS
> Changes in v3: Readd import that got missing in v2.
> 
>  recipes-support/swupdate/swupdate.inc | 22 +++++++++++++++++++++-
>  1 file changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/recipes-support/swupdate/swupdate.inc
> b/recipes-support/swupdate/swupdate.inc
> index 21de272..4894bc9 100644
> --- a/recipes-support/swupdate/swupdate.inc
> +++ b/recipes-support/swupdate/swupdate.inc
> @@ -79,6 +79,8 @@ def find_cfgs(d):
>      return [s for s in src_patches(d, True) if s.endswith('.cfg')]
>  
>  python () {
> +    import re
> +
>      try:
>          defconfig = bb.fetch2.localpath('file://defconfig', d)
>      except bb.fetch2.FetchError:
> @@ -89,9 +91,27 @@ python () {
>      except IOError:
>          return
>  
> -    features = configfile.readlines()
> +    features = configfile.read()
>      configfile.close()
>  
> +    for current_fragment in find_cfgs(d):
> +        try:
> +            fragment_fd = open(current_fragment)
> +        except IOError:
> +            continue
> +
> +        fragment = fragment_fd.read()
> +        fragment_fd.close()
> +
> +        fragment_search = re.findall('^(?:# )?(CONFIG_[a-zA-Z0-9_]*)[=
> ].*\n?', fragment, re.MULTILINE)
> +
> +        for feature in fragment_search:
> +            features = re.sub("^(?:# )?" + feature + "[= ].*\n?", "",
> features, flags=re.MULTILINE)
> +
> +        features = features + fragment
> +
> +    features = features.splitlines(True)
> +
>      if 'CONFIG_REMOTE_HANDLER=y\n' in features:
>          depends = d.getVar('DEPENDS', False)
>          d.setVar('DEPENDS', depends + ' zeromq')
> -- 
> 2.25.1
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+unsubscribe@googlegroups.com
> <mailto:swupdate+unsubscribe@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/swupdate/d8c55580-0b11-4743-a328-adb73ad2e527%40googlegroups.com
> <https://groups.google.com/d/msgid/swupdate/d8c55580-0b11-4743-a328-adb73ad2e527%40googlegroups.com?utm_medium=email&utm_source=footer>.