[buildroot-test] scripts/autobuild-run: make it Python 3.x compatible

With Python 3.7, the autobuild-run did not work due to the following
issues:

 - The urlparse module no longer exists, it's not urllib.parse

 - 0022 is no longer recognized as an octal value, we must use 0o022,
   which also works in Python 2.x

 - reading the CSV file with the list of branches through the CSV
   parser failed due to the lack of decoding, as urlopen_closing()
   returns a stream of bytes and not strings. So we need to call
   decode_bytes() on each element of the CSV array. Since the CSV file
   is typically 3 or 4 lines long, we don't really need to optimize
   things.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 scripts/autobuild-run | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

Hi Thomas,

On Tue, Dec 03, 2019 at 05:54:57PM +0100, Thomas Petazzoni wrote:
> With Python 3.7, the autobuild-run did not work due to the following
> issues:
> 
>  - The urlparse module no longer exists, it's not urllib.parse

s/not/now/

> 
>  - 0022 is no longer recognized as an octal value, we must use 0o022,
>    which also works in Python 2.x
> 
>  - reading the CSV file with the list of branches through the CSV
>    parser failed due to the lack of decoding, as urlopen_closing()
>    returns a stream of bytes and not strings. So we need to call
>    decode_bytes() on each element of the CSV array. Since the CSV file
>    is typically 3 or 4 lines long, we don't really need to optimize
>    things.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  scripts/autobuild-run | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/autobuild-run b/scripts/autobuild-run
> index 5921edd..e475ea8 100755
> --- a/scripts/autobuild-run
> +++ b/scripts/autobuild-run
> @@ -145,14 +145,15 @@ from distutils.version import StrictVersion
>  import platform
>  from threading import Thread, Event
>  import datetime
> -import urlparse
>  
>  if sys.hexversion >= 0x3000000:
>      import configparser
>      import urllib.request as _urllib
> +    import urllib.parse as urlparse

Would that work with any Python 3.x?

>  else:
>      import ConfigParser as configparser
>      import urllib2 as _urllib
> +    import urlparse
>  
>  urlopen = _urllib.urlopen
>  urlopen_closing = lambda uri: contextlib.closing(urlopen(uri))
> @@ -307,8 +308,10 @@ class Builder:
>          list. This way, branches with a higher weight are more likely to
>          be selected.
>          """
> +        csv_branches = []
>          with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r:
> -            csv_branches = r.readlines()
> +            for l in r.readlines():
> +                csv_branches.append(decode_bytes(l))
>          branches = []
>          for branch in csv.reader(csv_branches):
>              branches += [branch[0]] * int(branch[1])
> @@ -830,7 +833,7 @@ def main():
>      # Enforce the sanest umask here, to avoid buildroot doing it on its
>      # own and causing a double-make call, thus adding extraneous lines
>      # in case of failures.
> -    os.umask(0022)
> +    os.umask(0o022)
>  
>      def sigterm_handler(signum, frame):
>          """Kill all children"""

Hello,

On Tue, 3 Dec 2019 19:11:37 +0200
Baruch Siach <baruch@tkos.co.il> wrote:

> >  - The urlparse module no longer exists, it's not urllib.parse  
> 
> s/not/now/

Indeed.

> >  if sys.hexversion >= 0x3000000:
> >      import configparser
> >      import urllib.request as _urllib
> > +    import urllib.parse as urlparse  
> 
> Would that work with any Python 3.x?

https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
has some comments in the documentation of urllib.parse.urlparse() that
say:

"""
Changed in version 3.2: Added IPv6 URL parsing capabilities.
"""

So it means that urllib.parse.urlparse() was there at least before
Python 3.2, which means it was most likely there since Python 3.x.

Best regards,

Thomas

On 03/12/2019 21:00, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 3 Dec 2019 19:11:37 +0200
> Baruch Siach <baruch@tkos.co.il> wrote:
> 
>>>  - The urlparse module no longer exists, it's not urllib.parse  
>>
>> s/not/now/
> 
> Indeed.
> 
>>>  if sys.hexversion >= 0x3000000:
>>>      import configparser
>>>      import urllib.request as _urllib
>>> +    import urllib.parse as urlparse  
>>
>> Would that work with any Python 3.x?
> 
> https://docs.python.org/3/library/urllib.parse.html#module-urllib.parse
> has some comments in the documentation of urllib.parse.urlparse() that
> say:
> 
> """
> Changed in version 3.2: Added IPv6 URL parsing capabilities.
> """
> 
> So it means that urllib.parse.urlparse() was there at least before
> Python 3.2, which means it was most likely there since Python 3.x.

 And anyway, I think we generally should assume Python >= 3.4. Debian
oldoldstable (jessie) has 3.4 already. Trying to support older versions will
make our life unnecessarily difficult.

 Regards,
 Arnout

On Tue,  3 Dec 2019 17:54:57 +0100
Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> With Python 3.7, the autobuild-run did not work due to the following
> issues:
> 
>  - The urlparse module no longer exists, it's not urllib.parse
> 
>  - 0022 is no longer recognized as an octal value, we must use 0o022,
>    which also works in Python 2.x
> 
>  - reading the CSV file with the list of branches through the CSV
>    parser failed due to the lack of decoding, as urlopen_closing()
>    returns a stream of bytes and not strings. So we need to call
>    decode_bytes() on each element of the CSV array. Since the CSV file
>    is typically 3 or 4 lines long, we don't really need to optimize
>    things.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  scripts/autobuild-run | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)

Applied to buildroot-test, thanks.

Thomas