[1/1] get-developers: explicitly set devs document encoding

Message ID BN6PR2001MB094879FCCDB0C266DE027DFFA0D90@BN6PR2001MB0948.namprd20.prod.outlook.com
State Changes Requested
Headers show
Series
  • [1/1] get-developers: explicitly set devs document encoding
Related show

Commit Message

James Knight March 6, 2018, 2:39 a.m.
Explicitly indicate the file encoding (to UTF-8) for the DEVELOPERS
document. This prevents Unicode decoding errors where some systems
assume an alternative default encoding (ex. 'CP1252').

This corrects the following observed error:

    $ ./utils/get-developers outgoing/*
    Traceback (most recent call last):
      File "./utils/get-developers", line 101, in <module>
        __main__()
      File "./utils/get-developers", line 25, in __main__
        devs = getdeveloperlib.parse_developers()
      File "...\buildroot\utils\getdeveloperlib.py", line 164, in parse_developers
        for line in f:
      File "...\Python36\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 7087: character maps to <undefined>

Signed-off-by: James Knight <james.d.knight@live.com>
---
 utils/getdeveloperlib.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Yann E. MORIN March 31, 2018, 9:14 p.m. | #1
James, All,

On 2018-03-06 02:39 +0000, James Knight spake thusly:
> Explicitly indicate the file encoding (to UTF-8) for the DEVELOPERS
> document. This prevents Unicode decoding errors where some systems
> assume an alternative default encoding (ex. 'CP1252').
> 
> This corrects the following observed error:
> 
>     $ ./utils/get-developers outgoing/*
>     Traceback (most recent call last):
>       File "./utils/get-developers", line 101, in <module>
>         __main__()
>       File "./utils/get-developers", line 25, in __main__
>         devs = getdeveloperlib.parse_developers()
>       File "...\buildroot\utils\getdeveloperlib.py", line 164, in parse_developers
>         for line in f:
>       File "...\Python36\lib\encodings\cp1252.py", line 23, in decode
>         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
>     UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 7087: character maps to <undefined>
> 
> Signed-off-by: James Knight <james.d.knight@live.com>
> ---
>  utils/getdeveloperlib.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/utils/getdeveloperlib.py b/utils/getdeveloperlib.py
> index 2c8d477583..30374d77f8 100644
> --- a/utils/getdeveloperlib.py
> +++ b/utils/getdeveloperlib.py
> @@ -158,7 +158,7 @@ def parse_developers(basepath=None):
>      linen = 0
>      if basepath is None:
>          basepath = os.getcwd()
> -    with open(os.path.join(basepath, "DEVELOPERS"), "r") as f:
> +    with open(os.path.join(basepath, "DEVELOPERS"), "r", encoding="utf8") as f:

This does not work with oython2:

    $ python2
    Python 2.7.14 (default, Sep 23 2017, 22:06:14)
    [GCC 7.2.0] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> with open("DEVELOPPERS", "r", encoding="utf8") as f:
    ...     lines= f.readlines()
    ...
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: 'encoding' is an invalid keyword argument for this function

Recentyl, we had to do a similar encoding/decoding issue:

    5563a1c6a4 support/check-uniq-files: support weird locales and filenames

Do you think you could have a look at it and do a similar tweak to
get-developper instead, please?

Regards,
Yann E. MORIN.

>          files = []
>          name = None
>          for line in f:
> -- 
> 2.16.2.windows.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Patch

diff --git a/utils/getdeveloperlib.py b/utils/getdeveloperlib.py
index 2c8d477583..30374d77f8 100644
--- a/utils/getdeveloperlib.py
+++ b/utils/getdeveloperlib.py
@@ -158,7 +158,7 @@  def parse_developers(basepath=None):
     linen = 0
     if basepath is None:
         basepath = os.getcwd()
-    with open(os.path.join(basepath, "DEVELOPERS"), "r") as f:
+    with open(os.path.join(basepath, "DEVELOPERS"), "r", encoding="utf8") as f:
         files = []
         name = None
         for line in f: