scanpypi: rework download_package error handling

Message ID 20180613073710.5715-1-yegorslists@googlemail.com
State Accepted
Commit fb775f4c87c627537efbb56f2663d6079489f018
Headers show
Series
  • scanpypi: rework download_package error handling
Related show

Commit Message

Yegor Yefremov June 13, 2018, 7:37 a.m.
From: Yegor Yefremov <yegorslists@googlemail.com>

Some packages don't provide source archive but only a wheel file. In
this case download variable is not defined. So define this variable at
the very beginning and check whether it is None after searching for
source archives in the metadata.

Bonus: fix PEP8 issue with wrong indentation.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
 utils/scanpypi | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

Comments

Peter Korsgaard June 15, 2018, 12:13 p.m. | #1
>>>>> "yegorslists" == yegorslists  <yegorslists@googlemail.com> writes:

 > From: Yegor Yefremov <yegorslists@googlemail.com>
 > Some packages don't provide source archive but only a wheel file. In
 > this case download variable is not defined. So define this variable at
 > the very beginning and check whether it is None after searching for
 > source archives in the metadata.

 > Bonus: fix PEP8 issue with wrong indentation.

 > Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>

Committed, thanks.

Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
for the validators module on my Debian (so python 2.7) laptop:

./utils/scanpypi validators
spdx_lookup module is not installed. This can lead to an inaccurate licence detection. Please install it via
pip install spdx_lookup
buildroot package name for validators: python-validators
Package: python-validators
Fetching package validators
Downloading package validators from https://files.pythonhosted.org/packages/45/7b/5b7b74208a3e0744d1a0efbfb1935fa46fa4cfe58d3d63f17c49c58c429c/validators-0.12.2.tar.gz...
Added packages 'six, decorator' as dependencies of python-validators
Checking if package ./package/python-validators already exists...
Error: Package ./package/python-validators already exists
Do you want to delete existing package ? [y/N]y
Creating ./package/python-validators/python-validators.mk...
Creating ./package/python-validators/python-validators.hash...
Creating ./package/python-validators/Config.in...
Traceback (most recent call last):
  File "./utils/scanpypi", line 720, in <module>
    main()
  File "./utils/scanpypi", line 712, in main
    package.create_config_in()
  File "./utils/scanpypi", line 606, in create_config_in
    config_file.writelines(lines)
TypeError: writelines() argument must be a sequence of strings

Printing out 'lines', it seems to be a mix of normal strings and unicode:

lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']
Yegor Yefremov June 15, 2018, 12:30 p.m. | #2
On Fri, Jun 15, 2018 at 2:13 PM, Peter Korsgaard <peter@korsgaard.com> wrote:
>>>>>> "yegorslists" == yegorslists  <yegorslists@googlemail.com> writes:
>
>  > From: Yegor Yefremov <yegorslists@googlemail.com>
>  > Some packages don't provide source archive but only a wheel file. In
>  > this case download variable is not defined. So define this variable at
>  > the very beginning and check whether it is None after searching for
>  > source archives in the metadata.
>
>  > Bonus: fix PEP8 issue with wrong indentation.
>
>  > Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
>
> Committed, thanks.
>
> Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
> for the validators module on my Debian (so python 2.7) laptop:
>
> ./utils/scanpypi validators
> spdx_lookup module is not installed. This can lead to an inaccurate licence detection. Please install it via
> pip install spdx_lookup
> buildroot package name for validators: python-validators
> Package: python-validators
> Fetching package validators
> Downloading package validators from https://files.pythonhosted.org/packages/45/7b/5b7b74208a3e0744d1a0efbfb1935fa46fa4cfe58d3d63f17c49c58c429c/validators-0.12.2.tar.gz...
> Added packages 'six, decorator' as dependencies of python-validators
> Checking if package ./package/python-validators already exists...
> Error: Package ./package/python-validators already exists
> Do you want to delete existing package ? [y/N]y
> Creating ./package/python-validators/python-validators.mk...
> Creating ./package/python-validators/python-validators.hash...
> Creating ./package/python-validators/Config.in...
> Traceback (most recent call last):
>   File "./utils/scanpypi", line 720, in <module>
>     main()
>   File "./utils/scanpypi", line 712, in main
>     package.create_config_in()
>   File "./utils/scanpypi", line 606, in create_config_in
>     config_file.writelines(lines)
> TypeError: writelines() argument must be a sequence of strings
>
> Printing out 'lines', it seems to be a mix of normal strings and unicode:
>
> lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']

I'm still using Python 2 by default. In this case I think trademark
symbol is the problem. That's why it was important to use scanpypi
with Python 3.

Yegor
Jaap Crezee June 15, 2018, 12:59 p.m. | #3
On 06/15/18 14:13, Peter Korsgaard wrote:
> Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
> for the validators module on my Debian (so python 2.7) laptop:

I use python 3(.6) as default python interpreter. The error you posted "sounds" vaguely familliar 
however.

regards,


Jaap
Peter Korsgaard June 15, 2018, 1:05 p.m. | #4
>>>>> "Yegor" == Yegor Yefremov <yegorslists@googlemail.com> writes:

Hi,

 >> Printing out 'lines', it seems to be a mix of normal strings and unicode:
 >> 
 >> lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']

 > I'm still using Python 2 by default. In this case I think trademark
 > symbol is the problem.

Yes. Perhaps we should filter out these unicode characters? We probably
don't really want a TM character in the help text anyway.

Patch

diff --git a/utils/scanpypi b/utils/scanpypi
index 8a2ae00434..12c96b842e 100755
--- a/utils/scanpypi
+++ b/utils/scanpypi
@@ -178,6 +178,7 @@  class BuildrootPackage():
         """
         Download a package using metadata from pypi
         """
+        download = None
         try:
             self.metadata['urls'][0]['filename']
         except IndexError:
@@ -201,7 +202,7 @@  class BuildrootPackage():
                 continue
             try:
                 print('Downloading package {pkg} from {url}...'.format(
-                      pkg=self.real_name, url=download_url['url']))
+                    pkg=self.real_name, url=download_url['url']))
                 download = six.moves.urllib.request.urlopen(download_url['url'])
             except six.moves.urllib.error.HTTPError as http_error:
                 download = http_error
@@ -213,11 +214,14 @@  class BuildrootPackage():
                 self.md5_sum = hashlib.md5(self.as_string).hexdigest()
                 if self.md5_sum == download_url['digests']['md5']:
                     break
-        else:
-            if download.__class__ == six.moves.urllib.error.HTTPError:
-                raise download
-            raise DownloadFailed('Failed to download package {pkg}'
+
+        if download is None:
+            raise DownloadFailed('Failed to download package {pkg}: '
+                                 'No source archive available'
                                  .format(pkg=self.real_name))
+        elif download.__class__ == six.moves.urllib.error.HTTPError:
+            raise download
+
         self.filename = self.used_url['filename']
         self.url = self.used_url['url']