| Message ID | 20180613073710.5715-1-yegorslists@googlemail.com |
|---|---|
| State | Accepted |
| Commit | fb775f4c87c627537efbb56f2663d6079489f018 |
| Headers | show |
| Series | scanpypi: rework download_package error handling | expand |
>>>>> "yegorslists" == yegorslists <yegorslists@googlemail.com> writes: > From: Yegor Yefremov <yegorslists@googlemail.com> > Some packages don't provide source archive but only a wheel file. In > this case download variable is not defined. So define this variable at > the very beginning and check whether it is None after searching for > source archives in the metadata. > Bonus: fix PEP8 issue with wrong indentation. > Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> Committed, thanks. Out of interest, are you using python 2.x or 3.x? I just tried scanpypi for the validators module on my Debian (so python 2.7) laptop: ./utils/scanpypi validators spdx_lookup module is not installed. This can lead to an inaccurate licence detection. Please install it via pip install spdx_lookup buildroot package name for validators: python-validators Package: python-validators Fetching package validators Downloading package validators from https://files.pythonhosted.org/packages/45/7b/5b7b74208a3e0744d1a0efbfb1935fa46fa4cfe58d3d63f17c49c58c429c/validators-0.12.2.tar.gz... Added packages 'six, decorator' as dependencies of python-validators Checking if package ./package/python-validators already exists... Error: Package ./package/python-validators already exists Do you want to delete existing package ? [y/N]y Creating ./package/python-validators/python-validators.mk... Creating ./package/python-validators/python-validators.hash... Creating ./package/python-validators/Config.in... Traceback (most recent call last): File "./utils/scanpypi", line 720, in <module> main() File "./utils/scanpypi", line 712, in main package.create_config_in() File "./utils/scanpypi", line 606, in create_config_in config_file.writelines(lines) TypeError: writelines() argument must be a sequence of strings Printing out 'lines', it seems to be a mix of normal strings and unicode: lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t Python Data Validation for Humans\u2122.\n', '\n', u'\t https://github.com/kvesteri/validators\n']
On Fri, Jun 15, 2018 at 2:13 PM, Peter Korsgaard <peter@korsgaard.com> wrote: >>>>>> "yegorslists" == yegorslists <yegorslists@googlemail.com> writes: > > > From: Yegor Yefremov <yegorslists@googlemail.com> > > Some packages don't provide source archive but only a wheel file. In > > this case download variable is not defined. So define this variable at > > the very beginning and check whether it is None after searching for > > source archives in the metadata. > > > Bonus: fix PEP8 issue with wrong indentation. > > > Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com> > > Committed, thanks. > > Out of interest, are you using python 2.x or 3.x? I just tried scanpypi > for the validators module on my Debian (so python 2.7) laptop: > > ./utils/scanpypi validators > spdx_lookup module is not installed. This can lead to an inaccurate licence detection. Please install it via > pip install spdx_lookup > buildroot package name for validators: python-validators > Package: python-validators > Fetching package validators > Downloading package validators from https://files.pythonhosted.org/packages/45/7b/5b7b74208a3e0744d1a0efbfb1935fa46fa4cfe58d3d63f17c49c58c429c/validators-0.12.2.tar.gz... > Added packages 'six, decorator' as dependencies of python-validators > Checking if package ./package/python-validators already exists... > Error: Package ./package/python-validators already exists > Do you want to delete existing package ? [y/N]y > Creating ./package/python-validators/python-validators.mk... > Creating ./package/python-validators/python-validators.hash... > Creating ./package/python-validators/Config.in... > Traceback (most recent call last): > File "./utils/scanpypi", line 720, in <module> > main() > File "./utils/scanpypi", line 712, in main > package.create_config_in() > File "./utils/scanpypi", line 606, in create_config_in > config_file.writelines(lines) > TypeError: writelines() argument must be a sequence of strings > > Printing out 'lines', it seems to be a mix of normal strings and unicode: > > lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t Python Data Validation for Humans\u2122.\n', '\n', u'\t https://github.com/kvesteri/validators\n'] I'm still using Python 2 by default. In this case I think trademark symbol is the problem. That's why it was important to use scanpypi with Python 3. Yegor
On 06/15/18 14:13, Peter Korsgaard wrote: > Out of interest, are you using python 2.x or 3.x? I just tried scanpypi > for the validators module on my Debian (so python 2.7) laptop: I use python 3(.6) as default python interpreter. The error you posted "sounds" vaguely familliar however. regards, Jaap
>>>>> "Yegor" == Yegor Yefremov <yegorslists@googlemail.com> writes: Hi, >> Printing out 'lines', it seems to be a mix of normal strings and unicode: >> >> lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t Python Data Validation for Humans\u2122.\n', '\n', u'\t https://github.com/kvesteri/validators\n'] > I'm still using Python 2 by default. In this case I think trademark > symbol is the problem. Yes. Perhaps we should filter out these unicode characters? We probably don't really want a TM character in the help text anyway.
diff --git a/utils/scanpypi b/utils/scanpypi index 8a2ae00434..12c96b842e 100755 --- a/utils/scanpypi +++ b/utils/scanpypi @@ -178,6 +178,7 @@ class BuildrootPackage(): """ Download a package using metadata from pypi """ + download = None try: self.metadata['urls'][0]['filename'] except IndexError: @@ -201,7 +202,7 @@ class BuildrootPackage(): continue try: print('Downloading package {pkg} from {url}...'.format( - pkg=self.real_name, url=download_url['url'])) + pkg=self.real_name, url=download_url['url'])) download = six.moves.urllib.request.urlopen(download_url['url']) except six.moves.urllib.error.HTTPError as http_error: download = http_error @@ -213,11 +214,14 @@ class BuildrootPackage(): self.md5_sum = hashlib.md5(self.as_string).hexdigest() if self.md5_sum == download_url['digests']['md5']: break - else: - if download.__class__ == six.moves.urllib.error.HTTPError: - raise download - raise DownloadFailed('Failed to download package {pkg}' + + if download is None: + raise DownloadFailed('Failed to download package {pkg}: ' + 'No source archive available' .format(pkg=self.real_name)) + elif download.__class__ == six.moves.urllib.error.HTTPError: + raise download + self.filename = self.used_url['filename'] self.url = self.used_url['url']