From patchwork Wed Sep 13 13:36:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yegor Yefremov X-Patchwork-Id: 813447 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=googlemail.com header.i=@googlemail.com header.b="Q5wmoLTi"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xsjQ60Mtzz9s4s for ; Wed, 13 Sep 2017 23:37:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0817B30C1D; Wed, 13 Sep 2017 13:37:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id d20X1+jvejft; Wed, 13 Sep 2017 13:37:10 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id E8F6930C1B; Wed, 13 Sep 2017 13:37:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 6AE301C052D for ; Wed, 13 Sep 2017 13:37:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6583B888D1 for ; Wed, 13 Sep 2017 13:37:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YtDM6kUpQkFC for ; Wed, 13 Sep 2017 13:37:07 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f43.google.com (mail-wm0-f43.google.com [74.125.82.43]) by whitealder.osuosl.org (Postfix) with ESMTPS id 3031D88886 for ; Wed, 13 Sep 2017 13:37:07 +0000 (UTC) Received: by mail-wm0-f43.google.com with SMTP id g206so7081797wme.0 for ; Wed, 13 Sep 2017 06:37:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=o+yHFpixBZs5CO8BnO7uCheJK+JJRzDK2nY73yChxJ0=; b=Q5wmoLTi1GtDsWao3fvADZYP5EnO4qQMp0kUhyd6T3ndF56exA8JjuAyMBVbWgm473 sjQ2xURzYCgUMS4PtRZ8lVwZaXU5txpzZYibyLXjUfDav3sAVQL+q2HkePnpctcpj37y nxOnRS8tHL3zQx++kxvE3SfLIokg4iMomVLQBdnZybOrFUPiq1Ve6QZglgWodpTgL4qE 53/N+ttyNU2cvLpovdR15jKYN+oDtnsPippN8B7gnmypKz/qNiEw36TbTuNalzeFHbs6 mb4dc3FVCoVJ6KUXtDhEAohVLOCL+mGWT4OCvgTH2ZItd2Kj9cGQM9uHpIBe6aRl/Cph N4eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=o+yHFpixBZs5CO8BnO7uCheJK+JJRzDK2nY73yChxJ0=; b=kQw2u15KgLuTOHtV4uAmandD5Hc0rxno9xQxqhEOWXMis/wqrFP6nMIbuhtoiVgfTB QPdzK7bBJtfBSpj2v8Hd+yE9ORnWn7qGQx8Wck60vhsqjDFA2eazGUi+cO03EIXE1T9l c3aiHcDjclR2dgET2uugbpi0vgm/aKxRlvhCqcrrhRQ/5f+hMHygStAgAbpRne1DVbcb GrgTp8ePR7Twy7kKHQgI3zA+TOyFVwNBVLnDwK440ml2lUB2aCq+DHyDcamKwGopEGQo 7j4W/QaRO6K2O2FW7rCjXmFGmMzNc4u0TBdLHMlTiUkyCGVxDgiyuFpdMDKXWhMUctG8 urSg== X-Gm-Message-State: AHPjjUiFnrlAbsMs8EApz6pP8B8/E9K+21YN9Ai/7JT76zg9nBdXuKH6 1nQMeoL0IZprHDDx X-Google-Smtp-Source: ADKCNb4KCP+pbW9ROel8IlNQ6DTmsrCE82k5dSDszSoJMXkGsoZWEXVgtXlFH/vtIvneC+JNeWwgPw== X-Received: by 10.80.132.101 with SMTP id 92mr15222584edp.89.1505309824886; Wed, 13 Sep 2017 06:37:04 -0700 (PDT) Received: from debian9.visionsystems.de (mail.visionsystems.de. [213.209.99.202]) by smtp.gmail.com with ESMTPSA id a33sm7114816edd.67.2017.09.13.06.37.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Sep 2017 06:37:04 -0700 (PDT) From: yegorslists@googlemail.com To: buildroot@buildroot.org Date: Wed, 13 Sep 2017 15:36:33 +0200 Message-Id: <20170913133633.8477-1-yegorslists@googlemail.com> X-Mailer: git-send-email 2.11.0 Cc: bernd.kuhls@t-online.de, thomas.petazzoni@free-electrons.com, alexey.roslyakov@gmail.com, fperrad@gmail.com, yann.morin.1998@free.fr Subject: [Buildroot] [RFC] scanpypi: convert to Python 3 syntax X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Yegor Yefremov Some Python 3 only packages have Python 3 syntax in their setup.py files. Besides setup.py files can have UTF-8 encoding. Both cases would make scanpypi fail. Signed-off-by: Yegor Yefremov Signed-off-by: Yegor Yefremov --- utils/scanpypi | 63 +++++++++++++++++++++++++++++----------------------------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/utils/scanpypi b/utils/scanpypi index 02384f2569..9e632c3024 100755 --- a/utils/scanpypi +++ b/utils/scanpypi @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 """ Utility for building Buildroot packages for existing PyPI packages @@ -6,14 +6,16 @@ Utility for building Buildroot packages for existing PyPI packages Any package built by scanpypi should be manually checked for errors. """ -from __future__ import print_function + import argparse import json -import urllib2 +import urllib.request +import urllib.error +import urllib.parse import sys import os import shutil -import StringIO +import io import tarfile import zipfile import errno @@ -62,7 +64,7 @@ def find_file_upper_case(filenames, path='./'): for root, dirs, files in os.walk(path): for file in files: if file.upper() in filenames: - yield (os.path.join(root, file)) + yield os.path.join(root, file) def pkg_buildroot_name(pkg_name): @@ -131,15 +133,15 @@ class BuildrootPackage(): self.metadata_url = 'https://pypi.python.org/pypi/{pkg}/json'.format( pkg=self.real_name) try: - pkg_json = urllib2.urlopen(self.metadata_url).read().decode() - except urllib2.HTTPError as error: + pkg_json = urllib.request.urlopen(self.metadata_url).read().decode() + except urllib.error.HTTPError as error: print('ERROR:', error.getcode(), error.msg, file=sys.stderr) print('ERROR: Could not find package {pkg}.\n' 'Check syntax inside the python package index:\n' 'https://pypi.python.org/pypi/ ' .format(pkg=self.real_name)) raise - except urllib2.URLError: + except urllib.error.URLError: print('ERROR: Could not find package {pkg}.\n' 'Check syntax inside the python package index:\n' 'https://pypi.python.org/pypi/ ' @@ -165,7 +167,7 @@ class BuildrootPackage(): 'md5_digest': None}] # In this case, we can't get the name of the downloaded file # from the pypi api, so we need to find it, this should work - urlpath = urllib2.urlparse.urlparse( + urlpath = urllib.parse.urlparse( self.metadata['info']['download_url']).path # urlparse().path give something like # /path/to/file-version.tar.gz @@ -176,9 +178,9 @@ class BuildrootPackage(): continue try: print('Downloading package {pkg} from {url}...'.format( - pkg=self.real_name, url=download_url['url'])) - download = urllib2.urlopen(download_url['url']) - except urllib2.HTTPError as http_error: + pkg=self.real_name, url=download_url['url'])) + download = urllib.request.urlopen(download_url['url']) + except urllib.error.HTTPError as http_error: download = http_error else: self.used_url = download_url @@ -189,7 +191,7 @@ class BuildrootPackage(): if self.md5_sum == download_url['md5_digest']: break else: - if download.__class__ == urllib2.HTTPError: + if download.__class__ == urllib.error.HTTPError: raise download raise DownloadFailed('Failed to downloas package {pkg}' .format(pkg=self.real_name)) @@ -203,7 +205,7 @@ class BuildrootPackage(): Keyword arguments: tmp_path -- directory where you want the package to be extracted """ - as_file = StringIO.StringIO(self.as_string) + as_file = io.BytesIO(self.as_string) if self.filename[-3:] == 'zip': with zipfile.ZipFile(as_file) as as_zipfile: tmp_pkg = os.path.join(tmp_path, self.buildroot_name) @@ -211,9 +213,9 @@ class BuildrootPackage(): os.makedirs(tmp_pkg) except OSError as exception: if exception.errno != errno.EEXIST: - print("ERROR: ", exception.message, file=sys.stderr) + print("ERROR: ", exception.strerror, file=sys.stderr) return None, None - print('WARNING:', exception.message, file=sys.stderr) + print('WARNING:', exception.strerror, file=sys.stderr) print('Removing {pkg}...'.format(pkg=tmp_pkg)) shutil.rmtree(tmp_pkg) os.makedirs(tmp_pkg) @@ -225,9 +227,9 @@ class BuildrootPackage(): os.makedirs(tmp_pkg) except OSError as exception: if exception.errno != errno.EEXIST: - print("ERROR: ", exception.message, file=sys.stderr) + print("ERROR: ", exception.strerror, file=sys.stderr) return None, None - print('WARNING:', exception.message, file=sys.stderr) + print('WARNING:', exception.strerror, file=sys.stderr) print('Removing {pkg}...'.format(pkg=tmp_pkg)) shutil.rmtree(tmp_pkg) os.makedirs(tmp_pkg) @@ -262,7 +264,7 @@ class BuildrootPackage(): # setup from the __main__ but does not come with a 'main()' function, # for some reason setup.main() will successfully call the main # function of a previous package... - sys.modules.pop('setup',None) + sys.modules.pop('setup', None) del setup os.chdir(current_dir) sys.path.remove(self.tmp_extract) @@ -282,8 +284,8 @@ class BuildrootPackage(): self.pkg_req = [re.sub('([-.\w]+).*', r'\1', req) for req in self.pkg_req] req_not_found = self.pkg_req - self.pkg_req = map(pkg_buildroot_name, self.pkg_req) - pkg_tuples = zip(req_not_found, self.pkg_req) + self.pkg_req = list(map(pkg_buildroot_name, self.pkg_req)) + pkg_tuples = list(zip(req_not_found, self.pkg_req)) # pkg_tuples is a list of tuples that looks like # ('werkzeug','python-werkzeug') because I need both when checking if # dependencies already exist or are already in the download list @@ -403,8 +405,7 @@ class BuildrootPackage(): classifiers_licenses = [regexp.sub(r"\1", lic) for lic in self.metadata['info']['classifiers'] if regexp.match(lic)] - licenses = map(lambda x: license_dict[x] if x in license_dict else x, - classifiers_licenses) + licenses = [license_dict[x] if x in license_dict else x for x in classifiers_licenses] lines = [] if not len(licenses): print('WARNING: License has been set to "{license}". It is most' @@ -417,7 +418,7 @@ class BuildrootPackage(): lines.append(license_line) filenames = ['LICENCE', 'LICENSE', 'LICENSE.RST', 'LICENSE.TXT', - 'COPYING', 'COPYING.TXT'] + 'COPYING', 'COPYING.TXT'] license_files = list(find_file_upper_case(filenames, self.tmp_extract)) license_files = [license.replace(self.tmp_extract, '')[1:] for license in license_files] @@ -484,7 +485,7 @@ class BuildrootPackage(): lines = [] if self.used_url['md5_digest']: md5_comment = '# md5 from {url}, sha256 locally computed\n'.format( - url=self.metadata_url) + url=self.metadata_url) lines.append(md5_comment) hash_line = '{method}\t{digest} {filename}\n'.format( method='md5', @@ -533,7 +534,7 @@ class BuildrootPackage(): # \t + two spaces is 3 char long help_lines.append('') help_lines.append('\t ' + self.metadata['info']['home_page']) - help_lines = map(lambda x: x + '\n', help_lines) + help_lines = [x + '\n' for x in help_lines] lines += help_lines with open(path_to_config, 'w') as config_file: @@ -574,7 +575,7 @@ def main(): print('Fetching package', package.real_name) try: package.fetch_package_info() - except (urllib2.URLError, urllib2.HTTPError): + except (urllib.error.URLError, urllib.error.HTTPError): continue if package.metadata_name.lower() == 'setuptools': # setuptools imports itself, that does not work very well @@ -584,7 +585,7 @@ def main(): try: package.download_package() - except urllib2.HTTPError as error: + except urllib.error.HTTPError as error: print('Error: {code} {reason}'.format(code=error.code, reason=error.reason)) print('Error downloading package :', package.buildroot_name) @@ -603,7 +604,7 @@ def main(): try: package.load_setup() except ImportError as err: - if 'buildutils' in err.message: + if 'buildutils' == err.name: print('This package needs buildutils') else: raise @@ -628,11 +629,11 @@ def main(): os.makedirs(package.pkg_dir) except OSError as exception: if exception.errno != errno.EEXIST: - print("ERROR: ", exception.message, file=sys.stderr) + print("ERROR: ", exception.strerror, file=sys.stderr) continue print('Error: Package {name} already exists' .format(name=package.pkg_dir)) - del_pkg = raw_input( + del_pkg = input( 'Do you want to delete existing package ? [y/N]') if del_pkg.lower() == 'y': shutil.rmtree(package.pkg_dir)