From patchwork Sat Aug 11 03:48:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 956517 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="PC0Nyxpr"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41nSfR5FMKz9s82 for ; Sat, 11 Aug 2018 13:49:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id D212A81A1F; Sat, 11 Aug 2018 03:49:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2CsIzB1IhCn2; Sat, 11 Aug 2018 03:49:13 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id D6AE88657D; Sat, 11 Aug 2018 03:49:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 07E831C0492 for ; Sat, 11 Aug 2018 03:49:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 017678657D for ; Sat, 11 Aug 2018 03:49:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8P0uC_AOL39U for ; Sat, 11 Aug 2018 03:49:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-qt0-f195.google.com (mail-qt0-f195.google.com [209.85.216.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 0F0118655C for ; Sat, 11 Aug 2018 03:49:11 +0000 (UTC) Received: by mail-qt0-f195.google.com with SMTP id h4-v6so12400000qtj.7 for ; Fri, 10 Aug 2018 20:49:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gkIAu99oFQBbRhyqdCa5IhjVzYGwYIkOEH1nMgLWb80=; b=PC0NyxprEUUffxslY3WHvnX+HvsoJbcSsxKk58ep9PGEwbv+YADTXCZx5dGu6OUgzL +YOwsKr9q5xGZezHAZdWRLhWiWUPoK1mdLpkMFulnpXx5upXZ46ShadUzhc7kbYbJBbD 1u9EFe+QJ7GAHJG1E7tHCGr8cUy+pLjcG6LfRYYHypuvKcs4ypFyhOVz+iCBG4ZFGz0O ag/kqXOZhU1oyfKlCgAgAixIzpyptcIv/hBoeNmaeT/bpVVKd531LzMmnL+5rfSG0qD8 IKBY1K1HZMd21iBN8Ewe2ep8VkLqFdEp7Yg/8hsyr3SPJJxmjUMhd+Hs9bm4zyroIiQM DNig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=gkIAu99oFQBbRhyqdCa5IhjVzYGwYIkOEH1nMgLWb80=; b=OuigszyzwVtERsB5ST60e2tj1fhEbWjQAo1lyGrhX4qcFLbRYrBNXnIAdImr/kOMxj F3WfiQ8+j1qcZk6bykg1ViLmR193oXOEDjMvGWh7EhAAKQfvmQ6ziqHrZqYvlwABJJ/A VIURxx19UFUD4lD/5iezyZ3lvdDYV7kuyV+PvoGcYzb40im+bMxAcTEXnovDUoAjaS1l V8cBKB/FUdg5hMibV3iV2TlZjLsPnGXUIBLsNWUcom8jd3ayks6oKeyK8bAAL3uNz5Ek ycEVogA9uET/jM5gR7kIXC67SqZ1BurUv6dzW5pMdi/LGJYtXR3cWKPS1VOpZIC/N6by EL4w== X-Gm-Message-State: AOUpUlGlWGf3psktQ0QujJYvgjYuKIWNOasQGVyHxauKQX0V8mQ267xG B6H3n5cXrtWla8OPWIltaNSdjalt X-Google-Smtp-Source: AA+uWPwUULrhu9+9+IspH+A9zEXSz5doL8mij524U4mVQfbzkJg9IDy3LyWsFQyGvHZlqEk/NCdh3w== X-Received: by 2002:a0c:e505:: with SMTP id l5-v6mr8099959qvm.128.1533959349727; Fri, 10 Aug 2018 20:49:09 -0700 (PDT) Received: from ultri5.home ([179.181.66.128]) by smtp.gmail.com with ESMTPSA id t28-v6sm7263312qki.82.2018.08.10.20.49.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 10 Aug 2018 20:49:08 -0700 (PDT) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Sat, 11 Aug 2018 00:48:27 -0300 Message-Id: <20180811034827.14504-1-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180723004107.20985-1-ricardo.martincoski@gmail.com> References: <20180723004107.20985-1-ricardo.martincoski@gmail.com> Subject: [Buildroot] [PATCH v2 1/1] check-package: fix Python3 support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This script currently uses "/usr/bin/env python" as shebang but it does not really support Python3. Instead of limiting the script to Python2, fix it to support both versions. So change all imports to absolute imports because Python3 follows PEP328 and dropped implicit relative imports. In order to avoid errors when decoding files with the default 'utf-8' codec, use errors="surrogateescape" when opening files, the docs for open() states: "This is useful for processing files in an unknown encoding.". This argument is not compatible with Python2 open() so import 'six' to use it only when running in Python3. As a consequence the file handler becomes explicit, so use it to close() the file after it got processed. This "surrogateescape" is a simple alternative to the complete solution of opening files with "rb" and changing all functions in the lib*.py files to use bytes objects instead of strings. The only case we can have non-ascii/non-utf-8 files being checked by the script are for patch files when the upstream file to be patched is not ascii or utf-8. There is currently one case in the tree: package/urg/0002-urg-gcc6-fix-narrowing-conversion.patch. Signed-off-by: Ricardo Martincoski Cc: Arnout Vandecappelle Reviewed-by: Thomas De Schampheleire Tested-by: Thomas De Schampheleire --- Changes v1 -> v2: - do not use an automated tool (modernize); - use absolute import; - drop list() added to .keys(), it's not needed (Arnout); - drop 'from __future__ import absolute_import' everywhere; - list the (not used) approach of opening files in binary mode in the commit message; Using the default docker (debian) image which uses Python2: before this patch: https://gitlab.com/buildroot.org/buildroot/-/jobs/88548503 after this patch: https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/88561994 Using an arch docker image which uses Python3: before this patch: https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/88561488 after this patch: https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/88561704 The arch image for docker was generated using: http://patchwork.ozlabs.org/patch/943301/ --- utils/check-package | 8 +++++++- utils/checkpackagelib/lib.py | 2 +- utils/checkpackagelib/lib_config.py | 10 +++++----- utils/checkpackagelib/lib_hash.py | 10 +++++----- utils/checkpackagelib/lib_mk.py | 10 +++++----- utils/checkpackagelib/lib_patch.py | 4 ++-- 6 files changed, 25 insertions(+), 19 deletions(-) diff --git a/utils/check-package b/utils/check-package index 3dbc28b0a2..26439f08eb 100755 --- a/utils/check-package +++ b/utils/check-package @@ -6,6 +6,7 @@ import argparse import inspect import os import re +import six import sys import checkpackagelib.lib_config @@ -127,10 +128,15 @@ def check_file_using_lib(fname): for cf in objects: nwarnings += print_warnings(cf.before()) - for lineno, text in enumerate(open(fname, "r").readlines()): + if six.PY3: + f = open(fname, "r", errors="surrogateescape") + else: + f = open(fname, "r") + for lineno, text in enumerate(f.readlines()): nlines += 1 for cf in objects: nwarnings += print_warnings(cf.check_line(lineno + 1, text)) + f.close() for cf in objects: nwarnings += print_warnings(cf.after()) diff --git a/utils/checkpackagelib/lib.py b/utils/checkpackagelib/lib.py index 91f4ad49b7..6afe1aabce 100644 --- a/utils/checkpackagelib/lib.py +++ b/utils/checkpackagelib/lib.py @@ -1,6 +1,6 @@ # See utils/checkpackagelib/readme.txt before editing this file. -from base import _CheckFunction +from checkpackagelib.base import _CheckFunction class ConsecutiveEmptyLines(_CheckFunction): diff --git a/utils/checkpackagelib/lib_config.py b/utils/checkpackagelib/lib_config.py index 1d273f1c5f..89d44da57e 100644 --- a/utils/checkpackagelib/lib_config.py +++ b/utils/checkpackagelib/lib_config.py @@ -5,11 +5,11 @@ import re -from base import _CheckFunction -from lib import ConsecutiveEmptyLines # noqa: F401 -from lib import EmptyLastLine # noqa: F401 -from lib import NewlineAtEof # noqa: F401 -from lib import TrailingSpace # noqa: F401 +from checkpackagelib.base import _CheckFunction +from checkpackagelib.lib import ConsecutiveEmptyLines # noqa: F401 +from checkpackagelib.lib import EmptyLastLine # noqa: F401 +from checkpackagelib.lib import NewlineAtEof # noqa: F401 +from checkpackagelib.lib import TrailingSpace # noqa: F401 def _empty_or_comment(text): diff --git a/utils/checkpackagelib/lib_hash.py b/utils/checkpackagelib/lib_hash.py index 6d4cc9fd62..3e381119a5 100644 --- a/utils/checkpackagelib/lib_hash.py +++ b/utils/checkpackagelib/lib_hash.py @@ -5,11 +5,11 @@ import re -from base import _CheckFunction -from lib import ConsecutiveEmptyLines # noqa: F401 -from lib import EmptyLastLine # noqa: F401 -from lib import NewlineAtEof # noqa: F401 -from lib import TrailingSpace # noqa: F401 +from checkpackagelib.base import _CheckFunction +from checkpackagelib.lib import ConsecutiveEmptyLines # noqa: F401 +from checkpackagelib.lib import EmptyLastLine # noqa: F401 +from checkpackagelib.lib import NewlineAtEof # noqa: F401 +from checkpackagelib.lib import TrailingSpace # noqa: F401 def _empty_line_or_comment(text): diff --git a/utils/checkpackagelib/lib_mk.py b/utils/checkpackagelib/lib_mk.py index 86e9aa2d97..9bf417fb73 100644 --- a/utils/checkpackagelib/lib_mk.py +++ b/utils/checkpackagelib/lib_mk.py @@ -6,11 +6,11 @@ import re -from base import _CheckFunction -from lib import ConsecutiveEmptyLines # noqa: F401 -from lib import EmptyLastLine # noqa: F401 -from lib import NewlineAtEof # noqa: F401 -from lib import TrailingSpace # noqa: F401 +from checkpackagelib.base import _CheckFunction +from checkpackagelib.lib import ConsecutiveEmptyLines # noqa: F401 +from checkpackagelib.lib import EmptyLastLine # noqa: F401 +from checkpackagelib.lib import NewlineAtEof # noqa: F401 +from checkpackagelib.lib import TrailingSpace # noqa: F401 class Indent(_CheckFunction): diff --git a/utils/checkpackagelib/lib_patch.py b/utils/checkpackagelib/lib_patch.py index 555621afa1..453b782e6c 100644 --- a/utils/checkpackagelib/lib_patch.py +++ b/utils/checkpackagelib/lib_patch.py @@ -5,8 +5,8 @@ import re -from base import _CheckFunction -from lib import NewlineAtEof # noqa: F401 +from checkpackagelib.base import _CheckFunction +from checkpackagelib.lib import NewlineAtEof # noqa: F401 class ApplyOrder(_CheckFunction):