From patchwork Wed Jun 27 02:14:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 935224 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41Fmhs5d9Zz9s29 for ; Wed, 27 Jun 2018 12:15:25 +1000 (AEST) Received: from localhost ([::1]:56126 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXzzP-0006hS-9z for incoming@patchwork.ozlabs.org; Tue, 26 Jun 2018 22:15:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38554) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXzyk-0006fu-KY for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:14:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXzyj-00020l-PO for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:14:42 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52000) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fXzyj-0001zu-Ia for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:14:41 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D240F30820C2; Wed, 27 Jun 2018 02:14:40 +0000 (UTC) Received: from localhost (ovpn-116-35.gru2.redhat.com [10.97.116.35]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7CBE082F6D; Wed, 27 Jun 2018 02:14:32 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Jun 2018 23:14:18 -0300 Message-Id: <20180627021423.18404-2-ehabkost@redhat.com> In-Reply-To: <20180627021423.18404-1-ehabkost@redhat.com> References: <20180627021423.18404-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Wed, 27 Jun 2018 02:14:40 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 1/6] docker: Use BytesIO instead of StringIO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Cleber Rosa , =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The file passed as argument to TarFile.addfile() must be a binary file, so BytesIO is more appropriate than StringIO. This is necessary to make the code work on Python 3. Signed-off-by: Eduardo Habkost --- tests/docker/docker.py | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index 8e13f18e6c..0de7662146 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -24,10 +24,7 @@ import tempfile import re import signal from tarfile import TarFile, TarInfo -try: - from StringIO import StringIO -except ImportError: - from io import StringIO +from io import BytesIO from shutil import copy, rmtree from pwd import getpwuid from datetime import datetime,timedelta @@ -372,13 +369,13 @@ class UpdateCommand(SubCommand): tmp_tar.add(os.path.realpath(l), arcname=l) # Create a Docker buildfile - df = StringIO() - df.write("FROM %s\n" % args.tag) - df.write("ADD . /\n") - df.seek(0) + df = BytesIO() + df.write(b"FROM %s\n" % args.tag.encode()) + df.write(b"ADD . /\n") df_tar = TarInfo(name="Dockerfile") - df_tar.size = len(df.buf) + df_tar.size = df.tell() + df.seek(0) tmp_tar.addfile(df_tar, fileobj=df) tmp_tar.close() From patchwork Wed Jun 27 02:14:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 935225 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41FmjG0VWvz9s0w for ; Wed, 27 Jun 2018 12:15:46 +1000 (AEST) Received: from localhost ([::1]:56127 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXzzj-0006wE-ID for incoming@patchwork.ozlabs.org; Tue, 26 Jun 2018 22:15:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38642) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXzz0-0006tz-2H for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:14:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXzyu-0002BW-Uv for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:14:58 -0400 Received: from mx1.redhat.com ([209.132.183.28]:51654) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fXzyu-0002BF-Ox for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:14:52 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 177D23084029; Wed, 27 Jun 2018 02:14:52 +0000 (UTC) Received: from localhost (ovpn-116-35.gru2.redhat.com [10.97.116.35]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3692460BB4; Wed, 27 Jun 2018 02:14:42 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Jun 2018 23:14:19 -0300 Message-Id: <20180627021423.18404-3-ehabkost@redhat.com> In-Reply-To: <20180627021423.18404-1-ehabkost@redhat.com> References: <20180627021423.18404-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.40]); Wed, 27 Jun 2018 02:14:52 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 2/6] docker: Always return int on run() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Cleber Rosa , =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We'll add type annotations to the run() methods, so add 'return' statements to all the functions so the type checker won't complain. Signed-off-by: Eduardo Habkost --- tests/docker/docker.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index 0de7662146..e3bfa1cc9e 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -418,7 +418,7 @@ class ProbeCommand(SubCommand): except Exception: print("no") - return + return 0 class CcCommand(SubCommand): @@ -503,6 +503,7 @@ class CheckCommand(SubCommand): print ("Image less than %d minutes old" % (args.olderthan)) return 0 + return 0 def main(): parser = argparse.ArgumentParser(description="A Docker helper", From patchwork Wed Jun 27 02:14:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 935227 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41Fmmh4r1vz9s0w for ; Wed, 27 Jun 2018 12:18:44 +1000 (AEST) Received: from localhost ([::1]:56139 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fY02c-0000Sf-9D for incoming@patchwork.ozlabs.org; Tue, 26 Jun 2018 22:18:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38661) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXzz3-0006yx-Ma for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXzz2-0002G2-BR for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:01 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52072) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fXzz2-0002Fg-2s for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:00 -0400 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.25]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6580330820D5; Wed, 27 Jun 2018 02:14:59 +0000 (UTC) Received: from localhost (ovpn-116-35.gru2.redhat.com [10.97.116.35]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8504B20155E4; Wed, 27 Jun 2018 02:14:53 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Jun 2018 23:14:20 -0300 Message-Id: <20180627021423.18404-4-ehabkost@redhat.com> In-Reply-To: <20180627021423.18404-1-ehabkost@redhat.com> References: <20180627021423.18404-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.25 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.47]); Wed, 27 Jun 2018 02:14:59 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 3/6] docker: Add type annotations X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Cleber Rosa , =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Add type annotations that indicate how the code works today, to make the conversion to Python 3 easier and safer. With these type annotations, "mypy -2" is not reporting any issues, but "mypy" in Python 3 mode reports a few problems: tests/docker/docker.py:233: error: Argument 1 to "_text_checksum" has incompatible type "str"; expected "bytes" tests/docker/docker.py:358: error: "_Environ[str]" has no attribute "iteritems" tests/docker/docker.py:360: error: Argument 3 to "build_image" of "Docker" has incompatible type "bytes"; expected "str" These problems will be addressed by the following commits. Signed-off-by: Eduardo Habkost --- tests/docker/docker.py | 44 +++++++++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index e3bfa1cc9e..db6b463b92 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -29,6 +29,12 @@ from shutil import copy, rmtree from pwd import getpwuid from datetime import datetime,timedelta +try: + from typing import List, Union, Tuple +except ImportError: + # needed only to make type annotations work + pass + FILTERED_ENV_NAMES = ['ftp_proxy', 'http_proxy', 'https_proxy'] @@ -37,13 +43,16 @@ DEVNULL = open(os.devnull, 'wb') def _text_checksum(text): + # type: (bytes) -> str """Calculate a digest string unique to the text content""" return hashlib.sha1(text).hexdigest() def _file_checksum(filename): + # type: (str) -> str return _text_checksum(open(filename, 'rb').read()) def _guess_docker_command(): + # type: () -> List[str] """ Guess a working docker command or raise exception if not found""" commands = [["docker"], ["sudo", "-n", "docker"]] for cmd in commands: @@ -60,6 +69,7 @@ def _guess_docker_command(): commands_txt) def _copy_with_mkdir(src, root_dir, sub_path='.'): + # type: (str, str, str) -> None """Copy src into root_dir, creating sub_path as needed.""" dest_dir = os.path.normpath("%s/%s" % (root_dir, sub_path)) try: @@ -73,6 +83,7 @@ def _copy_with_mkdir(src, root_dir, sub_path='.'): def _get_so_libs(executable): + # type: (str) -> List[str] """Return a list of libraries associated with an executable. The paths may be symbolic links which would need to be resolved to @@ -94,6 +105,7 @@ def _get_so_libs(executable): return libs def _copy_binary_with_libs(src, dest_dir): + # type: (str, str) -> None """Copy a binary executable and all its dependant libraries. This does rely on the host file-system being fairly multi-arch @@ -108,11 +120,13 @@ def _copy_binary_with_libs(src, dest_dir): _copy_with_mkdir(l , dest_dir, so_path) def _read_qemu_dockerfile(img_name): + # type: (str) -> str df = os.path.join(os.path.dirname(__file__), "dockerfiles", img_name + ".docker") return open(df, "r").read() def _dockerfile_preprocess(df): + # type: (str) -> str out = "" for l in df.splitlines(): if len(l.strip()) == 0 or l.startswith("#"): @@ -194,11 +208,16 @@ class Docker(object): labels = json.loads(resp)[0]["Config"].get("Labels", {}) return labels.get("com.qemu.dockerfile-checksum", "") - def build_image(self, tag, docker_dir, dockerfile, - quiet=True, user=False, argv=None, extra_files_cksum=[]): - if argv == None: - argv = [] - + def build_image(self, + tag, # type: str + docker_dir, # type: str + dockerfile, # type: str + quiet=True, # type: bool + user=False, # type: bool + argv=[], # type: List[str] + extra_files_cksum=[] # List[Tuple[str, bytes]] + ): + # type(...) -> None tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker") tmp_df.write(dockerfile) @@ -249,7 +268,8 @@ class Docker(object): class SubCommand(object): """A SubCommand template base class""" - name = None # Subcommand name + # Subcommand name + name = None # type: str def shared_args(self, parser): parser.add_argument("--quiet", action="store_true", help="Run quietly unless an error occured") @@ -258,6 +278,7 @@ class SubCommand(object): """Setup argument parser""" pass def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int """Run command. args: parsed argument by argument parser. argv: remaining arguments from sys.argv. @@ -271,6 +292,7 @@ class RunCommand(SubCommand): parser.add_argument("--keep", action="store_true", help="Don't remove image when command completes") def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int return Docker().run(argv, args.keep, quiet=args.quiet) class BuildCommand(SubCommand): @@ -294,6 +316,7 @@ class BuildCommand(SubCommand): help="Dockerfile name") def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int dockerfile = open(args.dockerfile, "rb").read() tag = args.tag @@ -321,7 +344,7 @@ class BuildCommand(SubCommand): # Copy any extra files into the Docker context. These can be # included by the use of the ADD directive in the Dockerfile. - cksum = [] + cksum = [] # type: List[Tuple[bytes, str]] if args.include_executable: # FIXME: there is no checksum of this executable and the linked # libraries, once the image built any change of this executable @@ -352,6 +375,7 @@ class UpdateCommand(SubCommand): help="Executable to copy") def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int # Create a temporary tarball with our whole build context and # dockerfile for the update tmp = tempfile.NamedTemporaryFile(suffix="dckr.tar.gz") @@ -394,6 +418,7 @@ class CleanCommand(SubCommand): """Clean up docker instances""" name = "clean" def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int Docker().clean() return 0 @@ -401,6 +426,7 @@ class ImagesCommand(SubCommand): """Run "docker images" command""" name = "images" def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int return Docker().command("images", argv, args.quiet) @@ -409,6 +435,7 @@ class ProbeCommand(SubCommand): name = "probe" def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int try: docker = Docker() if docker._command[0] == "docker": @@ -437,6 +464,7 @@ class CcCommand(SubCommand): reading sources""") def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int if argv and argv[0] == "--": argv = argv[1:] cwd = os.getcwd() @@ -468,6 +496,7 @@ class CheckCommand(SubCommand): help="number of minutes") def run(self, args, argv): + # type: (argparse.Namespace, List[str]) -> int tag = args.tag dkr = Docker() @@ -506,6 +535,7 @@ class CheckCommand(SubCommand): return 0 def main(): + # type: () -> int parser = argparse.ArgumentParser(description="A Docker helper", usage="%s ..." % os.path.basename(sys.argv[0])) subparsers = parser.add_subparsers(title="subcommands", help=None) From patchwork Wed Jun 27 02:14:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 935228 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41Fmmx3Y46z9s0w for ; Wed, 27 Jun 2018 12:18:57 +1000 (AEST) Received: from localhost ([::1]:56141 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fY02p-0000dj-33 for incoming@patchwork.ozlabs.org; Tue, 26 Jun 2018 22:18:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38682) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXzz5-00070O-0X for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXzz4-0002J9-4B for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:03 -0400 Received: from mx1.redhat.com ([209.132.183.28]:37858) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fXzz3-0002IA-Uf for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:02 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 36FFB81F09; Wed, 27 Jun 2018 02:15:01 +0000 (UTC) Received: from localhost (ovpn-116-35.gru2.redhat.com [10.97.116.35]) by smtp.corp.redhat.com (Postfix) with ESMTP id B6E2360BB4; Wed, 27 Jun 2018 02:15:00 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Jun 2018 23:14:21 -0300 Message-Id: <20180627021423.18404-5-ehabkost@redhat.com> In-Reply-To: <20180627021423.18404-1-ehabkost@redhat.com> References: <20180627021423.18404-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Wed, 27 Jun 2018 02:15:01 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 4/6] docker: Use os.environ.items() instead of .iteritems() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Cleber Rosa , =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Mapping.iteritems() doesn't exist in Python 3. Note that Mapping.items() exists in both Python 3 and Python 2, but it returns a list (and not an iterator) in Python 2. The existing code will work on both cases, though. Signed-off-by: Eduardo Habkost Reviewed-by: Philippe Mathieu-Daudé --- tests/docker/docker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index db6b463b92..bc34bd872b 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -355,7 +355,7 @@ class BuildCommand(SubCommand): cksum += [(filename, _file_checksum(filename))] argv += ["--build-arg=" + k.lower() + "=" + v - for k, v in os.environ.iteritems() + for k, v in os.environ.items() if k.lower() in FILTERED_ENV_NAMES] dkr.build_image(tag, docker_dir, dockerfile, quiet=args.quiet, user=args.user, argv=argv, From patchwork Wed Jun 27 02:14:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 935226 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41Fmjl1ZJxz9s0w for ; Wed, 27 Jun 2018 12:16:11 +1000 (AEST) Received: from localhost ([::1]:56133 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fY008-0007Gq-Nu for incoming@patchwork.ozlabs.org; Tue, 26 Jun 2018 22:16:08 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38919) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXzzM-0007FW-NU for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXzzH-0002en-Nk for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:20 -0400 Received: from mx1.redhat.com ([209.132.183.28]:54150) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fXzzH-0002e4-IO for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:15 -0400 Received: from smtp.corp.redhat.com (int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D4BE3C057F4E; Wed, 27 Jun 2018 02:15:14 +0000 (UTC) Received: from localhost (ovpn-116-35.gru2.redhat.com [10.97.116.35]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8977DB74DD; Wed, 27 Jun 2018 02:15:02 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Jun 2018 23:14:22 -0300 Message-Id: <20180627021423.18404-6-ehabkost@redhat.com> In-Reply-To: <20180627021423.18404-1-ehabkost@redhat.com> References: <20180627021423.18404-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.27 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Wed, 27 Jun 2018 02:15:14 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 5/6] docker: Make _get_so_libs() work on Python 3 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Cleber Rosa , =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The "ldd" output is a byte sequence, not a string. Use bytes literals while handling the output, and use os.fsdecode() on the resulting file paths before returning. Signed-off-by: Eduardo Habkost --- tests/docker/docker.py | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index bc34bd872b..f58af8e894 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -41,6 +41,15 @@ FILTERED_ENV_NAMES = ['ftp_proxy', 'http_proxy', 'https_proxy'] DEVNULL = open(os.devnull, 'wb') +def _fsdecode(name): + # type: (bytes) -> str + """Decode filename to str, try to use os.fsdecode() if available""" + if hasattr(os, 'fsdecode'): + # Python 3 + return os.fsdecode(name) # type: ignore + else: + # Python 2.7 + return name # type: ignore def _text_checksum(text): # type: (bytes) -> str @@ -90,15 +99,15 @@ def _get_so_libs(executable): ensure theright data is copied.""" libs = [] - ldd_re = re.compile(r"(/.*/)(\S*)") + ldd_re = re.compile(b"(/.*/)(\S*)") try: ldd_output = subprocess.check_output(["ldd", executable]) - for line in ldd_output.split("\n"): + for line in ldd_output.split(b"\n"): search = ldd_re.search(line) if search and len(search.groups()) == 2: so_path = search.groups()[0] so_lib = search.groups()[1] - libs.append("%s/%s" % (so_path, so_lib)) + libs.append(_fsdecode(b"%s/%s" % (so_path, so_lib))) except subprocess.CalledProcessError: print("%s had no associated libraries (static build?)" % (executable)) From patchwork Wed Jun 27 02:14:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduardo Habkost X-Patchwork-Id: 935229 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41FmnK0dqKz9s29 for ; Wed, 27 Jun 2018 12:19:17 +1000 (AEST) Received: from localhost ([::1]:56143 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fY036-0000tC-5B for incoming@patchwork.ozlabs.org; Tue, 26 Jun 2018 22:19:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38964) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fXzzP-0007I6-GI for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fXzzO-0002mB-9z for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:23 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35922) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fXzzO-0002lN-2H for qemu-devel@nongnu.org; Tue, 26 Jun 2018 22:15:22 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5E56C13A9D; Wed, 27 Jun 2018 02:15:21 +0000 (UTC) Received: from localhost (ovpn-116-35.gru2.redhat.com [10.97.116.35]) by smtp.corp.redhat.com (Postfix) with ESMTP id 317CD100194A; Wed, 27 Jun 2018 02:15:16 +0000 (UTC) From: Eduardo Habkost To: qemu-devel@nongnu.org Date: Tue, 26 Jun 2018 23:14:23 -0300 Message-Id: <20180627021423.18404-7-ehabkost@redhat.com> In-Reply-To: <20180627021423.18404-1-ehabkost@redhat.com> References: <20180627021423.18404-1-ehabkost@redhat.com> X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Wed, 27 Jun 2018 02:15:21 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH 6/6] docker: Open dockerfiles in text mode X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Stefan Hajnoczi , Cleber Rosa , =?utf-8?q?Alex_Benn=C3=A9e?= Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Instead of treating dockerfile contents as byte sequences, always open dockerfiles in text mode and treat it as text. This is not strictly required to make the script compatible with Python 3, but it's a simpler and safer way than opening dockerfiles in binary mode and decoding the data data later. To make the code compatible with both Python 2 and 3, use io.open(), which accepts a 'encoding' argument on both versions. Signed-off-by: Eduardo Habkost --- tests/docker/docker.py | 46 ++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/tests/docker/docker.py b/tests/docker/docker.py index f58af8e894..412a031c1c 100755 --- a/tests/docker/docker.py +++ b/tests/docker/docker.py @@ -23,6 +23,7 @@ import argparse import tempfile import re import signal +import io from tarfile import TarFile, TarInfo from io import BytesIO from shutil import copy, rmtree @@ -30,7 +31,7 @@ from pwd import getpwuid from datetime import datetime,timedelta try: - from typing import List, Union, Tuple + from typing import List, Union, Tuple, Text except ImportError: # needed only to make type annotations work pass @@ -52,13 +53,13 @@ def _fsdecode(name): return name # type: ignore def _text_checksum(text): - # type: (bytes) -> str + # type: (Text) -> str """Calculate a digest string unique to the text content""" - return hashlib.sha1(text).hexdigest() + return hashlib.sha1(text.encode('utf-8')).hexdigest() def _file_checksum(filename): # type: (str) -> str - return _text_checksum(open(filename, 'rb').read()) + return _text_checksum(io.open(filename, 'r', encoding='utf-8').read()) def _guess_docker_command(): # type: () -> List[str] @@ -129,14 +130,14 @@ def _copy_binary_with_libs(src, dest_dir): _copy_with_mkdir(l , dest_dir, so_path) def _read_qemu_dockerfile(img_name): - # type: (str) -> str + # type: (Text) -> str df = os.path.join(os.path.dirname(__file__), "dockerfiles", img_name + ".docker") - return open(df, "r").read() + return io.open(df, "r", encoding='utf-8').read() def _dockerfile_preprocess(df): - # type: (str) -> str - out = "" + # type: (Text) -> Text + out = u"" for l in df.splitlines(): if len(l.strip()) == 0 or l.startswith("#"): continue @@ -149,7 +150,7 @@ def _dockerfile_preprocess(df): inlining = _read_qemu_dockerfile(l[len(from_pref):]) out += _dockerfile_preprocess(inlining) continue - out += l + "\n" + out += l + u"\n" return out class Docker(object): @@ -220,32 +221,37 @@ class Docker(object): def build_image(self, tag, # type: str docker_dir, # type: str - dockerfile, # type: str + dockerfile, # type: Text quiet=True, # type: bool user=False, # type: bool argv=[], # type: List[str] extra_files_cksum=[] # List[Tuple[str, bytes]] ): # type(...) -> None - tmp_df = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker") + tmp_ndf = tempfile.NamedTemporaryFile(dir=docker_dir, suffix=".docker") + # on Python 2.7, NamedTemporaryFile doesn't support encoding parameter, + # so reopen it in text mode: + tmp_df = io.open(tmp_ndf.name, mode='w+t', encoding='utf-8') tmp_df.write(dockerfile) if user: uid = os.getuid() uname = getpwuid(uid).pw_name - tmp_df.write("\n") - tmp_df.write("RUN id %s 2>/dev/null || useradd -u %d -U %s" % + tmp_df.write(u"\n") + tmp_df.write(u"RUN id %s 2>/dev/null || useradd -u %d -U %s" % (uname, uid, uname)) - tmp_df.write("\n") - tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s" % - _text_checksum(_dockerfile_preprocess(dockerfile))) + dockerfile = _dockerfile_preprocess(dockerfile) + + tmp_df.write(u"\n") + tmp_df.write(u"LABEL com.qemu.dockerfile-checksum=%s" % + _text_checksum(dockerfile)) for f, c in extra_files_cksum: - tmp_df.write("LABEL com.qemu.%s-checksum=%s" % (f, c)) + tmp_df.write(u"LABEL com.qemu.%s-checksum=%s" % (f, c)) tmp_df.flush() - self._do_check(["build", "-t", tag, "-f", tmp_df.name] + argv + \ + self._do_check(["build", "-t", tag, "-f", tmp_ndf.name] + argv + \ [docker_dir], quiet=quiet) @@ -326,7 +332,7 @@ class BuildCommand(SubCommand): def run(self, args, argv): # type: (argparse.Namespace, List[str]) -> int - dockerfile = open(args.dockerfile, "rb").read() + dockerfile = io.open(args.dockerfile, "r", encoding='utf-8').read() tag = args.tag dkr = Docker() @@ -519,7 +525,7 @@ class CheckCommand(SubCommand): print("Need a dockerfile for tag:%s" % (tag)) return 1 - dockerfile = open(args.dockerfile, "rb").read() + dockerfile = io.open(args.dockerfile, "r", encoding='utf-8').read() if dkr.image_matches_dockerfile(tag, dockerfile): if not args.quiet: