From patchwork Sun Apr 11 04:27:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1464745 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=Qf714ZZA; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FHzRT14Pcz9sVt for ; Sun, 11 Apr 2021 14:29:41 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id E6FCD80C69; Sun, 11 Apr 2021 06:28:52 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="Qf714ZZA"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 2862380C5B; Sun, 11 Apr 2021 06:28:42 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 9ED978051D for ; Sun, 11 Apr 2021 06:28:31 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-pf1-x42e.google.com with SMTP id d124so6930012pfa.13 for ; Sat, 10 Apr 2021 21:28:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yyrkOzoeWR24obpUOD7nH6fq87hBfapNQJ6UVDVr8ys=; b=Qf714ZZAOzpP36RU/dcpcTKqS5HefREMluo08sDSPaIMYzpu/N9AIK005fNcVWjrzT uZZdZX+3E5Q9zkP7cAxBiPOwBgk21JhoNot8WXDPeDeucBi3byHeWtrsW5+/kfeUrne1 rN2kIBxq+Aavur+1PvEPJBmx5/GwsI6z5FhJY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yyrkOzoeWR24obpUOD7nH6fq87hBfapNQJ6UVDVr8ys=; b=G0rOUQ8KerSh1wEawII2XFzSqXMNGRR0GJunp4SeMHBfLFM0lVDt2Ftbnl6Iq5aK2o RRz0cg0V/GdhBGdlERKmQGsUN3IAs3qylHSOW0XGBwYvmCzXFMNYmQeH4/cTm2PtZJ72 iC6uRvZN9qcoOKUtJ0VJRh9tEKf7yo0eBP3VkbZ0azn2SHq1YDFt7EgBb/yMr9kZyerM 8qvmvEVTZsBUm+x9NvGyXQjXf7u2T38DXTVYjFXRm5LZQ3itna7Qje3UHLB+vA1ijkqC BcOO+xpc8R2TGTWPRyc5bDXQyMIVMCY7JBq2R14b0H9sjpI3Y2aVEbxjUnOq0K5FqDtg 8yqg== X-Gm-Message-State: AOAM532N8pGriKBGwm/ndwZTOcJxa4EtCHOjWLB1MfC0cD9I6K2hE2fQ sbUtZIrHiYKjEUrtn+ZL34/gU1g5ofLV+rAD X-Google-Smtp-Source: ABdhPJzyr4HH8pxeYa1nYxy7VPPzTwTCqn/As31SG01WGniHoaM/jngIwe/NZcER1AnfaJyYuQLKpg== X-Received: by 2002:a63:f914:: with SMTP id h20mr20842216pgi.114.1618115309897; Sat, 10 Apr 2021 21:28:29 -0700 (PDT) Received: from localhost.localdomain (203-173-153-56.dsl.dyn.ihug.co.nz. [203.173.153.56]) by smtp.gmail.com with ESMTPSA id 186sm2856589pfe.212.2021.04.10.21.28.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 21:28:29 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Tom Rini , Marek Vasut , Simon Glass Subject: [PATCH 4/4] buildman: Use bytes for the environment Date: Sun, 11 Apr 2021 16:27:28 +1200 Message-Id: <20210411162720.4.I7de274bda337a2f2c3cb952c78aeda3f2155d521@changeid> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog In-Reply-To: <20210411042728.3549261-1-sjg@chromium.org> References: <20210411042728.3549261-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.4 at phobos.denx.de X-Virus-Status: Clean At present we sometimes see problems in gitlab where the environment has 0x80 characters or sequences which are not valid UTF-8. Avoid this by using bytes for the environment, both internal to buildman and when writing out the 'env' file. Add a test to make sure this works as expected. Reported-by: Marek Vasut Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file") Signed-off-by: Simon Glass Reported-by: Marek Vasut Signed-off-by: Simon Glass --- tools/buildman/builderthread.py | 5 ++--- tools/buildman/func_test.py | 12 ++++++++++++ tools/buildman/toolchain.py | 24 ++++++++++++++++-------- 3 files changed, 30 insertions(+), 11 deletions(-) diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index ddb3eab8c03..48128cf6732 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -351,10 +351,9 @@ class BuilderThread(threading.Thread): # Write out the image and function size information and an objdump env = result.toolchain.MakeEnvironment(self.builder.full_path) - with open(os.path.join(build_dir, 'out-env'), 'w', - encoding='utf-8') as fd: + with open(os.path.join(build_dir, 'out-env'), 'wb') as fd: for var in sorted(env.keys()): - print('%s="%s"' % (var, env[var]), file=fd) + fd.write(b'%s="%s"' % (var, env[var])) lines = [] for fname in BASE_ELF_FILENAMES: cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname] diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py index 61e3012d2b1..7edbee0652f 100644 --- a/tools/buildman/func_test.py +++ b/tools/buildman/func_test.py @@ -572,6 +572,18 @@ class TestFunctional(unittest.TestCase): self.assertTrue(os.path.exists(os.path.join(board0_dir, 'done'))) self.assertTrue(os.path.exists(os.path.join(board0_dir, 'out-env'))) + def testEnvironmentUnicode(self): + """Test there are no unicode errors when the env has non-ASCII chars""" + try: + varname = b'buildman_test_var' + os.environb[varname] = b'strange\x80chars' + self.assertEqual(0, self._RunControl('-o', self._output_dir)) + board0_dir = os.path.join(self._output_dir, 'current', 'board0') + self.assertTrue(os.path.exists(os.path.join(board0_dir, 'done'))) + self.assertTrue(os.path.exists(os.path.join(board0_dir, 'out-env'))) + finally: + del os.environb[varname] + def testWorkInOutput(self): """Test the -w option which should write directly to the output dir""" board_list = board.Boards() diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py index acb5a29c8fb..fd137f7300e 100644 --- a/tools/buildman/toolchain.py +++ b/tools/buildman/toolchain.py @@ -179,27 +179,35 @@ class Toolchain: output and possibly unicode encoded output of all build tools by adding LC_ALL=C. + Note that os.environb is used to obtain the environment, since in some + cases the environment many contain non-ASCII characters and we see + errors like: + + UnicodeEncodeError: 'utf-8' codec can't encode characters in position + 569-570: surrogates not allowed + Args: full_path: Return the full path in CROSS_COMPILE and don't set PATH Returns: - Dict containing the environemnt to use. This is based on the current - environment, with changes as needed to CROSS_COMPILE, PATH and - LC_ALL. + Dict containing the (bytes) environment to use. This is based on the + current environment, with changes as needed to CROSS_COMPILE, PATH + and LC_ALL. """ - env = dict(os.environ) + env = dict(os.environb) wrapper = self.GetWrapper() if self.override_toolchain: # We'll use MakeArgs() to provide this pass elif full_path: - env['CROSS_COMPILE'] = wrapper + os.path.join(self.path, self.cross) + env[b'CROSS_COMPILE'] = tools.ToBytes( + wrapper + os.path.join(self.path, self.cross)) else: - env['CROSS_COMPILE'] = wrapper + self.cross - env['PATH'] = self.path + ':' + env['PATH'] + env[b'CROSS_COMPILE'] = tools.ToBytes(wrapper + self.cross) + env[b'PATH'] = tools.ToBytes(self.path) + b':' + env[b'PATH'] - env['LC_ALL'] = 'C' + env[b'LC_ALL'] = b'C' return env