From patchwork Wed Apr 5 12:46:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Robin Jarry X-Patchwork-Id: 747268 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3vyly45yLlz9s86 for ; Wed, 5 Apr 2017 22:48:28 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=6wind-com.20150623.gappssmtp.com header.i=@6wind-com.20150623.gappssmtp.com header.b="J36NWY0E"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3vyly44x5wzDqJ6 for ; Wed, 5 Apr 2017 22:48:28 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=6wind-com.20150623.gappssmtp.com header.i=@6wind-com.20150623.gappssmtp.com header.b="J36NWY0E"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Received: from mail-wr0-x232.google.com (mail-wr0-x232.google.com [IPv6:2a00:1450:400c:c0c::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3vylxz53X2zDqHK for ; Wed, 5 Apr 2017 22:48:22 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=6wind-com.20150623.gappssmtp.com header.i=@6wind-com.20150623.gappssmtp.com header.b="J36NWY0E"; dkim-atps=neutral Received: by mail-wr0-x232.google.com with SMTP id k6so13004553wre.2 for ; Wed, 05 Apr 2017 05:48:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DJM1XirP+dUMsC2Fg4f419aVsPUxjaZJyO+7b04VNo4=; b=J36NWY0EPcoM9Sx6imbtI/fhpSQiR0+sDhzolMcXHTPbpXhxmwSLc9hfcLMpLZn2R3 iiUIia6LKYxmlE/opUIKTSB0lNjBF7uj3USaGRdEQUGs5RJb7E//RuuWQndB8kiqylZi Z8UkDVigRiy7e3//jjTOJpNba5OnLOIzPo0gkQX/IwuqoO2ESrnjitw3CbWyjHz+2v+6 pu4D8uCMG+8nWwRJQhpHCo21OrrqVDp3LsrXnUeQJPqMiem3fBm5h+0gVnxtRknPEu6i aZlS0HQiFucAEuB1OB3gHcMyOjMcmvuH9pSrIx3JnaWrBUh6/6MCnTHtlDCktxkKEfeE oBPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DJM1XirP+dUMsC2Fg4f419aVsPUxjaZJyO+7b04VNo4=; b=heUes0LRwDnCqNd7ST2DbblWZVawbzjYMlfSBlkk2HPHFYW93UeaGKaC6OdMcyjfzp PJzbIRrtS2WWQXAogH4e+O9TUvJgsRlzTDe1+FLXglwpCPE9vrBZgRVCNmk8gAV8mtab 9BkpWrWncxeki5A1k2zCv+IN2FqSpd61JNwmfmJBibFo6ur1MTHPMBPtAHedbVH8K5SZ OvnthxRZ7AfPZ2Ik0tczVJzIK08/MkHwd8OvYp+0fi2sFyg2zHacGo9y6zdqYtAPjn+B DfyI1sV7O8P55Iqa9pawaQImaWBHn3urptYRaEEnkadA/STB4loqLYEP296Dd/grc8TL GZgw== X-Gm-Message-State: AFeK/H0Bk9GlZV4uRIvTBHgfWoPAa6XWYSasD+hK1047lL0zpAQefDpLbSlsq9S05YGU9+v6 X-Received: by 10.223.147.100 with SMTP id 91mr26059513wro.89.1491396496867; Wed, 05 Apr 2017 05:48:16 -0700 (PDT) Received: from tom.dev.6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id n6sm2552472wrb.62.2017.04.05.05.48.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 05 Apr 2017 05:48:16 -0700 (PDT) From: Robin Jarry To: Stephen Finucane Subject: [PATCH v2] pwclient: Fix silent crash on python 2 Date: Wed, 5 Apr 2017 14:46:04 +0200 Message-Id: <20170405124604.28022-1-robin.jarry@6wind.com> X-Mailer: git-send-email 2.11.0.193.g1d1bdafd6426 In-Reply-To: <20170405113552.7vbiyjgnfykoojmn@6wind.com> References: <20170405113552.7vbiyjgnfykoojmn@6wind.com> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: patchwork@lists.ozlabs.org Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Replacing sys.stdout and sys.stderr can cause obscure crashes when trying to write non unicode data. The interpreter is terminated with SIGINT without any specific error writen on the console. rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f964820e8d0}, {0x559f50, [], SA_RESTORER, 0x7f964820e8d0}, 8) = 0 This happens easily when there is an untrapped exception which should lead to printing a traceback on stderr. The only way to prevent UnicodeEncodeErrors is to make sure that PYTHONIOENCODING is set with the ':replace' suffix and this can only be done *before* starting the interpreter as the initialization is made very early on and the encoding cannot be set or modified after. From the official documentation: PYTHONIOENCODING Overrides the encoding used for stdin/stdout/stderr, in the syntax encodingname:errorhandler. The :errorhandler part is optional and has the same meaning as in str.encode(). https://docs.python.org/2/using/cmdline.html https://docs.python.org/3/using/cmdline.html Of course, for proper encoding of unicode characters, one of the locale-related environment variables (LC_ALL, LANG, LANGUAGE, etc.) must be set. Python will use the correct encoding accordingly and PYTHONIOENCODING will be set to "encoding:replace". Examples: $ grep utf8 ~/.Xresources xterm*utf8: 2 $ env - PYTHONIOENCODING=utf-8:replace python2 -c "print u's\u00e9duisante'" séduisante $ env - PYTHONIOENCODING=utf-8:replace python3 -c "print('s\u00e9duisante')" séduisante $ env - PYTHONIOENCODING=ascii:replace python2 -c "print u's\u00e9duisante'" s?duisante $ env - PYTHONIOENCODING=ascii:replace python3 -c "print('s\u00e9duisante')" s?duisante $ env - PYTHONIOENCODING=ISO-8859-1:replace python2 -c "print u's\u00e9duisante'" s�duisante $ env - PYTHONIOENCODING=ISO-8859-1:replace python3 -c "print('s\u00e9duisante')" s�duisante Fixes: 046419a3bf8f ("pwclient: Fix encoding problems") Signed-off-by: Robin Jarry --- v2: - Always set PYTHONIOENCODING=:replace (on python 2 and 3) to prevent from UnicodeEncodeErrors patchwork/bin/pwclient | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/patchwork/bin/pwclient b/patchwork/bin/pwclient index ed0351bf5288..5a7b6723afe3 100755 --- a/patchwork/bin/pwclient +++ b/patchwork/bin/pwclient @@ -41,16 +41,7 @@ except ImportError: import shutil import re import io -import locale -if sys.version_info.major == 2: - # hack to make writing unicode to standard output/error work on Python 2 - OUT_ENCODING = (sys.stdout.encoding or locale.getpreferredencoding() or - os.getenv('PYTHONIOENCODING', 'utf-8')) - sys.stdout = io.open(sys.stdout.fileno(), mode='w', - encoding=OUT_ENCODING, errors='replace') - sys.stderr = io.open(sys.stderr.fileno(), mode='w', - encoding=OUT_ENCODING, errors='replace') # Default Patchwork remote XML-RPC server URL # This script will check the PW_XMLRPC_URL environment variable @@ -821,5 +812,31 @@ def main(): sys.exit(1) +def force_io_encoding(): + """ + Force PYTHONIOENCODING ":errorhandler" to avoid UnicodeEncodeErrors. The + only way to do it is to set the environment variable *before* starting the + interpreter. From the python docs: + + PYTHONIOENCODING + + Overrides the encoding used for stdin/stdout/stderr, in the syntax + encodingname:errorhandler. The :errorhandler part is optional and has the + same meaning as in str.encode(). + + Note that this only prevents interpreter crashes, it does not exempt from + correctly setting the LANG or LC_ALL variables in order to have valid + output. + """ + if 'PYTHONIOENCODING' in os.environ: + return + + encoding = sys.stdout.encoding or 'utf-8' + + os.environ['PYTHONIOENCODING'] = encoding + ':replace' + os.execvp(sys.executable, [sys.executable] + sys.argv) # no return + + if __name__ == "__main__": + force_io_encoding() main()