From patchwork Sun Feb 18 12:51:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Guoshuai Li X-Patchwork-Id: 874865 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zkmwT2P1rz9sWW for ; Sun, 18 Feb 2018 23:51:37 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 947EAD8C; Sun, 18 Feb 2018 12:51:32 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 44E27D17 for ; Sun, 18 Feb 2018 12:51:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from smtp2203-239.mail.aliyun.com (smtp2203-239.mail.aliyun.com [121.197.203.239]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 3D1965B5 for ; Sun, 18 Feb 2018 12:51:29 +0000 (UTC) X-Alimail-AntiSpam: AC=CONTINUE; BC=0.07461952|-1; CH=green; FP=13277434917833198659|1|1|1|0|-1|-1|-1; HT=e01l07447; MF=ligs@dtdream.com; NM=1; PH=DS; RN=2; RT=2; SR=0; TI=SMTPD_---.AzXd4V5_1518958286; Received: from localhost.localdomain(mailfrom:ligs@dtdream.com fp:222.128.6.212) by smtp.aliyun-inc.com(10.147.42.197); Sun, 18 Feb 2018 20:51:27 +0800 From: Guoshuai Li To: ovs-dev@openvswitch.org Date: Sun, 18 Feb 2018 20:51:16 +0800 Message-Id: <20180218125116.5032-1-ligs@dtdream.com> X-Mailer: git-send-email 2.13.2.windows.1 MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2] python: Fix decoding error when the received data is larger than 4096. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It can only receive 4096 bytes of data each time in jsonrpc, when there are similar and Chinese characters occupy multiple bytes, it may receive half a character, this time the decoding will be abnormal. We need to receive the completed character to decode. Signed-off-by: Guoshuai Li --- python/ovs/jsonrpc.py | 14 +++++++++----- tests/ovsdb-idl.at | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 5 deletions(-) diff --git a/python/ovs/jsonrpc.py b/python/ovs/jsonrpc.py index d284190e0..a46821093 100644 --- a/python/ovs/jsonrpc.py +++ b/python/ovs/jsonrpc.py @@ -262,17 +262,21 @@ class Connection(object): if self.status: return self.status, None + recv_buffer = bytearray() while True: if not self.input: + recv_data = None error, data = self.stream.recv(4096) + recv_buffer.extend(bytes(data)) # Python 3 has separate types for strings and bytes. We # received bytes from a socket. We expect it to be string # data, so we convert it here as soon as possible. if data and not error: try: - data = data.decode('utf-8') + recv_data = recv_buffer.decode('utf-8') + recv_buffer = bytearray() except UnicodeError: - error = errno.EILSEQ + continue if error: if (sys.platform == "win32" and error == errno.WSAEWOULDBLOCK): @@ -287,12 +291,12 @@ class Connection(object): % (self.name, os.strerror(error))) self.error(error) return self.status, None - elif not data: + elif not recv_data: self.error(EOF) return EOF, None else: - self.input += data - self.received_bytes += len(data) + self.input += recv_data + self.received_bytes += len(recv_data) else: if self.parser is None: self.parser = ovs.json.Parser() diff --git a/tests/ovsdb-idl.at b/tests/ovsdb-idl.at index 59b2c1991..d63b7758f 100644 --- a/tests/ovsdb-idl.at +++ b/tests/ovsdb-idl.at @@ -328,6 +328,40 @@ OVSDB_CHECK_IDL([simple idl, writing via IDL with unicode], 003: done ]]) +# same as OVSDB_CHECK_IDL but uses the Python IDL implementation. +m4_define([OVSDB_CHECK_IDL_PYN_WITH_EXPOUT], + [AT_SETUP([$1]) + AT_SKIP_IF([test $7 = no]) + AT_KEYWORDS([ovsdb server idl positive Python $5]) + AT_CHECK([ovsdb_start_idltest]) + m4_if([$2], [], [], + [AT_CHECK([ovsdb-client transact unix:socket $2], [0], [ignore], [ignore])]) + AT_CHECK([$8 $srcdir/test-ovsdb.py -t10 idl $srcdir/idltest.ovsschema unix:socket $3], + [0], [stdout], [ignore]) + echo "$4" > expout + AT_CHECK([sort stdout | uuidfilt]m4_if([$6],,, [[| $6]]), + [0], [expout]) + OVSDB_SERVER_SHUTDOWN + AT_CLEANUP]) + +m4_define([OVSDB_CHECK_IDL_PY_WITH_EXPOUT], + [OVSDB_CHECK_IDL_PYN_WITH_EXPOUT([$1 - Python2], [$2], [$3], [$4], [$5], [$6], + [$HAVE_PYTHON], [$PYTHON]) + OVSDB_CHECK_IDL_PYN_WITH_EXPOUT([$1 - Python3], [$2], [$3], [$4], [$5], [$6], + [$HAVE_PYTHON3], [$PYTHON3])]) + +OVSDB_CHECK_IDL_PY_WITH_EXPOUT([simple idl, writing large data via IDL with unicode], + [['["idltest", + {"op": "insert", + "table": "simple", + "row": {"s": "'$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..50})'"}}]']], + [['set 0 b 1, insert 1, set 1 s '$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..100})'']], + [[000: i=0 r=0 b=false s=$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..50}) u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<1> +001: commit, status=success +002: i=0 r=0 b=true s=$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..50}) u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<1> +002: i=1 r=0 b=false s=$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..100}) u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<2> +003: done]]) + OVSDB_CHECK_IDL([simple idl, handling verification failure], [['["idltest", {"op": "insert",