From patchwork Thu Mar 1 06:27:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Guoshuai Li X-Patchwork-Id: 879585 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=dtdream.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zsMtn2cRmz9s2B for ; Thu, 1 Mar 2018 17:28:00 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 86EC6130E; Thu, 1 Mar 2018 06:27:57 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 19A97130C for ; Thu, 1 Mar 2018 06:27:56 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from smtp2203-239.mail.aliyun.com (smtp2203-239.mail.aliyun.com [121.197.203.239]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8376025A for ; Thu, 1 Mar 2018 06:27:53 +0000 (UTC) X-Alimail-AntiSpam: AC=CONTINUE; BC=0.07684739|-1; CH=green; FP=0|0|0|0|0|-1|-1|-1; HT=e01e01454; MF=ligs@dtdream.com; NM=1; PH=DS; RN=2; RT=2; SR=0; TI=SMTPD_---.B7g3JuW_1519885668; Received: from localhost.localdomain(mailfrom:ligs@dtdream.com fp:222.128.6.202) by smtp.aliyun-inc.com(10.147.41.178); Thu, 01 Mar 2018 14:27:49 +0800 From: Guoshuai Li To: ovs-dev@openvswitch.org Date: Thu, 1 Mar 2018 14:27:37 +0800 Message-Id: <20180301062737.7028-1-ligs@dtdream.com> X-Mailer: git-send-email 2.13.2.windows.1 MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v4] python: Fix decoding error when the received data is larger than 4096. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It can only receive 4096 bytes of data each time in jsonrpc, when there are similar and Chinese characters occupy multiple bytes, it may receive half a character, this time the decoding will be abnormal. We need to receive the completed character to decode. Signed-off-by: Guoshuai Li --- python/ovs/jsonrpc.py | 5 +++-- tests/ovsdb-idl.at | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/python/ovs/jsonrpc.py b/python/ovs/jsonrpc.py index d284190e0..7c24e574a 100644 --- a/python/ovs/jsonrpc.py +++ b/python/ovs/jsonrpc.py @@ -11,7 +11,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +import codecs import errno import os import sys @@ -262,6 +262,7 @@ class Connection(object): if self.status: return self.status, None + decoder = codecs.getincrementaldecoder('utf-8')() while True: if not self.input: error, data = self.stream.recv(4096) @@ -270,7 +271,7 @@ class Connection(object): # data, so we convert it here as soon as possible. if data and not error: try: - data = data.decode('utf-8') + data = decoder.decode(data) except UnicodeError: error = errno.EILSEQ if error: diff --git a/tests/ovsdb-idl.at b/tests/ovsdb-idl.at index 59b2c1991..cd02851b3 100644 --- a/tests/ovsdb-idl.at +++ b/tests/ovsdb-idl.at @@ -328,6 +328,39 @@ OVSDB_CHECK_IDL([simple idl, writing via IDL with unicode], 003: done ]]) +m4_define([OVSDB_CHECK_IDL_PYN_WITH_EXPOUT], + [AT_SETUP([$1]) + AT_SKIP_IF([test $7 = no]) + AT_KEYWORDS([ovsdb server idl positive Python $5]) + AT_CHECK([ovsdb_start_idltest]) + m4_if([$2], [], [], + [AT_CHECK([ovsdb-client transact unix:socket $2], [0], [ignore], [ignore])]) + AT_CHECK([$8 $srcdir/test-ovsdb.py -t10 idl $srcdir/idltest.ovsschema unix:socket $3], + [0], [stdout], [ignore]) + echo "$4" > expout + AT_CHECK([sort stdout | uuidfilt]m4_if([$6],,, [[| $6]]), + [0], [expout]) + OVSDB_SERVER_SHUTDOWN + AT_CLEANUP]) + +m4_define([OVSDB_CHECK_IDL_PY_WITH_EXPOUT], + [OVSDB_CHECK_IDL_PYN_WITH_EXPOUT([$1 - Python2], [$2], [$3], [$4], [$5], [$6], + [$HAVE_PYTHON], [$PYTHON]) + OVSDB_CHECK_IDL_PYN_WITH_EXPOUT([$1 - Python3], [$2], [$3], [$4], [$5], [$6], + [$HAVE_PYTHON3], [$PYTHON3])]) + +OVSDB_CHECK_IDL_PY_WITH_EXPOUT([simple idl, writing large data via IDL with unicode], + [['["idltest", + {"op": "insert", + "table": "simple", + "row": {"s": "'$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..50})'"}}]']], + [['set 0 b 1, insert 1, set 1 s '$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..100})'']], + [[000: i=0 r=0 b=false s=$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..50}) u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<1> +001: commit, status=success +002: i=0 r=0 b=true s=$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..50}) u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<1> +002: i=1 r=0 b=false s=$(printf "测试超过四千零九十六个字节的中文字符串以使解码出现问题。%.0s" {1..100}) u=<0> ia=[] ra=[] ba=[] sa=[] ua=[] uuid=<2> +003: done]]) + OVSDB_CHECK_IDL([simple idl, handling verification failure], [['["idltest", {"op": "insert",