From patchwork Thu Apr 20 08:14:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Hoffmann X-Patchwork-Id: 1771136 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Q29Rh4ylHz23hs for ; Thu, 20 Apr 2023 18:14:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id BED3881F60; Thu, 20 Apr 2023 08:14:22 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org BED3881F60 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MUStDp1ebbwY; Thu, 20 Apr 2023 08:14:21 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id BA43381D67; Thu, 20 Apr 2023 08:14:20 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org BA43381D67 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9A350C0037; Thu, 20 Apr 2023 08:14:20 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 20459C002A for ; Thu, 20 Apr 2023 08:14:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id D4F9681E3B for ; Thu, 20 Apr 2023 08:14:18 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org D4F9681E3B X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XxY3bf0dcxtV for ; Thu, 20 Apr 2023 08:14:18 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org E3D1A81D67 Received: from mx3.cloudandheat.com (mx3.cloudandheat.com [185.128.118.157]) by smtp1.osuosl.org (Postfix) with ESMTPS id E3D1A81D67 for ; Thu, 20 Apr 2023 08:14:17 +0000 (UTC) Received: by mx3.cloudandheat.com with esmtpsa (TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.93) (envelope-from ) id 1ppPQl-006LT3-WE for dev@openvswitch.org; Thu, 20 Apr 2023 08:14:16 +0000 Message-ID: <3f70ca7bafad296e18ed9579f30fd7044c47fc61.camel@cloudandheat.com> From: Stefan Hoffmann To: dev@openvswitch.org Date: Thu, 20 Apr 2023 10:14:14 +0200 User-Agent: Evolution 3.44.4-0ubuntu1 MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [ovs-dev] [PATCH] python-stream: handle SSL error in do_handshake X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In some cases ovsdb server or relay gets restarted, ovsdb python clients may keep the local socket open. Instead of reconnecting a lot of failures will be logged. This can be reproduced with ssl connections to the server/relay and restarting it, so it has the same IP after restart. This patch catches the Exceptions at do_handshake to recreate the connection on the client side. Tracebacks from the issue: Traceback (most recent call last): File \"/usr/local/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py\", line 107, in run self.idl.run() File \"/usr/local/lib/python3.9/site-packages/ovs-3.1.0-py3.9.egg/ovs/db/idl.py\", line 433, in run self._session.run() File \"/usr/local/lib/python3.9/site-packages/ovs-3.1.0-py3.9.egg/ovs/jsonrpc.py\", line 519, in run error = self.stream.connect() File \"/usr/local/lib/python3.9/site-packages/ovs-3.1.0-py3.9.egg/ovs/stream.py\", line 824, in connect self.socket.do_handshake() File \"/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py\", line 312, in do_handshake return self._call_trampolining( File \"/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py\", line 158, in _call_trampolining return func(*a, **kw) File \"/usr/local/lib/python3.9/ssl.py\", line 1310, in do_handshake self._sslobj.do_handshake() ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1129) 2023-04-03 14:06:43.458 1 ERROR ovsdbapp.backend.ovs_idl.connection 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection [-] TLS/SSL connection has been closed (EOF) (_ssl.c:997): ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997) 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last): 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/local/lib/python3.10/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 107, in run 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection     self.idl.run() 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/local/lib/python3.10/dist-packages/ovs/db/idl.py", line 433, in run 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection     self._session.run() 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/local/lib/python3.10/dist-packages/ovs/jsonrpc.py", line 519, in run 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection     error = self.stream.connect() 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/local/lib/python3.10/dist-packages/ovs/stream.py", line 824, in connect 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection     self.socket.do_handshake() 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection   File "/usr/lib/python3.10/ssl.py", line 1342, in do_handshake 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection     self._sslobj.do_handshake() 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997) 2023-04-03 14:06:43.513 1 ERROR ovsdbapp.backend.ovs_idl.connection 2023-04-03 14:06:43.567 1 ERROR ovsdbapp.backend.ovs_idl.connection [-] TLS/SSL connection has been closed (EOF) (_ssl.c:997): ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997) Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 107, in run self.idl.run() File "/usr/local/lib/python3.9/site-packages/ovs-3.1.0-py3.9.egg/ovs/db/idl.py", line 433, in run self._session.run() File "/usr/local/lib/python3.9/site-packages/ovs-3.1.0-py3.9.egg/ovs/jsonrpc.py", line 519, in run error = self.stream.connect() File "/usr/local/lib/python3.9/site-packages/ovs-3.1.0-py3.9.egg/ovs/stream.py", line 824, in connect self.socket.do_handshake() File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 312, in do_handshake return self._call_trampolining( File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 158, in _call_trampolining return func(*a, **kw) File "/usr/local/lib/python3.9/ssl.py", line 1305, in do_handshake self._check_connected() File "/usr/local/lib/python3.9/ssl.py", line 1089, in _check_connected self.getpeername() OSError: [Errno 107] Transport endpoint is not connected Signed-off-by: Stefan Hoffmann Signed-off-by: Luca Czesla Signed-off-by: Max Lamprecht Co-authored-by: Luca Czesla Co-authored-by: Max Lamprecht Reviewed-by: Simon Horman Reviewed-by: Simon Horman --- python/ovs/stream.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/python/ovs/stream.py b/python/ovs/stream.py index ac5b0fd0c..b32341076 100644 --- a/python/ovs/stream.py +++ b/python/ovs/stream.py @@ -824,7 +824,8 @@ class SSLStream(Stream): self.socket.do_handshake() except ssl.SSLWantReadError: return errno.EAGAIN - except ssl.SSLSyscallError as e: + except (ssl.SSLSyscallError, ssl.SSLZeroReturnError, + ssl.SSLEOFError, OSError) as e: return ovs.socket_util.get_exception_errno(e) return 0