From patchwork Tue Feb 23 13:18:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1443479 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DlKPP3XWdz9sRN for ; Wed, 24 Feb 2021 00:18:33 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2599685815; Tue, 23 Feb 2021 13:18:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WVokYb1Z9I0P; Tue, 23 Feb 2021 13:18:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id 12BBD855CC; Tue, 23 Feb 2021 13:18:31 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E9FEAC000A; Tue, 23 Feb 2021 13:18:30 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 05297C0001 for ; Tue, 23 Feb 2021 13:18:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id E483A83A52 for ; Tue, 23 Feb 2021 13:18:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1y0cvpL1gciM for ; Tue, 23 Feb 2021 13:18:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by smtp1.osuosl.org (Postfix) with ESMTPS id 5792083A4D for ; Tue, 23 Feb 2021 13:18:27 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 921856000C; Tue, 23 Feb 2021 13:18:22 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 23 Feb 2021 14:18:20 +0100 Message-Id: <20210223131820.53813-1-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH] raft: Add 'stop-raft-rpc' failure test command. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This command will stop sending and receiving any RAFT-related traffic or accepting new connections. Useful to simulate network problems between cluster members. There is no unit test that uses it yet, but it's convenient for manual testing. Signed-off-by: Ilya Maximets Acked-by: Han Zhou --- ovsdb/raft.c | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/ovsdb/raft.c b/ovsdb/raft.c index 192f7f0a9..d2ff643b2 100644 --- a/ovsdb/raft.c +++ b/ovsdb/raft.c @@ -75,7 +75,8 @@ enum raft_failure_test { FT_CRASH_AFTER_SEND_EXEC_REQ, FT_CRASH_AFTER_RECV_APPEND_REQ_UPDATE, FT_DELAY_ELECTION, - FT_DONT_SEND_VOTE_REQUEST + FT_DONT_SEND_VOTE_REQUEST, + FT_STOP_RAFT_RPC, }; static enum raft_failure_test failure_test; @@ -1474,6 +1475,10 @@ raft_send_add_server_request(struct raft *raft, struct raft_conn *conn) static void raft_conn_run(struct raft *raft, struct raft_conn *conn) { + if (failure_test == FT_STOP_RAFT_RPC) { + return; + } + jsonrpc_session_run(conn->js); unsigned int new_seqno = jsonrpc_session_get_seqno(conn->js); @@ -1794,7 +1799,8 @@ static void raft_open_conn(struct raft *raft, const char *address, const struct uuid *sid) { if (strcmp(address, raft->local_address) - && !raft_find_conn_by_address(raft, address)) { + && !raft_find_conn_by_address(raft, address) + && failure_test != FT_STOP_RAFT_RPC) { raft_add_conn(raft, jsonrpc_session_open(address, true), sid, false); } } @@ -1870,7 +1876,7 @@ raft_run(struct raft *raft) free(paddr); } - if (raft->listener) { + if (raft->listener && failure_test != FT_STOP_RAFT_RPC) { struct stream *stream; int error = pstream_accept(raft->listener, &stream); if (!error) { @@ -1995,7 +2001,7 @@ raft_run(struct raft *raft) static void raft_wait_session(struct jsonrpc_session *js) { - if (js) { + if (js && failure_test != FT_STOP_RAFT_RPC) { jsonrpc_session_wait(js); jsonrpc_session_recv_wait(js); } @@ -2012,10 +2018,12 @@ raft_wait(struct raft *raft) raft_waiters_wait(raft); - if (raft->listener) { - pstream_wait(raft->listener); - } else { - poll_timer_wait_until(raft->listen_backoff); + if (failure_test != FT_STOP_RAFT_RPC) { + if (raft->listener) { + pstream_wait(raft->listener); + } else { + poll_timer_wait_until(raft->listen_backoff); + } } struct raft_conn *conn; @@ -4361,8 +4369,9 @@ raft_send_to_conn_at(struct raft *raft, const union raft_rpc *rpc, struct raft_conn *conn, int line_number) { log_rpc(rpc, "-->", conn, line_number); - return !jsonrpc_session_send( - conn->js, raft_rpc_to_jsonrpc(&raft->cid, &raft->sid, rpc)); + return failure_test == FT_STOP_RAFT_RPC + || !jsonrpc_session_send( + conn->js, raft_rpc_to_jsonrpc(&raft->cid, &raft->sid, rpc)); } static bool @@ -4856,6 +4865,8 @@ raft_unixctl_failure_test(struct unixctl_conn *conn OVS_UNUSED, } } else if (!strcmp(test, "dont-send-vote-request")) { failure_test = FT_DONT_SEND_VOTE_REQUEST; + } else if (!strcmp(test, "stop-raft-rpc")) { + failure_test = FT_STOP_RAFT_RPC; } else if (!strcmp(test, "clear")) { failure_test = FT_NO_TEST; unixctl_command_reply(conn, "test dismissed");