From patchwork Wed Nov 13 17:33:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1194416 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=ovn.org Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47CsDc2FpTz9sPc for ; Thu, 14 Nov 2019 04:34:23 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 9C196E2D; Wed, 13 Nov 2019 17:34:20 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 3806DE28 for ; Wed, 13 Nov 2019 17:34:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 1D233CF for ; Wed, 13 Nov 2019 17:34:17 +0000 (UTC) X-Originating-IP: 73.241.94.255 Received: from localhost.localdomain.localdomain (unknown [73.241.94.255]) (Authenticated sender: hzhou@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 62EE61BF205; Wed, 13 Nov 2019 17:34:15 +0000 (UTC) From: Han Zhou To: dev@openvswitch.org Date: Wed, 13 Nov 2019 09:33:59 -0800 Message-Id: <1573666439-47650-1-git-send-email-hzhou@ovn.org> X-Mailer: git-send-email 2.1.0 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Han Zhou Subject: [ovs-dev] [PATCH] ovsdb raft: Fix election timer parsing in snapshot RPC. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org Commit a76ba825 took care of saving and restoring election timer in file header snapshot, but it didn't handle the parsing of election timer in install_snapshot_request/reply RPC, which results in problems, e.g. when election timer change log is compacted in snapshot and then a new node join the cluster, the new node will use the default timer instead of the new value. This patch fixed it by parsing election timer in snapshot RPC. At the same time the patch updates the test case to cover the DB compact and join senario. The test reveals another 2 problems related to clustered DB compact, as commented in the test case's XXX, which need to be addressed separately. Signed-off-by: Han Zhou --- ovsdb/raft-rpc.c | 5 +++++ ovsdb/raft.c | 2 +- tests/ovsdb-cluster.at | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 55 insertions(+), 1 deletion(-) diff --git a/ovsdb/raft-rpc.c b/ovsdb/raft-rpc.c index 56c07d4..18c83fe 100644 --- a/ovsdb/raft-rpc.c +++ b/ovsdb/raft-rpc.c @@ -511,6 +511,7 @@ raft_install_snapshot_request_to_jsonrpc( json_object_put(args, "last_servers", json_clone(rq->last_servers)); json_object_put_format(args, "last_eid", UUID_FMT, UUID_ARGS(&rq->last_eid)); + raft_put_uint64(args, "election_timer", rq->election_timer); json_object_put(args, "data", json_clone(rq->data)); } @@ -527,6 +528,9 @@ raft_install_snapshot_request_from_jsonrpc( rq->last_index = raft_parse_required_uint64(p, "last_index"); rq->last_term = raft_parse_required_uint64(p, "last_term"); rq->last_eid = raft_parse_required_uuid(p, "last_eid"); + /* election_timer is optional in file header, but is always populated in + * install_snapshot_request. */ + rq->election_timer = raft_parse_required_uint64(p, "election_timer"); rq->data = json_nullable_clone( ovsdb_parser_member(p, "data", OP_OBJECT | OP_ARRAY)); @@ -541,6 +545,7 @@ raft_format_install_snapshot_request( ds_put_format(s, " last_term=%"PRIu64, rq->last_term); ds_put_format(s, " last_eid="UUID_FMT, UUID_ARGS(&rq->last_eid)); ds_put_cstr(s, " last_servers="); + ds_put_format(s, " election_timer=%"PRIu64, rq->election_timer); struct hmap servers; struct ovsdb_error *error = diff --git a/ovsdb/raft.c b/ovsdb/raft.c index a45c7f8..f354d50 100644 --- a/ovsdb/raft.c +++ b/ovsdb/raft.c @@ -3257,7 +3257,7 @@ raft_send_install_snapshot_request(struct raft *raft, .last_servers = raft->snap.servers, .last_eid = raft->snap.eid, .data = raft->snap.data, - .election_timer = raft->election_timer, + .election_timer = raft->election_timer, /* use latest value */ } }; raft_send(raft, &rpc); diff --git a/tests/ovsdb-cluster.at b/tests/ovsdb-cluster.at index 23ed7ec..72f8740 100644 --- a/tests/ovsdb-cluster.at +++ b/tests/ovsdb-cluster.at @@ -233,6 +233,55 @@ done OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/s1 cluster/status $schema_name | grep "Election timer: 4000"]) OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/s2 cluster/status $schema_name | grep "Election timer: 4000"]) +# Latest timer should be restored after DB compact and restart. +# This is to test the install_snapshot RPC. + +# XXX: Insert data before compact, because otherwise transaction will trigger +# busy loop after compact. +# poll_loop|DBG|wakeup due to 0-ms timeout at ../ovsdb/trigger.c:164 (89% CPU usage) +AT_CHECK([ovsdb-client transact unix:s1.ovsdb '[["idltest", + {"op": "insert", + "table": "simple", + "row": {"i": 1}}]]'], [0], [ignore], [ignore]) + +# Compact online +for i in `seq $n`; do + AT_CHECK([ovs-appctl -t "`pwd`"/s$i ovsdb-server/compact]) +done + +# XXX: Insert data after compact, because otherwise vote will fail after +# cluster restart after compact. There will be error logs like: +# raft|ERR|internal error: deferred vote_request message completed but not ready to send because message index 9 is past last synced index 0: s2 vote_request: term=6 last_log_index=9 last_log_term=4 +AT_CHECK([ovsdb-client transact unix:s1.ovsdb '[["idltest", + {"op": "insert", + "table": "simple", + "row": {"i": 1}}]]'], [0], [ignore], [ignore]) + +for i in `seq $n`; do + printf "\ns$i: stopping\n" + OVS_APP_EXIT_AND_WAIT_BY_TARGET([`pwd`/s$i], [s$i.pid]) +done +for i in `seq $n`; do + AT_CHECK([ovsdb-server -v -vconsole:off -vsyslog:off --detach --no-chdir --log-file=s$i.log --pidfile=s$i.pid --unixctl=s$i --remote=punix:s$i.ovsdb s$i.db]) +done +for i in `seq $n`; do + OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/s$i cluster/status $schema_name | grep "Election timer: 4000"]) +done + +# Wait until cluster is ready +for i in `seq $n`; do + OVS_WAIT_WHILE([ovs-appctl -t "`pwd`"/s$i cluster/status $schema_name | grep "Leader: unknown"]) +done + +# Newly joined member should use latest timer value +AT_CHECK([ovsdb-tool join-cluster s4.db $schema_name unix:s4.raft unix:s1.raft]) +AT_CHECK([ovsdb-server -v -vconsole:off -vsyslog:off --detach --no-chdir --log-file=s4.log --pidfile=s4.pid --unixctl=s4 --remote=punix:s4.ovsdb s4.db]) +OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/s4 cluster/status $schema_name | grep "Election timer: 4000"]) +# for i in `seq 10`; do +# ovs-appctl -t "`pwd`"/s4 cluster/status $schema_name +# sleep 1 +# done + AT_CLEANUP