From patchwork Wed Aug 14 22:37:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1147275 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Fz9M+CvV"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4684Gw2WFKz9sN1 for ; Thu, 15 Aug 2019 08:38:00 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id E2393CCC; Wed, 14 Aug 2019 22:37:26 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 7E566CAA for ; Wed, 14 Aug 2019 22:37:25 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 8035076D for ; Wed, 14 Aug 2019 22:37:24 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id s15so642849edx.0 for ; Wed, 14 Aug 2019 15:37:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sjHLrc5tBk1BMC7N2Gqe2KeA/oPgnloD9SuKcolwBbw=; b=Fz9M+CvVI44J/KGYEzRUTBm88nDNhIdDokna7Kf9ELPt4R1rqAKUPxgU3pTzgipnru 8/CZ9t2q/p9u9hrbNE+CL8PWCiSYhRVLUXP4zaRC4P5UTev37NQ/258c0CLtKZQ/eNwU lGxQwSXCb6jTIWbhzQHblTxAh99lFpH6OmfRuK7WgYwvN+9SjVH1yLyCP94gIkkSAt23 SfTHbktd1xSaqhGs2JJASbO8je50JkR9oSvgFBjiMo2nUuaizY+zzft1D8LUNZfRXtqV WYVwPByL4VZF845qjXWBjFKq0zm9xhutygxiLgCb72WAIOB2OnxR0lYMR29tI4oqSp8Y u5qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sjHLrc5tBk1BMC7N2Gqe2KeA/oPgnloD9SuKcolwBbw=; b=setohCj6WF9W6rLy6IuBZLT80PXlGxuJFPOrsMtnLJnRDpbtO8T3sD/YDiCYkRZ7W3 dpWnucXzHLOZUhBzKw08rsdwTtXXpC1rbnT+D2kvBOso7vYwAw2TWXp7nD30SwX0pUG3 tGx9nOSACbSSpgA50usaeszxvu0ugnhRwjRo7yQqBAJWlMecRpKDMgfFVGN/zvtBnr7P wfwaZ263aUIDYSMcYKE6jMY/usCDcQR/6FYG9NVoya96Q3+/Qhn/03tWZ/oBU4Bze/5w DlMxG4eJicRIywWgEK/1cJuXrOQWEIqHyII+UxjztgLQNTr8LSCL/1VVTCGWXexxdlFR OH4w== X-Gm-Message-State: APjAAAWbZuwV/BbQRPRhm02gCAOCT25kjyazQPwGmHCBHXdVjMw9lv9r ZGZP4rn6O1BB2/nt+BaPDAcMxww3voE= X-Google-Smtp-Source: APXvYqzRTaNE2Yi1qt3KeOol9sd+0WTBCzL1T1XzU4ZNwOskdrLzpCgMLPeJJWPfBnTN+xCrZcM62g== X-Received: by 2002:a50:ba57:: with SMTP id 23mr2187254eds.196.1565822242814; Wed, 14 Aug 2019 15:37:22 -0700 (PDT) Received: from localhost.localdomain.localdomain ([216.113.160.77]) by smtp.gmail.com with ESMTPSA id c15sm198167edf.37.2019.08.14.15.37.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Aug 2019 15:37:22 -0700 (PDT) From: Han Zhou X-Google-Original-From: Han Zhou To: dev@openvswitch.org Date: Wed, 14 Aug 2019 15:37:01 -0700 Message-Id: <1565822223-90171-2-git-send-email-hzhou8@ebay.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1565822223-90171-1-git-send-email-hzhou8@ebay.com> References: <1565822223-90171-1-git-send-email-hzhou8@ebay.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2 2/4] ovsdb-idl.c: Allows retry even when using a single remote. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Han Zhou When clustered mode is used, the client needs to retry connecting to new servers when certain failures happen. Today it is allowed to retry new connection only if multiple remotes are used, which prevents using LB VIP with clustered nodes. This patch makes sure the retry logic works when using LB VIP: although same IP is used for retrying, the LB can actually redirect the connection to a new node. Signed-off-by: Han Zhou --- lib/ovsdb-idl.c | 11 +++------- tests/ovsdb-cluster.at | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++ tests/test-ovsdb.c | 1 + 3 files changed, 61 insertions(+), 8 deletions(-) diff --git a/lib/ovsdb-idl.c b/lib/ovsdb-idl.c index 1a6a4af..190143f 100644 --- a/lib/ovsdb-idl.c +++ b/lib/ovsdb-idl.c @@ -657,12 +657,8 @@ ovsdb_idl_state_to_string(enum ovsdb_idl_state state) static void ovsdb_idl_retry_at(struct ovsdb_idl *idl, const char *where) { - if (idl->session && jsonrpc_session_get_n_remotes(idl->session) > 1) { - ovsdb_idl_force_reconnect(idl); - ovsdb_idl_transition_at(idl, IDL_S_RETRY, where); - } else { - ovsdb_idl_transition_at(idl, IDL_S_ERROR, where); - } + ovsdb_idl_force_reconnect(idl); + ovsdb_idl_transition_at(idl, IDL_S_RETRY, where); } static void @@ -1895,8 +1891,7 @@ ovsdb_idl_check_server_db(struct ovsdb_idl *idl) if (!database) { VLOG_INFO_RL(&rl, "%s: server does not have %s database", server_name, idl->data.class_->database); - } else if (!strcmp(database->model, "clustered") - && jsonrpc_session_get_n_remotes(idl->session) > 1) { + } else if (!strcmp(database->model, "clustered")) { uint64_t index = database->n_index ? *database->index : 0; if (!database->schema) { diff --git a/tests/ovsdb-cluster.at b/tests/ovsdb-cluster.at index 4701272..6a13843 100644 --- a/tests/ovsdb-cluster.at +++ b/tests/ovsdb-cluster.at @@ -63,6 +63,63 @@ m4_define([OVSDB_CHECK_EXECUTION], EXECUTION_EXAMPLES +AT_BANNER([OVSDB - disconnect from cluster]) + +# When a node is disconnected from the cluster, the IDL should disconnect +# and retry even if it uses a single remote, because the remote IP can be +# a VIP on a load-balance. +AT_SETUP([OVSDB cluster - disconnect from cluster, single remote]) +AT_KEYWORDS([ovsdb server negative unix cluster disconnect]) + +schema_name=`ovsdb-tool schema-name $abs_srcdir/idltest.ovsschema` +ordinal_schema > schema +AT_CHECK([ovsdb-tool '-vPATTERN:console:%c|%p|%m' create-cluster s1.db $abs_srcdir/idltest.ovsschema unix:s1.raft], [0], [], [stderr]) +cid=`ovsdb-tool db-cid s1.db` +schema_name=`ovsdb-tool schema-name $abs_srcdir/idltest.ovsschema` +for i in `seq 2 3`; do + AT_CHECK([ovsdb-tool join-cluster s$i.db $schema_name unix:s$i.raft unix:s1.raft]) +done + +on_exit 'kill `cat *.pid`' +for i in `seq 3`; do + AT_CHECK([ovsdb-server -v -vconsole:off -vsyslog:off --detach --no-chdir --log-file=s$i.log --pidfile=s$i.pid --unixctl=s$i --remote=punix:s$i.ovsdb s$i.db]) +done +for i in `seq 3`; do + AT_CHECK([ovsdb_client_wait unix:s$i.ovsdb $schema_name connected]) +done + +AT_CHECK([ovsdb-client transact unix:s1.ovsdb '[["idltest", + {"op": "insert", + "table": "simple", + "row": {"i": 1}}]]'], [0], [ignore], [ignore]) + +# Connect to a single remote s3. Use "wait" to trigger a non-op transaction so +# that test-ovsdb will not quit. + +on_exit 'pkill test-ovsdb' +test-ovsdb '-vPATTERN:console:test-ovsdb|%c|%m' -v -t10 idl unix:s3.ovsdb '[["idltest", + {"op": "wait", + "table": "simple", + "where": [["i", "==", 1]], + "columns": ["i"], + "until": "==", + "rows": [{"i": 1}]}]]' > test-ovsdb.log 2>&1 & + +OVS_WAIT_UNTIL([grep "000: i=1" test-ovsdb.log]) + +# Shutdown the other 2 servers so that s3 is disconnected from the cluster. +for i in 2 1; do + OVS_APP_EXIT_AND_WAIT_BY_TARGET([`pwd`/s$i], [s$i.pid]) +done + +# The test-ovsdb should detect the disconnect and retry. +OVS_WAIT_UNTIL([grep disconnect test-ovsdb.log]) + +OVS_APP_EXIT_AND_WAIT_BY_TARGET([`pwd`/s3], [s3.pid]) + +AT_CLEANUP + + OVS_START_SHELL_HELPERS # ovsdb_cluster_failure_test SCHEMA_FUNC OUTPUT TRANSACTION... ovsdb_cluster_failure_test () { diff --git a/tests/test-ovsdb.c b/tests/test-ovsdb.c index 187eb28..b1a4be3 100644 --- a/tests/test-ovsdb.c +++ b/tests/test-ovsdb.c @@ -2412,6 +2412,7 @@ do_idl(struct ovs_cmdl_context *ctx) track = ((struct test_ovsdb_pvt_context *)(ctx->pvt))->track; idl = ovsdb_idl_create(ctx->argv[1], &idltest_idl_class, true, true); + ovsdb_idl_set_leader_only(idl, false); if (ctx->argc > 2) { struct stream *stream;