From patchwork Mon Aug 19 16:29:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1149390 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="swgoFNWD"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Bzv00vYWz9sNf for ; Tue, 20 Aug 2019 02:30:52 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 1A851CB6; Mon, 19 Aug 2019 16:30:13 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 1FFE9CB2 for ; Mon, 19 Aug 2019 16:30:12 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 9AFD367F for ; Mon, 19 Aug 2019 16:30:11 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id i18so1495949pgl.11 for ; Mon, 19 Aug 2019 09:30:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vc89oLI1/z4YSXG5Pr7LitjDxyhczvHgn618qgDuoQs=; b=swgoFNWDs6UPzfGg4Kp9/pKRRgpZtIXFKkpbhGbjsuKSyrjJ6w0ozs9zM5tZFLlpH5 4VcOUFYhhW3m7NnLMLzTcgZBERD4kcHJIL6UoBKtT3FvgGJIB8JXSvTu661M+JcCYZRF u6TLlV6thok7jEk+1n4VjVMp6aIOskd5VT3PMg8tuE09zL/WGubECnojNRT1PZc43RWN DSjWEkFX9UaNIfLCNb+edPb9QwCibrJmXoQFqA2Q/YW/1ve/x2NJN5zZu014kx/BiAxK /7YBdFk+af0zptT6pzfsCSveg0mGu3fI6pUX/HnSd193ZzYgtVyurMhf4wkkUCBeLRSR th1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vc89oLI1/z4YSXG5Pr7LitjDxyhczvHgn618qgDuoQs=; b=qpMUW18j0gRsvOgvgEcVw1EW2YVJF6OKvsCozXZ1GIw/pCkLSYzwbR0TXt88MTzD2K krWKwZO7tLui9Tm8eLRsFQxBx/lkDrKjsIWSuQw8SBo3gzBqORxMfP8y4wBc4GzkDzT4 ogMeNyfiL/L1sTw8HKpz1QFwb33ijVNeRKBtxt3rKaVspposjDdwZzHQEQ0Vlp+HxPP+ 6o65flpWN6q6q8+X1VT5a8vIRLVSJjJRCpBPJSnPOY7R/eq1anGf8npetrnvw58xW0kr WZiluYi9X6SjqXz9WnH/Sjr8Hx+mYLay2J293bEv+PFDhfQe0krwWbojGOG+h/erQqAn ri6g== X-Gm-Message-State: APjAAAUnfWUhJ6C2kR7OeTN9U6e1f8XsBx0bcYxWeNAPgMLhqUmAcnV7 rBttoYCUCCwPSKHrJEHaKw+h81nJ+ss= X-Google-Smtp-Source: APXvYqxTc8qUB2Asp1H2bXLqjSgFoTfAXGLNdXl7cuHInCJ5Zs4pZ/5d5USTarzMPh8a3SRh6xk7lg== X-Received: by 2002:aa7:9edc:: with SMTP id r28mr24314986pfq.219.1566232210350; Mon, 19 Aug 2019 09:30:10 -0700 (PDT) Received: from localhost.localdomain.localdomain ([73.241.94.255]) by smtp.gmail.com with ESMTPSA id a5sm12285418pjs.31.2019.08.19.09.30.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Aug 2019 09:30:08 -0700 (PDT) From: Han Zhou X-Google-Original-From: Han Zhou To: dev@openvswitch.org Date: Mon, 19 Aug 2019 09:29:57 -0700 Message-Id: <1566232200-35099-2-git-send-email-hzhou8@ebay.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1566232200-35099-1-git-send-email-hzhou8@ebay.com> References: <1566232200-35099-1-git-send-email-hzhou8@ebay.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v3 2/5] ovsdb-idl.c: Allows retry even when using a single remote. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Han Zhou When clustered mode is used, the client needs to retry connecting to new servers when certain failures happen. Today it is allowed to retry new connection only if multiple remotes are used, which prevents using LB VIP with clustered nodes. This patch makes sure the retry logic works when using LB VIP: although same IP is used for retrying, the LB can actually redirect the connection to a new node. Signed-off-by: Han Zhou --- v2 -> v3: Minor fix for test case. lib/ovsdb-idl.c | 11 +++------- tests/ovsdb-cluster.at | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++ tests/test-ovsdb.c | 1 + 3 files changed, 61 insertions(+), 8 deletions(-) diff --git a/lib/ovsdb-idl.c b/lib/ovsdb-idl.c index 1a6a4af..190143f 100644 --- a/lib/ovsdb-idl.c +++ b/lib/ovsdb-idl.c @@ -657,12 +657,8 @@ ovsdb_idl_state_to_string(enum ovsdb_idl_state state) static void ovsdb_idl_retry_at(struct ovsdb_idl *idl, const char *where) { - if (idl->session && jsonrpc_session_get_n_remotes(idl->session) > 1) { - ovsdb_idl_force_reconnect(idl); - ovsdb_idl_transition_at(idl, IDL_S_RETRY, where); - } else { - ovsdb_idl_transition_at(idl, IDL_S_ERROR, where); - } + ovsdb_idl_force_reconnect(idl); + ovsdb_idl_transition_at(idl, IDL_S_RETRY, where); } static void @@ -1895,8 +1891,7 @@ ovsdb_idl_check_server_db(struct ovsdb_idl *idl) if (!database) { VLOG_INFO_RL(&rl, "%s: server does not have %s database", server_name, idl->data.class_->database); - } else if (!strcmp(database->model, "clustered") - && jsonrpc_session_get_n_remotes(idl->session) > 1) { + } else if (!strcmp(database->model, "clustered")) { uint64_t index = database->n_index ? *database->index : 0; if (!database->schema) { diff --git a/tests/ovsdb-cluster.at b/tests/ovsdb-cluster.at index 4701272..02e9926 100644 --- a/tests/ovsdb-cluster.at +++ b/tests/ovsdb-cluster.at @@ -63,6 +63,63 @@ m4_define([OVSDB_CHECK_EXECUTION], EXECUTION_EXAMPLES +AT_BANNER([OVSDB - disconnect from cluster]) + +# When a node is disconnected from the cluster, the IDL should disconnect +# and retry even if it uses a single remote, because the remote IP can be +# a VIP on a load-balance. +AT_SETUP([OVSDB cluster - disconnect from cluster, single remote]) +AT_KEYWORDS([ovsdb server negative unix cluster disconnect]) + +schema_name=`ovsdb-tool schema-name $abs_srcdir/idltest.ovsschema` +ordinal_schema > schema +AT_CHECK([ovsdb-tool '-vPATTERN:console:%c|%p|%m' create-cluster s1.db $abs_srcdir/idltest.ovsschema unix:s1.raft], [0], [], [stderr]) +cid=`ovsdb-tool db-cid s1.db` +schema_name=`ovsdb-tool schema-name $abs_srcdir/idltest.ovsschema` +for i in `seq 2 3`; do + AT_CHECK([ovsdb-tool join-cluster s$i.db $schema_name unix:s$i.raft unix:s1.raft]) +done + +on_exit 'kill `cat *.pid`' +for i in `seq 3`; do + AT_CHECK([ovsdb-server -v -vconsole:off -vsyslog:off --detach --no-chdir --log-file=s$i.log --pidfile=s$i.pid --unixctl=s$i --remote=punix:s$i.ovsdb s$i.db]) +done +for i in `seq 3`; do + AT_CHECK([ovsdb_client_wait unix:s$i.ovsdb $schema_name connected]) +done + +AT_CHECK([ovsdb-client transact unix:s1.ovsdb '[["idltest", + {"op": "insert", + "table": "simple", + "row": {"i": 1}}]]'], [0], [ignore], [ignore]) + +# Connect to a single remote s3. Use "wait" to trigger a non-op transaction so +# that test-ovsdb will not quit. + +test-ovsdb '-vPATTERN:console:test-ovsdb|%c|%m' -v -t10 idl unix:s3.ovsdb '[["idltest", + {"op": "wait", + "table": "simple", + "where": [["i", "==", 1]], + "columns": ["i"], + "until": "==", + "rows": [{"i": 1}]}]]' > test-ovsdb.log 2>&1 & +echo $! > test-ovsdb.pid + +OVS_WAIT_UNTIL([grep "000: i=1" test-ovsdb.log]) + +# Shutdown the other 2 servers so that s3 is disconnected from the cluster. +for i in 2 1; do + OVS_APP_EXIT_AND_WAIT_BY_TARGET([`pwd`/s$i], [s$i.pid]) +done + +# The test-ovsdb should detect the disconnect and retry. +OVS_WAIT_UNTIL([grep disconnect test-ovsdb.log]) + +OVS_APP_EXIT_AND_WAIT_BY_TARGET([`pwd`/s3], [s3.pid]) + +AT_CLEANUP + + OVS_START_SHELL_HELPERS # ovsdb_cluster_failure_test SCHEMA_FUNC OUTPUT TRANSACTION... ovsdb_cluster_failure_test () { diff --git a/tests/test-ovsdb.c b/tests/test-ovsdb.c index 187eb28..b1a4be3 100644 --- a/tests/test-ovsdb.c +++ b/tests/test-ovsdb.c @@ -2412,6 +2412,7 @@ do_idl(struct ovs_cmdl_context *ctx) track = ((struct test_ovsdb_pvt_context *)(ctx->pvt))->track; idl = ovsdb_idl_create(ctx->argv[1], &idltest_idl_class, true, true); + ovsdb_idl_set_leader_only(idl, false); if (ctx->argc > 2) { struct stream *stream;