From patchwork Wed Apr 10 01:21:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1083052 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="kq2K5y7X"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44f5xD5hwSz9sTW for ; Wed, 10 Apr 2019 11:22:24 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 751B31180; Wed, 10 Apr 2019 01:22:02 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 4B18D1152 for ; Wed, 10 Apr 2019 01:21:40 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id BC016F4 for ; Wed, 10 Apr 2019 01:21:39 +0000 (UTC) Received: by mail-pf1-f180.google.com with SMTP id t21so364591pfh.2 for ; Tue, 09 Apr 2019 18:21:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tyt6NNk9LWXgA9VHCUYGO1H6Lp3nb+haAOdmipB/vzA=; b=kq2K5y7XYovnAwrHT3SlkT/P8Wb3Er4RWgknBjow8cY9IjhDOvcK0TZj5G6Wq+M7JY zwQ0kuykKDfQzC/jRYAZ1vjKITLhopTMaSVm3wFlO9lcCf9mljkeXob/3aX8cIiCSEgd cOrCFMdLvGZ8Q126v5vBuqMncJgIHJEI+Vup/Z6sy9DwOlCMKdehuJEZjYw4BUTqqwNd CXhxKJtNv9zTxwdgoE6aPo/9sf5M/eu9VxBGGTRCO94TicCNlmD25CcOJMyR5JTmztZy ikhBKLD/KDHe1DItPCrCUco23mLSTj4km+74YdxVskPcMDNfdsgkNDbqpuDLseTH0LcR xhuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tyt6NNk9LWXgA9VHCUYGO1H6Lp3nb+haAOdmipB/vzA=; b=X+ma+pR4WecA6KcgenV3ezcDN8aPmZPEE5g9ZkW494Mc3mjqN/6qeRhJQ/oZSUw8Fr hnPS9lFAwTD4z4M3p3KDJNVZinzSXWQtPD+QOP+VU3x3GHuyUYYq8aWOyosnG4s2rpmK vOBS7JrQtNVGvZ2+mrS/4Yv/aX0vNWTovsXFpOqwX0KPvc2B5BzMnnKmWGyG6O6fL0b9 1WGmON5NufrBows+tszedBnBXNBukKPgCZPSg+kyerY55fO1+ABf4HW732dn0S9XKABX cNBPKsye4edAdVxlCTjQBhhMbnM7zD7vMYHbEDp8ReA4dM3ovoUHNgZw65hlWB+vMmup FOIQ== X-Gm-Message-State: APjAAAUsUFaide+QjtvZWcx+zkcbZNmBeGqR/74yThlWvS9fXlOZaCxD 2FfA8xopA8aSgB/pis08QRixPejz X-Google-Smtp-Source: APXvYqxrPvPXXBGjR5MDEYqhk5B8L62aGj2GPNnIV2DguaMPgHm2w3palOIsaenZHz2v4RfP+J0/og== X-Received: by 2002:a63:f448:: with SMTP id p8mr35604003pgk.50.1554859299075; Tue, 09 Apr 2019 18:21:39 -0700 (PDT) Received: from localhost.localdomain.localdomain ([216.113.160.71]) by smtp.gmail.com with ESMTPSA id u63sm17068802pgd.16.2019.04.09.18.21.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 18:21:38 -0700 (PDT) From: Han Zhou X-Google-Original-From: Han Zhou To: dev@openvswitch.org Date: Tue, 9 Apr 2019 18:21:17 -0700 Message-Id: <1554859282-15144-2-git-send-email-hzhou8@ebay.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1554859282-15144-1-git-send-email-hzhou8@ebay.com> References: <1554859282-15144-1-git-send-email-hzhou8@ebay.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 2/7] ovsdb raft: Support commands that are required for testing failure scenarios. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Han Zhou Added unix commands cluster/... for ovsdb raft, which will be used in a future patch to test more fine-grained failure scenarios. The commands either causes a node to crash at certain point, or manipulate the election timer so that we can control the election process to elect a new leader we desired for the test cases. Signed-off-by: Han Zhou --- ovsdb/raft.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/ovsdb/raft.c b/ovsdb/raft.c index 31e9e72..abcc9c0 100644 --- a/ovsdb/raft.c +++ b/ovsdb/raft.c @@ -63,6 +63,20 @@ enum raft_role { RAFT_LEADER }; +/* Flags for unit tests. */ +enum raft_failure_test { + FT_NO_TEST, + FT_CRASH_BEFORE_SEND_APPEND_REQ, + FT_CRASH_AFTER_SEND_APPEND_REQ, + FT_CRASH_BEFORE_SEND_EXEC_REP, + FT_CRASH_AFTER_SEND_EXEC_REP, + FT_CRASH_BEFORE_SEND_EXEC_REQ, + FT_CRASH_AFTER_SEND_EXEC_REQ, + FT_CRASH_AFTER_RECV_APPEND_REQ_UPDATE, + FT_DELAY_ELECTION +}; +static enum raft_failure_test failure_test; + /* A connection between this Raft server and another one. */ struct raft_conn { struct ovs_list list_node; /* In struct raft's 'conns' list. */ @@ -871,6 +885,10 @@ raft_reset_election_timer(struct raft *raft) unsigned int duration = (ELECTION_BASE_MSEC + random_range(ELECTION_RANGE_MSEC)); raft->election_base = time_msec(); + if (failure_test == FT_DELAY_ELECTION) { + /* Slow down this node so that it won't win the next election. */ + duration += ELECTION_BASE_MSEC; + } raft->election_timeout = raft->election_base + duration; } @@ -1962,6 +1980,9 @@ raft_command_initiate(struct raft *raft, raft_waiter_create(raft, RAFT_W_ENTRY, true)->entry.index = cmd->index; + if (failure_test == FT_CRASH_BEFORE_SEND_APPEND_REQ) { + ovs_fatal(0, "Raft test: crash before sending append_request."); + } /* Write to remote logs. */ struct raft_server *s; HMAP_FOR_EACH (s, hmap_node, &raft->servers) { @@ -1970,6 +1991,9 @@ raft_command_initiate(struct raft *raft, s->next_index++; } } + if (failure_test == FT_CRASH_AFTER_SEND_APPEND_REQ) { + ovs_fatal(0, "Raft test: crash after sending append_request."); + } raft_reset_ping_timer(raft); return cmd; @@ -2014,10 +2038,18 @@ raft_command_execute__(struct raft *raft, .result = eid, } }; + if (failure_test == FT_CRASH_BEFORE_SEND_EXEC_REQ) { + ovs_fatal(0, "Raft test: crash before sending " + "execute_command_request"); + } if (!raft_send(raft, &rpc)) { /* Couldn't send command, so it definitely failed. */ return raft_command_create_completed(RAFT_CMD_NOT_LEADER); } + if (failure_test == FT_CRASH_AFTER_SEND_EXEC_REQ) { + ovs_fatal(0, "Raft test: crash after sending " + "execute_command_request"); + } struct raft_command *cmd = raft_command_create_incomplete(raft, 0); cmd->timestamp = time_msec(); @@ -2731,6 +2763,10 @@ raft_handle_append_entries(struct raft *raft, } } + if (failure_test == FT_CRASH_AFTER_RECV_APPEND_REQ_UPDATE) { + ovs_fatal(0, "Raft test: crash after receiving append_request with " + "update."); + } /* Figure 3.1: "Append any entries not already in the log." */ struct ovsdb_error *error = NULL; bool any_written = false; @@ -3939,6 +3975,9 @@ raft_send_execute_command_reply(struct raft *raft, enum raft_command_status status, uint64_t commit_index) { + if (failure_test == FT_CRASH_BEFORE_SEND_EXEC_REP) { + ovs_fatal(0, "Raft test: crash before sending execute_command_reply"); + } union raft_rpc rpc = { .execute_command_reply = { .common = { @@ -3951,6 +3990,9 @@ raft_send_execute_command_reply(struct raft *raft, }, }; raft_send(raft, &rpc); + if (failure_test == FT_CRASH_AFTER_SEND_EXEC_REP) { + ovs_fatal(0, "Raft test: crash after sending execute_command_reply."); + } } static enum raft_command_status @@ -4387,6 +4429,45 @@ raft_unixctl_kick(struct unixctl_conn *conn, int argc OVS_UNUSED, } static void +raft_unixctl_failure_test(struct unixctl_conn *conn OVS_UNUSED, + int argc OVS_UNUSED, const char *argv[], + void *aux OVS_UNUSED) +{ + const char *test = argv[1]; + if (!strcmp(test, "crash-before-sending-append-request")) { + failure_test = FT_CRASH_BEFORE_SEND_APPEND_REQ; + } else if (!strcmp(test, "crash-after-sending-append-request")) { + failure_test = FT_CRASH_AFTER_SEND_APPEND_REQ; + } else if (!strcmp(test, "crash-before-sending-execute-command-reply")) { + failure_test = FT_CRASH_BEFORE_SEND_EXEC_REP; + } else if (!strcmp(test, "crash-after-sending-execute-command-reply")) { + failure_test = FT_CRASH_AFTER_SEND_EXEC_REP; + } else if (!strcmp(test, "crash-before-sending-execute-command-request")) { + failure_test = FT_CRASH_BEFORE_SEND_EXEC_REQ; + } else if (!strcmp(test, "crash-after-sending-execute-command-request")) { + failure_test = FT_CRASH_AFTER_SEND_EXEC_REQ; + } else if (!strcmp(test, "crash-after-receiving-append-request-update")) { + failure_test = FT_CRASH_AFTER_RECV_APPEND_REQ_UPDATE; + } else if (!strcmp(test, "delay-election")) { + failure_test = FT_DELAY_ELECTION; + struct raft *raft; + HMAP_FOR_EACH (raft, hmap_node, &all_rafts) { + if (raft->role == RAFT_FOLLOWER) { + raft_reset_election_timer(raft); + } + } + } else if (!strcmp(test, "clear")) { + failure_test = FT_NO_TEST; + unixctl_command_reply(conn, "test dismissed"); + return; + } else { + unixctl_command_reply_error(conn, "unknown test scenario"); + return; + } + unixctl_command_reply(conn, "test engaged"); +} + +static void raft_init(void) { static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; @@ -4403,5 +4484,7 @@ raft_init(void) raft_unixctl_leave, NULL); unixctl_command_register("cluster/kick", "DB SERVER", 2, 2, raft_unixctl_kick, NULL); + unixctl_command_register("cluster/failure-test", "FAILURE SCENARIO", 1, 1, + raft_unixctl_failure_test, NULL); ovsthread_once_done(&once); }