From patchwork Fri Apr 12 23:26:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1084988 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LDaDEaoR"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44gvFn55w4z9s6w for ; Sat, 13 Apr 2019 09:27:57 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id A05D918DD; Fri, 12 Apr 2019 23:26:30 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 3CC1518C4 for ; Fri, 12 Apr 2019 23:26:27 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 3972F76D for ; Fri, 12 Apr 2019 23:26:26 +0000 (UTC) Received: by mail-pf1-f181.google.com with SMTP id 188so5824264pfd.8 for ; Fri, 12 Apr 2019 16:26:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=tyt6NNk9LWXgA9VHCUYGO1H6Lp3nb+haAOdmipB/vzA=; b=LDaDEaoRE+eVwI5NFSn7I2PeXrP+SFn+ZWi9DuocSmq6xArIAdnZNXh9SBOg6xsMHj Iaw7R+JEfUznlEVTpquX0q0LQlrJoAZQcrfTcWzfxuDkrykXTM+FGhtgFRplTUEl3yq7 bljn/lPcszuB0Ljd0IxApGPzKH1shl1S8Je7qnqn2cZDtjcM+GuwMu8Nu4o7TzqW74K7 7+Q4C2c/XMwoPvdwTQNncyujHa2rvDWWSK+85jlZ7j3wZ2WDTOqB9akIAYVD4Uf4oELa HuwfRafyAVZuk3ldW7U4tQ2elGw7PEBa3FQDMEPv/ORAw45VqiQ981dICXaaYH3mt+NR a+SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=tyt6NNk9LWXgA9VHCUYGO1H6Lp3nb+haAOdmipB/vzA=; b=WiZ7ipFwCW5jRe0yrAfhzJriH3SGRWT704arJaOQBpUE4tu7z52D8J7kfouW129LZ1 7Gj4S54ehNcEFM0kzgkCGnX76c9QDj0eFpSqbIxaKk6NFScKIHzur0gyWcsoFsffEXzQ Q6qEHsFJ4Am5v3wxS3tqXbydfM2yCGA/Oh+5Tbvgc/q94s7VBODipgtvKaEiDcHOm2U6 R19SqDWVjyZkIxY7BBPXV34se6OCnoxsqZ/9bG6707xY8z/KXdUIETwktKl0xOmGeKqx MRV2SecrV/tjCuy9CNx4G61+JQcfCNO9FU17xikGj2W4tEyS1LqukhmRPSlWfQYq9gVe /+aQ== X-Gm-Message-State: APjAAAUo/1xCLa+IhwkX3fzwag6yBATbGlyJzEZS6cKGj/nFgDWvyi+5 JYVK9gWnN6jw1chMiS6lkk9P+QrM X-Google-Smtp-Source: APXvYqzXt0blSP5RurNPtX7A8iLcV2nh6YHoksGzur64Wj38JpBbUIS5tyzlFCK6pnRF9gYfaEu5zQ== X-Received: by 2002:aa7:8e0d:: with SMTP id c13mr23075248pfr.193.1555111585477; Fri, 12 Apr 2019 16:26:25 -0700 (PDT) Received: from localhost.localdomain.localdomain ([216.113.160.77]) by smtp.gmail.com with ESMTPSA id o68sm105570138pfi.140.2019.04.12.16.26.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Apr 2019 16:26:24 -0700 (PDT) From: Han Zhou X-Google-Original-From: Han Zhou To: dev@openvswitch.org Date: Fri, 12 Apr 2019 16:26:23 -0700 Message-Id: <1555111588-79659-4-git-send-email-hzhou8@ebay.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1555111588-79659-1-git-send-email-hzhou8@ebay.com> References: <1555111588-79659-1-git-send-email-hzhou8@ebay.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2 4/9] ovsdb raft: Support commands that are required for testing failure scenarios. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Han Zhou Added unix commands cluster/... for ovsdb raft, which will be used in a future patch to test more fine-grained failure scenarios. The commands either causes a node to crash at certain point, or manipulate the election timer so that we can control the election process to elect a new leader we desired for the test cases. Signed-off-by: Han Zhou --- ovsdb/raft.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/ovsdb/raft.c b/ovsdb/raft.c index 31e9e72..abcc9c0 100644 --- a/ovsdb/raft.c +++ b/ovsdb/raft.c @@ -63,6 +63,20 @@ enum raft_role { RAFT_LEADER }; +/* Flags for unit tests. */ +enum raft_failure_test { + FT_NO_TEST, + FT_CRASH_BEFORE_SEND_APPEND_REQ, + FT_CRASH_AFTER_SEND_APPEND_REQ, + FT_CRASH_BEFORE_SEND_EXEC_REP, + FT_CRASH_AFTER_SEND_EXEC_REP, + FT_CRASH_BEFORE_SEND_EXEC_REQ, + FT_CRASH_AFTER_SEND_EXEC_REQ, + FT_CRASH_AFTER_RECV_APPEND_REQ_UPDATE, + FT_DELAY_ELECTION +}; +static enum raft_failure_test failure_test; + /* A connection between this Raft server and another one. */ struct raft_conn { struct ovs_list list_node; /* In struct raft's 'conns' list. */ @@ -871,6 +885,10 @@ raft_reset_election_timer(struct raft *raft) unsigned int duration = (ELECTION_BASE_MSEC + random_range(ELECTION_RANGE_MSEC)); raft->election_base = time_msec(); + if (failure_test == FT_DELAY_ELECTION) { + /* Slow down this node so that it won't win the next election. */ + duration += ELECTION_BASE_MSEC; + } raft->election_timeout = raft->election_base + duration; } @@ -1962,6 +1980,9 @@ raft_command_initiate(struct raft *raft, raft_waiter_create(raft, RAFT_W_ENTRY, true)->entry.index = cmd->index; + if (failure_test == FT_CRASH_BEFORE_SEND_APPEND_REQ) { + ovs_fatal(0, "Raft test: crash before sending append_request."); + } /* Write to remote logs. */ struct raft_server *s; HMAP_FOR_EACH (s, hmap_node, &raft->servers) { @@ -1970,6 +1991,9 @@ raft_command_initiate(struct raft *raft, s->next_index++; } } + if (failure_test == FT_CRASH_AFTER_SEND_APPEND_REQ) { + ovs_fatal(0, "Raft test: crash after sending append_request."); + } raft_reset_ping_timer(raft); return cmd; @@ -2014,10 +2038,18 @@ raft_command_execute__(struct raft *raft, .result = eid, } }; + if (failure_test == FT_CRASH_BEFORE_SEND_EXEC_REQ) { + ovs_fatal(0, "Raft test: crash before sending " + "execute_command_request"); + } if (!raft_send(raft, &rpc)) { /* Couldn't send command, so it definitely failed. */ return raft_command_create_completed(RAFT_CMD_NOT_LEADER); } + if (failure_test == FT_CRASH_AFTER_SEND_EXEC_REQ) { + ovs_fatal(0, "Raft test: crash after sending " + "execute_command_request"); + } struct raft_command *cmd = raft_command_create_incomplete(raft, 0); cmd->timestamp = time_msec(); @@ -2731,6 +2763,10 @@ raft_handle_append_entries(struct raft *raft, } } + if (failure_test == FT_CRASH_AFTER_RECV_APPEND_REQ_UPDATE) { + ovs_fatal(0, "Raft test: crash after receiving append_request with " + "update."); + } /* Figure 3.1: "Append any entries not already in the log." */ struct ovsdb_error *error = NULL; bool any_written = false; @@ -3939,6 +3975,9 @@ raft_send_execute_command_reply(struct raft *raft, enum raft_command_status status, uint64_t commit_index) { + if (failure_test == FT_CRASH_BEFORE_SEND_EXEC_REP) { + ovs_fatal(0, "Raft test: crash before sending execute_command_reply"); + } union raft_rpc rpc = { .execute_command_reply = { .common = { @@ -3951,6 +3990,9 @@ raft_send_execute_command_reply(struct raft *raft, }, }; raft_send(raft, &rpc); + if (failure_test == FT_CRASH_AFTER_SEND_EXEC_REP) { + ovs_fatal(0, "Raft test: crash after sending execute_command_reply."); + } } static enum raft_command_status @@ -4387,6 +4429,45 @@ raft_unixctl_kick(struct unixctl_conn *conn, int argc OVS_UNUSED, } static void +raft_unixctl_failure_test(struct unixctl_conn *conn OVS_UNUSED, + int argc OVS_UNUSED, const char *argv[], + void *aux OVS_UNUSED) +{ + const char *test = argv[1]; + if (!strcmp(test, "crash-before-sending-append-request")) { + failure_test = FT_CRASH_BEFORE_SEND_APPEND_REQ; + } else if (!strcmp(test, "crash-after-sending-append-request")) { + failure_test = FT_CRASH_AFTER_SEND_APPEND_REQ; + } else if (!strcmp(test, "crash-before-sending-execute-command-reply")) { + failure_test = FT_CRASH_BEFORE_SEND_EXEC_REP; + } else if (!strcmp(test, "crash-after-sending-execute-command-reply")) { + failure_test = FT_CRASH_AFTER_SEND_EXEC_REP; + } else if (!strcmp(test, "crash-before-sending-execute-command-request")) { + failure_test = FT_CRASH_BEFORE_SEND_EXEC_REQ; + } else if (!strcmp(test, "crash-after-sending-execute-command-request")) { + failure_test = FT_CRASH_AFTER_SEND_EXEC_REQ; + } else if (!strcmp(test, "crash-after-receiving-append-request-update")) { + failure_test = FT_CRASH_AFTER_RECV_APPEND_REQ_UPDATE; + } else if (!strcmp(test, "delay-election")) { + failure_test = FT_DELAY_ELECTION; + struct raft *raft; + HMAP_FOR_EACH (raft, hmap_node, &all_rafts) { + if (raft->role == RAFT_FOLLOWER) { + raft_reset_election_timer(raft); + } + } + } else if (!strcmp(test, "clear")) { + failure_test = FT_NO_TEST; + unixctl_command_reply(conn, "test dismissed"); + return; + } else { + unixctl_command_reply_error(conn, "unknown test scenario"); + return; + } + unixctl_command_reply(conn, "test engaged"); +} + +static void raft_init(void) { static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; @@ -4403,5 +4484,7 @@ raft_init(void) raft_unixctl_leave, NULL); unixctl_command_register("cluster/kick", "DB SERVER", 2, 2, raft_unixctl_kick, NULL); + unixctl_command_register("cluster/failure-test", "FAILURE SCENARIO", 1, 1, + raft_unixctl_failure_test, NULL); ovsthread_once_done(&once); }