From patchwork Fri Mar 15 23:17:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1057278 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tnQJ/a7S"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44LhMT0yJhz9s4V for ; Sat, 16 Mar 2019 10:18:13 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 99FBC1D2B; Fri, 15 Mar 2019 23:18:10 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id C8BD71CEF for ; Fri, 15 Mar 2019 23:18:09 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 368AE827 for ; Fri, 15 Mar 2019 23:18:09 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id y124so7331473pfy.7 for ; Fri, 15 Mar 2019 16:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=bTJce/LlRCaLr0UF4VBsr0pnmKB4lgwStJXtwHhhN0w=; b=tnQJ/a7Sfg/bhrlm2Cx5ne6iGQbtilLTzNGwOocTJGREYmM2tKEPD3GV0JBtWverHz Ya+NgCYzvV7rXwHJHrdOLZgfEr/De4vWq4nX/WcrtYZTr5nGjr1AXZy3pbyxVcwXFyfI xjuvmdw+RjzmBEBnqr/5LhKtMttWuL1AxUR8jCzyItKZ5t4ojblpc5ay3FZllQxA2slf 5DBzVWQJqpgkGdQTm0JVH55pxWEAt4VHCxi1pRFkSKCZmBtprEacKSiVsCrlfY/Tx0+3 aMjP+zp7sP47/SnuK9sTwViG96qdzK/Ieuh9cUM6mFEx4hBU3o8dGmn1g4njeCSYw3Ns sntg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=bTJce/LlRCaLr0UF4VBsr0pnmKB4lgwStJXtwHhhN0w=; b=fNwTFysaYZJEfRYC5HBmUvQdpqSLT3MeFqfpjW+cXavbwhCJ7wb/k4MZo9VY5Jyrnk y6zdHaCrpWKKj678aTcnZDILm6nEzW6Zhe9+DUZJ4m/bjQFzD9Og0wZLWzFbMQCEhHBu 7pO4w5mj/AWyn3mUuZeeNOgXCZTmdSfkUBGyDd9M2V0zYBY+X8aarynRC5jRBpMEdRaB sbpHlTs5BKqVVdhocJIKLjJJ/gzZCfgzs4m8DsVTgpouVpHegDKFnyj9O9ZTy6mWpBkG nZ4BMRkDn/oydZG9KfAUikdemQ3Za9nRhVb2qRiY8YdSRSLa7zgTitlWm6ptu4oUm7JE Ljdw== X-Gm-Message-State: APjAAAW26j//7ODTOpC15qg9RPonxf0yAiHDItSn5Yh4Ih4C0/UUZGUJ urAJi6HFAk9T3JoQR9xf2nng+ofd X-Google-Smtp-Source: APXvYqxdHty+y/ldtX276Z8l+2/1qOn+zGUYPPUCguaKVvLbx/bXXSft6pOS7PTLALto96BySLM9sg== X-Received: by 2002:a17:902:a604:: with SMTP id u4mr6629432plq.302.1552691888325; Fri, 15 Mar 2019 16:18:08 -0700 (PDT) Received: from localhost.localdomain.localdomain ([216.113.160.77]) by smtp.gmail.com with ESMTPSA id q207sm7071841pgq.88.2019.03.15.16.18.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Mar 2019 16:18:07 -0700 (PDT) From: Han Zhou X-Google-Original-From: Han Zhou To: dev@openvswitch.org Date: Fri, 15 Mar 2019 16:17:35 -0700 Message-Id: <1552691855-114951-1-git-send-email-hzhou8@ebay.com> X-Mailer: git-send-email 2.1.0 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2] ovsdb raft: Sync commit index to followers without delay. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Han Zhou When update is requested from follower, the leader sends AppendRequest to all followers and wait until AppendReply received from majority, and then it will update commit index - the new entry is regarded as committed in raft log. However, this commit will not be notified to followers (including the one initiated the request) until next heartbeat (ping timeout), if no other pending requests. This results in long latency for updates made through followers, especially when a batch of updates are requested through the same follower. $ time for i in `seq 1 100`; do ovn-nbctl ls-add ls$i; done real 0m34.154s user 0m0.083s sys 0m0.250s This patch solves the problem by sending heartbeat as soon as the commit index is updated in leader. It also avoids unnessary heartbeat by resetting the ping timer whenever AppendRequest is broadcasted. With this patch the performance is improved more than 50 times in same test: $ time for i in `seq 1 100`; do ovn-nbctl ls-add ls$i; done real 0m0.564s user 0m0.080s sys 0m0.199s Some sleep is added in torture test cases because of the improved performance, otherwise the tests will all be skipped. Signed-off-by: Han Zhou --- Notes: v1->v2: adjust torture test case so that it passes without overload CPU. ovsdb/raft.c | 43 +++++++++++++++++++++++++++++-------------- tests/ovsdb-cluster.at | 3 ++- 2 files changed, 31 insertions(+), 15 deletions(-) diff --git a/ovsdb/raft.c b/ovsdb/raft.c index eee4f33..31e9e72 100644 --- a/ovsdb/raft.c +++ b/ovsdb/raft.c @@ -320,7 +320,8 @@ static void raft_send_append_request(struct raft *, static void raft_become_leader(struct raft *); static void raft_become_follower(struct raft *); -static void raft_reset_timer(struct raft *); +static void raft_reset_election_timer(struct raft *); +static void raft_reset_ping_timer(struct raft *); static void raft_send_heartbeats(struct raft *); static void raft_start_election(struct raft *, bool leadership_transfer); static bool raft_truncate(struct raft *, uint64_t new_end); @@ -376,8 +377,8 @@ raft_alloc(void) hmap_init(&raft->add_servers); hmap_init(&raft->commands); - raft->ping_timeout = time_msec() + PING_TIME_MSEC; - raft_reset_timer(raft); + raft_reset_ping_timer(raft); + raft_reset_election_timer(raft); return raft; } @@ -865,7 +866,7 @@ raft_read_log(struct raft *raft) } static void -raft_reset_timer(struct raft *raft) +raft_reset_election_timer(struct raft *raft) { unsigned int duration = (ELECTION_BASE_MSEC + random_range(ELECTION_RANGE_MSEC)); @@ -874,6 +875,12 @@ raft_reset_timer(struct raft *raft) } static void +raft_reset_ping_timer(struct raft *raft) +{ + raft->ping_timeout = time_msec() + PING_TIME_MSEC; +} + +static void raft_add_conn(struct raft *raft, struct jsonrpc_session *js, const struct uuid *sid, bool incoming) { @@ -1603,7 +1610,7 @@ raft_start_election(struct raft *raft, bool leadership_transfer) VLOG_INFO("term %"PRIu64": starting election", raft->term); } } - raft_reset_timer(raft); + raft_reset_election_timer(raft); struct raft_server *peer; HMAP_FOR_EACH (peer, hmap_node, &raft->servers) { @@ -1782,8 +1789,8 @@ raft_run(struct raft *raft) raft_command_complete(raft, cmd, RAFT_CMD_TIMEOUT); } } + raft_reset_ping_timer(raft); } - raft->ping_timeout = time_msec() + PING_TIME_MSEC; } /* Do this only at the end; if we did it as soon as we set raft->left or @@ -1963,6 +1970,7 @@ raft_command_initiate(struct raft *raft, s->next_index++; } } + raft_reset_ping_timer(raft); return cmd; } @@ -2313,7 +2321,7 @@ raft_become_follower(struct raft *raft) } raft->role = RAFT_FOLLOWER; - raft_reset_timer(raft); + raft_reset_election_timer(raft); /* Notify clients about lost leadership. * @@ -2387,6 +2395,8 @@ raft_send_heartbeats(struct raft *raft) RAFT_CMD_INCOMPLETE, 0); } } + + raft_reset_ping_timer(raft); } /* Initializes the fields in 's' that represent the leader's view of the @@ -2413,7 +2423,7 @@ raft_become_leader(struct raft *raft) raft->role = RAFT_LEADER; raft->leader_sid = raft->sid; raft->election_timeout = LLONG_MAX; - raft->ping_timeout = time_msec() + PING_TIME_MSEC; + raft_reset_ping_timer(raft); struct raft_server *s; HMAP_FOR_EACH (s, hmap_node, &raft->servers) { @@ -2573,11 +2583,13 @@ raft_get_next_entry(struct raft *raft, struct uuid *eid) return data; } -static void +/* Updates commit index in raft log. If commit index is already up-to-date + * it does nothing and return false, otherwise, returns true. */ +static bool raft_update_commit_index(struct raft *raft, uint64_t new_commit_index) { if (new_commit_index <= raft->commit_index) { - return; + return false; } if (raft->role == RAFT_LEADER) { @@ -2610,6 +2622,7 @@ raft_update_commit_index(struct raft *raft, uint64_t new_commit_index) .commit_index = raft->commit_index, }; ignore(ovsdb_log_write_and_free(raft->log, raft_record_to_json(&r))); + return true; } /* This doesn't use rq->entries (but it does use rq->n_entries). */ @@ -2809,7 +2822,7 @@ raft_handle_append_request(struct raft *raft, "usurped leadership"); return; } - raft_reset_timer(raft); + raft_reset_election_timer(raft); /* First check for the common case, where the AppendEntries request is * entirely for indexes covered by 'log_start' ... 'log_end - 1', something @@ -3045,7 +3058,9 @@ raft_consider_updating_commit_index(struct raft *raft) } } } - raft_update_commit_index(raft, new_commit_index); + if (raft_update_commit_index(raft, new_commit_index)) { + raft_send_heartbeats(raft); + } } static void @@ -3274,7 +3289,7 @@ raft_handle_vote_request__(struct raft *raft, return false; } - raft_reset_timer(raft); + raft_reset_election_timer(raft); return true; } @@ -3697,7 +3712,7 @@ static bool raft_handle_install_snapshot_request__( struct raft *raft, const struct raft_install_snapshot_request *rq) { - raft_reset_timer(raft); + raft_reset_election_timer(raft); /* * Our behavior here depend on new_log_start in the snapshot compared to diff --git a/tests/ovsdb-cluster.at b/tests/ovsdb-cluster.at index c7f1e34..889fb7f 100644 --- a/tests/ovsdb-cluster.at +++ b/tests/ovsdb-cluster.at @@ -142,11 +142,12 @@ ovsdb|WARN|schema: changed 2 columns in 'OVN_Southbound' database from ephemeral echo "$i-$j exited with status $status" > $i-$j:$status fi rm $i-$j.running + sleep 0.5 done : > $i.done)& + sleep 0.3 done echo "...done" - sleep 2 echo "waiting for ovn-sbctl processes to exit..." # Use file instead of var because code inside "while" runs in a subshell.