From patchwork Tue Jun 8 09:27:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Ivanov X-Patchwork-Id: 1489277 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Fzm8v09fMz9sSn for ; Tue, 8 Jun 2021 20:06:07 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 5B51140444; Tue, 8 Jun 2021 10:06:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DhW08jwvyMM4; Tue, 8 Jun 2021 10:06:00 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTP id B88254020E; Tue, 8 Jun 2021 10:05:59 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8F132C000D; Tue, 8 Jun 2021 10:05:59 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3E0DBC0001 for ; Tue, 8 Jun 2021 10:05:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 259F9400E6 for ; Tue, 8 Jun 2021 10:05:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mlHQdtLFBchD for ; Tue, 8 Jun 2021 10:05:54 +0000 (UTC) X-Greylist: delayed 00:38:22 by SQLgrey-1.8.0 Received: from www.kot-begemot.co.uk (ivanoab7.miniserver.com [37.128.132.42]) by smtp2.osuosl.org (Postfix) with ESMTPS id E0B8440291 for ; Tue, 8 Jun 2021 10:05:53 +0000 (UTC) Received: from tun252.jain.kot-begemot.co.uk ([192.168.18.6] helo=jain.kot-begemot.co.uk) by www.kot-begemot.co.uk with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lqY15-0006aS-Nl for ovs-dev@openvswitch.org; Tue, 08 Jun 2021 09:27:28 +0000 Received: from jain.kot-begemot.co.uk ([192.168.3.3]) by jain.kot-begemot.co.uk with esmtp (Exim 4.92) (envelope-from ) id 1lqY12-00046P-O6; Tue, 08 Jun 2021 10:27:22 +0100 From: anton.ivanov@cambridgegreys.com To: ovs-dev@openvswitch.org Date: Tue, 8 Jun 2021 10:27:08 +0100 Message-Id: <20210608092708.15711-1-anton.ivanov@cambridgegreys.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-Clacks-Overhead: GNU Terry Pratchett Cc: Anton Ivanov Subject: [ovs-dev] [PATCH] ovsdb: provide raft and command interfaces with priority X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Anton Ivanov Set a soft time limit of "raft election timer"/2 on ovsdb processing. This improves behaviour in large heavily loaded clusters. While it cannot fully eliminate spurious raft elections under heavy load, it significantly decreases their number. TODO: randomize session processing order to ensure individual sessions towards the end of the remotes list are not starved Signed-off-by: Anton Ivanov --- ovsdb/jsonrpc-server.c | 43 ++++++++++++++++++++++++++++++++++++++---- ovsdb/jsonrpc-server.h | 2 +- ovsdb/ovsdb-server.c | 15 ++++++++++++++- ovsdb/raft.c | 5 +++++ ovsdb/raft.h | 3 +++ ovsdb/storage.c | 8 ++++++++ ovsdb/storage.h | 2 ++ 7 files changed, 72 insertions(+), 6 deletions(-) diff --git a/ovsdb/jsonrpc-server.c b/ovsdb/jsonrpc-server.c index 4e2dfc3d7..18b8a5deb 100644 --- a/ovsdb/jsonrpc-server.c +++ b/ovsdb/jsonrpc-server.c @@ -60,7 +60,8 @@ static struct ovsdb_jsonrpc_session *ovsdb_jsonrpc_session_create( struct ovsdb_jsonrpc_remote *, struct jsonrpc_session *, bool); static void ovsdb_jsonrpc_session_preremove_db(struct ovsdb_jsonrpc_remote *, struct ovsdb *); -static void ovsdb_jsonrpc_session_run_all(struct ovsdb_jsonrpc_remote *); +static void ovsdb_jsonrpc_session_run_all(struct ovsdb_jsonrpc_remote *, + uint64_t limit); static void ovsdb_jsonrpc_session_wait_all(struct ovsdb_jsonrpc_remote *); static void ovsdb_jsonrpc_session_get_memory_usage_all( const struct ovsdb_jsonrpc_remote *, struct simap *usage); @@ -128,6 +129,7 @@ struct ovsdb_jsonrpc_server { bool read_only; /* This server is does not accept any transactions that can modify the database. */ struct shash remotes; /* Contains "struct ovsdb_jsonrpc_remote *"s. */ + bool yield_immediately; }; /* A configured remote. This is either a passive stream listener plus a list @@ -159,6 +161,7 @@ ovsdb_jsonrpc_server_create(bool read_only) ovsdb_server_init(&server->up); shash_init(&server->remotes); server->read_only = read_only; + server->yield_immediately = false; return server; } @@ -378,9 +381,16 @@ ovsdb_jsonrpc_server_set_read_only(struct ovsdb_jsonrpc_server *svr, } void -ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *svr) +ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *svr, uint64_t limit) { struct shash_node *node; + uint64_t elapsed = 0, start_time = 0; + + if (limit) { + start_time = time_now(); + } + + svr->yield_immediately = false; SHASH_FOR_EACH (node, &svr->remotes) { struct ovsdb_jsonrpc_remote *remote = node->data; @@ -403,7 +413,15 @@ ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *svr) } } - ovsdb_jsonrpc_session_run_all(remote); + ovsdb_jsonrpc_session_run_all(remote, limit - elapsed); + + if (limit) { + elapsed = start_time - time_now(); + if (elapsed >= limit) { + svr->yield_immediately = true; + break; + } + } } } @@ -412,6 +430,12 @@ ovsdb_jsonrpc_server_wait(struct ovsdb_jsonrpc_server *svr) { struct shash_node *node; + if (svr->yield_immediately) { + poll_immediate_wake(); + svr->yield_immediately = false; + return; + } + SHASH_FOR_EACH (node, &svr->remotes) { struct ovsdb_jsonrpc_remote *remote = node->data; @@ -583,15 +607,26 @@ ovsdb_jsonrpc_session_set_options(struct ovsdb_jsonrpc_session *session, } static void -ovsdb_jsonrpc_session_run_all(struct ovsdb_jsonrpc_remote *remote) +ovsdb_jsonrpc_session_run_all(struct ovsdb_jsonrpc_remote *remote, + uint64_t limit) { struct ovsdb_jsonrpc_session *s, *next; + uint64_t start_time; + + if (limit) { + start_time = time_msec(); + } LIST_FOR_EACH_SAFE (s, next, node, &remote->sessions) { int error = ovsdb_jsonrpc_session_run(s); if (error) { ovsdb_jsonrpc_session_close(s); } + if (limit) { + if (time_msec() - start_time > limit) { + break; + } + } } } diff --git a/ovsdb/jsonrpc-server.h b/ovsdb/jsonrpc-server.h index e0653aa39..218152e9d 100644 --- a/ovsdb/jsonrpc-server.h +++ b/ovsdb/jsonrpc-server.h @@ -67,7 +67,7 @@ void ovsdb_jsonrpc_server_free_remote_status( void ovsdb_jsonrpc_server_reconnect(struct ovsdb_jsonrpc_server *, bool force, char *comment); -void ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *); +void ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *, uint64_t limit); void ovsdb_jsonrpc_server_wait(struct ovsdb_jsonrpc_server *); void ovsdb_jsonrpc_server_set_read_only(struct ovsdb_jsonrpc_server *, diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index b09232c65..97bfa05b2 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -184,6 +184,7 @@ main_loop(struct server_config *config, char *remotes_error, *ssl_error; struct shash_node *node; long long int status_timer = LLONG_MIN; + uint64_t limit = 0; *exiting = false; ssl_error = NULL; @@ -215,7 +216,7 @@ main_loop(struct server_config *config, reconfigure_remotes(jsonrpc, all_dbs, remotes), &remotes_error); report_error_if_changed(reconfigure_ssl(all_dbs), &ssl_error); - ovsdb_jsonrpc_server_run(jsonrpc); + ovsdb_jsonrpc_server_run(jsonrpc, limit); if (*is_backup) { replication_run(); @@ -228,8 +229,20 @@ main_loop(struct server_config *config, struct shash_node *next; SHASH_FOR_EACH_SAFE (node, next, all_dbs) { struct db *db = node->data; + uint64_t db_limit = 0; + ovsdb_txn_history_run(db->db); ovsdb_storage_run(db->db->storage); + + db_limit = max_processing_time(db->db->storage); + if (db_limit) { + if (!limit) { + limit = db_limit; + } else { + limit = MIN(db_limit, limit); + } + } + read_db(config, db); /* Run triggers after storage_run and read_db to make sure new raft * updates are utilized in current iteration. */ diff --git a/ovsdb/raft.c b/ovsdb/raft.c index e06c1f1ab..c473e5d32 100644 --- a/ovsdb/raft.c +++ b/ovsdb/raft.c @@ -407,6 +407,11 @@ raft_make_address_passive(const char *address_) } } +uint64_t raft_election_timer_value(const struct raft *raft) +{ + return raft->election_timer; +} + static struct raft * raft_alloc(void) { diff --git a/ovsdb/raft.h b/ovsdb/raft.h index 3545c41c2..1d270ec0c 100644 --- a/ovsdb/raft.h +++ b/ovsdb/raft.h @@ -188,4 +188,7 @@ void raft_take_leadership(struct raft *); void raft_transfer_leadership(struct raft *, const char *reason); const struct uuid *raft_current_eid(const struct raft *); + +uint64_t raft_election_timer_value(const struct raft *); + #endif /* lib/raft.h */ diff --git a/ovsdb/storage.c b/ovsdb/storage.c index 40415fcf6..a2142851a 100644 --- a/ovsdb/storage.c +++ b/ovsdb/storage.c @@ -640,3 +640,11 @@ ovsdb_storage_peek_last_eid(struct ovsdb_storage *storage) } return raft_current_eid(storage->raft); } + +uint64_t max_processing_time(struct ovsdb_storage *storage) +{ + if (!storage->raft) { + return UINT64_MAX; + } + return raft_election_timer_value(storage->raft) / 2; +} diff --git a/ovsdb/storage.h b/ovsdb/storage.h index 02b6e7e6c..9e195bbe8 100644 --- a/ovsdb/storage.h +++ b/ovsdb/storage.h @@ -97,4 +97,6 @@ struct ovsdb_schema *ovsdb_storage_read_schema(struct ovsdb_storage *); const struct uuid *ovsdb_storage_peek_last_eid(struct ovsdb_storage *); +uint64_t max_processing_time(struct ovsdb_storage *); + #endif /* ovsdb/storage.h */