From patchwork Sat Jun 12 02:00:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491223 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21Bj0Tz7z9sVm for ; Sat, 12 Jun 2021 12:00:28 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 67198607BB; Sat, 12 Jun 2021 02:00:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B9_9vB-D2qYB; Sat, 12 Jun 2021 02:00:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 9045160658; Sat, 12 Jun 2021 02:00:22 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B0610C0026; Sat, 12 Jun 2021 02:00:20 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 40647C000B for ; Sat, 12 Jun 2021 02:00:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2331340233 for ; Sat, 12 Jun 2021 02:00:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9xnofA0b5dg2 for ; Sat, 12 Jun 2021 02:00:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp2.osuosl.org (Postfix) with ESMTPS id 694CA40103 for ; Sat, 12 Jun 2021 02:00:18 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id D34F41BF206; Sat, 12 Jun 2021 02:00:15 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:00 +0200 Message-Id: <20210612020008.3944088-2-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 1/9] jsonrpc-server: Wake up jsonrpc session if there are completed triggers. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" If there are completed triggers, jsonrpc server should wake up and update clients with the new data, but there is no such condition in ovsdb_jsonrpc_session_wait(). For some reason this doesn't result in any processing delays in current code, probably because there are always some other types of events in this case that could wake ovsdb server up. But it will become a problem in upcoming ovsdb 'relay' service model because triggers could be completed from a different place, i.e. after receiving transaction reply from the relay source. Fix that by waking up ovsdb-server in case there are completed triggers that needs to be handled. Signed-off-by: Ilya Maximets Acked-by: Mark D. Gray Acked-by: Dumitru Ceara --- ovsdb/jsonrpc-server.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ovsdb/jsonrpc-server.c b/ovsdb/jsonrpc-server.c index 4e2dfc3d7..351c39d8a 100644 --- a/ovsdb/jsonrpc-server.c +++ b/ovsdb/jsonrpc-server.c @@ -600,7 +600,8 @@ ovsdb_jsonrpc_session_wait(struct ovsdb_jsonrpc_session *s) { jsonrpc_session_wait(s->js); if (!jsonrpc_session_get_backlog(s->js)) { - if (ovsdb_jsonrpc_monitor_needs_flush(s)) { + if (ovsdb_jsonrpc_monitor_needs_flush(s) + || !ovs_list_is_empty(&s->up.completions)) { poll_immediate_wake(); } else { jsonrpc_session_recv_wait(s->js); From patchwork Sat Jun 12 02:00:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491224 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21Bj18h4z9sWD for ; Sat, 12 Jun 2021 12:00:28 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D80C540264; Sat, 12 Jun 2021 02:00:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iC143j6geava; Sat, 12 Jun 2021 02:00:25 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id E4ED840406; Sat, 12 Jun 2021 02:00:23 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B4E06C0011; Sat, 12 Jun 2021 02:00:23 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 979AFC0011 for ; Sat, 12 Jun 2021 02:00:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 67F1340263 for ; Sat, 12 Jun 2021 02:00:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sM3CHjBt8qYB for ; Sat, 12 Jun 2021 02:00:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp2.osuosl.org (Postfix) with ESMTPS id 41F6840103 for ; Sat, 12 Jun 2021 02:00:21 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id CF4041BF203; Sat, 12 Jun 2021 02:00:18 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:01 +0200 Message-Id: <20210612020008.3944088-3-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 2/9] ovsdb: storage: Allow setting the name for the unbacked storage. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" ovsdb_create() requires schema or storage to be nonnull, but in practice it requires to have schema name or a storage name to use it as a database name. Only clustered storage has a name. This means that only clustered database can be created without schema, Changing that by allowing unbacked storage to have a name. This way we can create database with unbacked storage without schema. Will be used in next commits to create database for ovsdb 'relay' service model. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- ovsdb/file.c | 2 +- ovsdb/ovsdb-server.c | 2 +- ovsdb/storage.c | 13 ++++++++++--- ovsdb/storage.h | 2 +- tests/test-ovsdb.c | 6 +++--- 5 files changed, 16 insertions(+), 9 deletions(-) diff --git a/ovsdb/file.c b/ovsdb/file.c index 0b8bdfe37..59220824f 100644 --- a/ovsdb/file.c +++ b/ovsdb/file.c @@ -318,7 +318,7 @@ ovsdb_convert(const struct ovsdb *src, const struct ovsdb_schema *new_schema, struct ovsdb **dstp) { struct ovsdb *dst = ovsdb_create(ovsdb_schema_clone(new_schema), - ovsdb_storage_create_unbacked()); + ovsdb_storage_create_unbacked(NULL)); struct ovsdb_txn *txn = ovsdb_txn_create(dst); struct ovsdb_error *error = NULL; diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index b09232c65..23bd226a3 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -738,7 +738,7 @@ add_server_db(struct server_config *config) /* We don't need txn_history for server_db. */ db->filename = xstrdup(""); - db->db = ovsdb_create(schema, ovsdb_storage_create_unbacked()); + db->db = ovsdb_create(schema, ovsdb_storage_create_unbacked(NULL)); bool ok OVS_UNUSED = ovsdb_jsonrpc_server_add_db(config->jsonrpc, db->db); ovs_assert(ok); add_db(config, db); diff --git a/ovsdb/storage.c b/ovsdb/storage.c index 40415fcf6..d727b1eac 100644 --- a/ovsdb/storage.c +++ b/ovsdb/storage.c @@ -45,6 +45,8 @@ struct ovsdb_storage { struct ovsdb_log *log; struct raft *raft; + char *unbacked_name; /* Name of the unbacked storage. */ + /* All kinds of storage. */ struct ovsdb_error *error; /* If nonnull, a permanent error. */ long long next_snapshot_min; /* Earliest time to take next snapshot. */ @@ -121,12 +123,14 @@ ovsdb_storage_open_standalone(const char *filename, bool rw) } /* Creates and returns new storage without any backing. Nothing will be read - * from the storage, and writes are discarded. */ + * from the storage, and writes are discarded. If 'name' is nonnull, it will + * be used as a storage name. */ struct ovsdb_storage * -ovsdb_storage_create_unbacked(void) +ovsdb_storage_create_unbacked(const char *name) { struct ovsdb_storage *storage = xzalloc(sizeof *storage); schedule_next_snapshot(storage, false); + storage->unbacked_name = nullable_xstrdup(name); return storage; } @@ -137,6 +141,7 @@ ovsdb_storage_close(struct ovsdb_storage *storage) ovsdb_log_close(storage->log); raft_close(storage->raft); ovsdb_error_destroy(storage->error); + free(storage->unbacked_name); free(storage); } } @@ -230,7 +235,9 @@ ovsdb_storage_wait(struct ovsdb_storage *storage) const char * ovsdb_storage_get_name(const struct ovsdb_storage *storage) { - return storage->raft ? raft_get_name(storage->raft) : NULL; + return storage->unbacked_name ? storage->unbacked_name + : storage->raft ? raft_get_name(storage->raft) + : NULL; } /* Attempts to read a log record from 'storage'. diff --git a/ovsdb/storage.h b/ovsdb/storage.h index 02b6e7e6c..e120094d7 100644 --- a/ovsdb/storage.h +++ b/ovsdb/storage.h @@ -29,7 +29,7 @@ struct uuid; struct ovsdb_error *ovsdb_storage_open(const char *filename, bool rw, struct ovsdb_storage **) OVS_WARN_UNUSED_RESULT; -struct ovsdb_storage *ovsdb_storage_create_unbacked(void); +struct ovsdb_storage *ovsdb_storage_create_unbacked(const char *name); void ovsdb_storage_close(struct ovsdb_storage *); const char *ovsdb_storage_get_model(const struct ovsdb_storage *); diff --git a/tests/test-ovsdb.c b/tests/test-ovsdb.c index a886f971e..fb6b3acca 100644 --- a/tests/test-ovsdb.c +++ b/tests/test-ovsdb.c @@ -1485,7 +1485,7 @@ do_execute__(struct ovs_cmdl_context *ctx, bool ro) json = parse_json(ctx->argv[1]); check_ovsdb_error(ovsdb_schema_from_json(json, &schema)); json_destroy(json); - db = ovsdb_create(schema, ovsdb_storage_create_unbacked()); + db = ovsdb_create(schema, ovsdb_storage_create_unbacked(NULL)); for (i = 2; i < ctx->argc; i++) { struct json *params, *result; @@ -1551,7 +1551,7 @@ do_trigger(struct ovs_cmdl_context *ctx) json = parse_json(ctx->argv[1]); check_ovsdb_error(ovsdb_schema_from_json(json, &schema)); json_destroy(json); - db = ovsdb_create(schema, ovsdb_storage_create_unbacked()); + db = ovsdb_create(schema, ovsdb_storage_create_unbacked(NULL)); ovsdb_server_init(&server); ovsdb_server_add_db(&server, db); @@ -1781,7 +1781,7 @@ do_transact(struct ovs_cmdl_context *ctx) " \"j\": {\"type\": \"integer\"}}}}}"); check_ovsdb_error(ovsdb_schema_from_json(json, &schema)); json_destroy(json); - do_transact_db = ovsdb_create(schema, ovsdb_storage_create_unbacked()); + do_transact_db = ovsdb_create(schema, ovsdb_storage_create_unbacked(NULL)); do_transact_table = ovsdb_get_table(do_transact_db, "mytable"); ovs_assert(do_transact_table != NULL); From patchwork Sat Jun 12 02:00:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491226 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21Bn03Tzz9sVm for ; Sat, 12 Jun 2021 12:00:32 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id B924D414A3; Sat, 12 Jun 2021 02:00:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K-sdFddgMOd9; Sat, 12 Jun 2021 02:00:29 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 739A141492; Sat, 12 Jun 2021 02:00:28 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B486BC002B; Sat, 12 Jun 2021 02:00:26 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 49833C000E for ; Sat, 12 Jun 2021 02:00:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 331F983EC6 for ; Sat, 12 Jun 2021 02:00:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ATCnKp5OVcuj for ; Sat, 12 Jun 2021 02:00:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp1.osuosl.org (Postfix) with ESMTPS id C031D83E92 for ; Sat, 12 Jun 2021 02:00:23 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 2552C1BF207; Sat, 12 Jun 2021 02:00:20 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:02 +0200 Message-Id: <20210612020008.3944088-4-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 3/9] ovsdb: table: Expose functions to execute operations on ovsdb tables. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" These functions will be used later for ovsdb 'relay' service model, so moving them to a common code. Warnings translated to ovsdb errors, caller in replication.c only printed inconsistency warnings, but mostly ignored them. Implementing the same logic by checking the error tag. Also ovsdb_execute_insert() previously printed incorrect warning about duplicate row while it was a syntax error in json. Fixing that by actually checking for the duplicate and reporting correct ovsdb error. Signed-off-by: Ilya Maximets Acked-by: Mark D. Gray Acked-by: Dumitru Ceara --- ovsdb/replication.c | 83 ++++----------------------------------------- ovsdb/table.c | 69 +++++++++++++++++++++++++++++++++++++ ovsdb/table.h | 14 ++++++++ 3 files changed, 90 insertions(+), 76 deletions(-) diff --git a/ovsdb/replication.c b/ovsdb/replication.c index bb1bd4250..b755976b0 100644 --- a/ovsdb/replication.c +++ b/ovsdb/replication.c @@ -54,18 +54,6 @@ static struct ovsdb_error *process_table_update(struct json *table_update, const char *table_name, struct ovsdb *database, struct ovsdb_txn *txn); - -static struct ovsdb_error *execute_insert(struct ovsdb_txn *txn, - const struct uuid *row_uuid, - struct ovsdb_table *table, - struct json *new); -static struct ovsdb_error *execute_delete(struct ovsdb_txn *txn, - const struct uuid *row_uuid, - struct ovsdb_table *table); -static struct ovsdb_error *execute_update(struct ovsdb_txn *txn, - const struct uuid *row_uuid, - struct ovsdb_table *table, - struct json *new); /* Maps from db name to sset of table names. */ static struct shash excluded_tables = SHASH_INITIALIZER(&excluded_tables); @@ -687,77 +675,20 @@ process_table_update(struct json *table_update, const char *table_name, new = shash_find_data(json_object(row_update), "new"); struct ovsdb_error *error; - error = (!new ? execute_delete(txn, &uuid, table) - : !old ? execute_insert(txn, &uuid, table, new) - : execute_update(txn, &uuid, table, new)); + error = (!new ? ovsdb_table_execute_delete(txn, &uuid, table) + : !old ? ovsdb_table_execute_insert(txn, &uuid, table, new) + : ovsdb_table_execute_update(txn, &uuid, table, new)); if (error) { + if (!strcmp(ovsdb_error_get_tag(error), "consistency violation")) { + ovsdb_error_assert(error); + error = NULL; + } return error; } } return NULL; } -static struct ovsdb_error * -execute_insert(struct ovsdb_txn *txn, const struct uuid *row_uuid, - struct ovsdb_table *table, struct json *json_row) -{ - struct ovsdb_row *row = ovsdb_row_create(table); - struct ovsdb_error *error = ovsdb_row_from_json(row, json_row, NULL, NULL); - if (!error) { - *ovsdb_row_get_uuid_rw(row) = *row_uuid; - ovsdb_txn_row_insert(txn, row); - } else { - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); - VLOG_WARN_RL(&rl, "cannot add existing row "UUID_FMT" to table %s", - UUID_ARGS(row_uuid), table->schema->name); - ovsdb_row_destroy(row); - } - - return error; -} - -static struct ovsdb_error * -execute_delete(struct ovsdb_txn *txn, const struct uuid *row_uuid, - struct ovsdb_table *table) -{ - const struct ovsdb_row *row = ovsdb_table_get_row(table, row_uuid); - if (row) { - ovsdb_txn_row_delete(txn, row); - } else { - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); - VLOG_WARN_RL(&rl, "cannot delete missing row "UUID_FMT" from table %s", - UUID_ARGS(row_uuid), table->schema->name); - } - return NULL; -} - -static struct ovsdb_error * -execute_update(struct ovsdb_txn *txn, const struct uuid *row_uuid, - struct ovsdb_table *table, struct json *json_row) -{ - const struct ovsdb_row *row = ovsdb_table_get_row(table, row_uuid); - if (!row) { - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); - VLOG_WARN_RL(&rl, "cannot modify missing row "UUID_FMT" in table %s", - UUID_ARGS(row_uuid), table->schema->name); - return NULL; - } - - struct ovsdb_column_set columns = OVSDB_COLUMN_SET_INITIALIZER; - struct ovsdb_row *update = ovsdb_row_create(table); - struct ovsdb_error *error = ovsdb_row_from_json(update, json_row, - NULL, &columns); - - if (!error && !ovsdb_row_equal_columns(row, update, &columns)) { - ovsdb_row_update_columns(ovsdb_txn_row_modify(txn, row), - update, &columns); - } - - ovsdb_column_set_destroy(&columns); - ovsdb_row_destroy(update); - return error; -} - void request_ids_add(const struct json *id, struct ovsdb *db) { diff --git a/ovsdb/table.c b/ovsdb/table.c index 6cd2d886d..2935bd897 100644 --- a/ovsdb/table.c +++ b/ovsdb/table.c @@ -25,6 +25,7 @@ #include "ovsdb-parser.h" #include "ovsdb-types.h" #include "row.h" +#include "transaction.h" static void add_column(struct ovsdb_table_schema *ts, struct ovsdb_column *column) @@ -339,3 +340,71 @@ ovsdb_table_get_row(const struct ovsdb_table *table, const struct uuid *uuid) return NULL; } + +struct ovsdb_error * +ovsdb_table_execute_insert(struct ovsdb_txn *txn, const struct uuid *row_uuid, + struct ovsdb_table *table, struct json *json_row) +{ + const struct ovsdb_row *old_row = ovsdb_table_get_row(table, row_uuid); + if (old_row) { + return ovsdb_error( + "consistency violation", + "cannot delete missing row "UUID_FMT" from table %s", + UUID_ARGS(row_uuid), table->schema->name); + } + + struct ovsdb_row *row = ovsdb_row_create(table); + + struct ovsdb_error *error = ovsdb_row_from_json(row, json_row, NULL, NULL); + if (!error) { + *ovsdb_row_get_uuid_rw(row) = *row_uuid; + ovsdb_txn_row_insert(txn, row); + } else { + ovsdb_row_destroy(row); + } + + return error; +} + +struct ovsdb_error * +ovsdb_table_execute_delete(struct ovsdb_txn *txn, const struct uuid *row_uuid, + struct ovsdb_table *table) +{ + const struct ovsdb_row *row = ovsdb_table_get_row(table, row_uuid); + if (!row) { + return ovsdb_error( + "consistency violation", + "cannot delete missing row "UUID_FMT" from table %s", + UUID_ARGS(row_uuid), table->schema->name); + } + + ovsdb_txn_row_delete(txn, row); + return NULL; +} + +struct ovsdb_error * +ovsdb_table_execute_update(struct ovsdb_txn *txn, const struct uuid *row_uuid, + struct ovsdb_table *table, struct json *json_row) +{ + const struct ovsdb_row *row = ovsdb_table_get_row(table, row_uuid); + if (!row) { + return ovsdb_error( + "consistency violation", + "cannot modify missing row "UUID_FMT" from table %s", + UUID_ARGS(row_uuid), table->schema->name); + } + + struct ovsdb_column_set columns = OVSDB_COLUMN_SET_INITIALIZER; + struct ovsdb_row *update = ovsdb_row_create(table); + struct ovsdb_error *error = ovsdb_row_from_json(update, json_row, + NULL, &columns); + + if (!error && !ovsdb_row_equal_columns(row, update, &columns)) { + ovsdb_row_update_columns(ovsdb_txn_row_modify(txn, row), + update, &columns); + } + + ovsdb_column_set_destroy(&columns); + ovsdb_row_destroy(update); + return error; +} diff --git a/ovsdb/table.h b/ovsdb/table.h index 69dd649df..e21ec7b31 100644 --- a/ovsdb/table.h +++ b/ovsdb/table.h @@ -23,6 +23,7 @@ struct json; struct uuid; +struct ovsdb_txn; /* Schema for a database table. */ struct ovsdb_table_schema { @@ -70,4 +71,17 @@ void ovsdb_table_destroy(struct ovsdb_table *); const struct ovsdb_row *ovsdb_table_get_row(const struct ovsdb_table *, const struct uuid *); +/* Below functions adds row modification for ovsdb table to the transaction. */ +struct ovsdb_error *ovsdb_table_execute_insert(struct ovsdb_txn *txn, + const struct uuid *row_uuid, + struct ovsdb_table *table, + struct json *new); +struct ovsdb_error *ovsdb_table_execute_delete(struct ovsdb_txn *txn, + const struct uuid *row_uuid, + struct ovsdb_table *table); +struct ovsdb_error *ovsdb_table_execute_update(struct ovsdb_txn *txn, + const struct uuid *row_uuid, + struct ovsdb_table *table, + struct json *new); + #endif /* ovsdb/table.h */ From patchwork Sat Jun 12 02:00:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491227 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21Br49dxz9sVm for ; Sat, 12 Jun 2021 12:00:36 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 1DA7C41521; Sat, 12 Jun 2021 02:00:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jx6zQEXhcmfi; Sat, 12 Jun 2021 02:00:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 17077414B8; Sat, 12 Jun 2021 02:00:30 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C7AEDC000E; Sat, 12 Jun 2021 02:00:29 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 19E94C000B for ; Sat, 12 Jun 2021 02:00:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id D32C183E81 for ; Sat, 12 Jun 2021 02:00:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bTK2bwJ0yWJp for ; Sat, 12 Jun 2021 02:00:26 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp1.osuosl.org (Postfix) with ESMTPS id D38BB83EC6 for ; Sat, 12 Jun 2021 02:00:25 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 78E3B1BF204; Sat, 12 Jun 2021 02:00:23 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:03 +0200 Message-Id: <20210612020008.3944088-5-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 4/9] ovsdb: row: Add support for xor-based row updates. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This will be used to apply update3 type updates to ovsdb tables while processing updates for future ovsdb 'relay' service model. 'ovsdb_datum_apply_diff' is allowed to fail, so adding support to return this error. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- ovsdb/execution.c | 5 +++-- ovsdb/replication.c | 2 +- ovsdb/row.c | 30 +++++++++++++++++++++++++----- ovsdb/row.h | 6 ++++-- ovsdb/table.c | 9 +++++---- ovsdb/table.h | 2 +- 6 files changed, 39 insertions(+), 15 deletions(-) diff --git a/ovsdb/execution.c b/ovsdb/execution.c index 3a0dad5d0..f6150e944 100644 --- a/ovsdb/execution.c +++ b/ovsdb/execution.c @@ -483,8 +483,9 @@ update_row_cb(const struct ovsdb_row *row, void *ur_) ur->n_matches++; if (!ovsdb_row_equal_columns(row, ur->row, ur->columns)) { - ovsdb_row_update_columns(ovsdb_txn_row_modify(ur->txn, row), - ur->row, ur->columns); + ovsdb_error_assert(ovsdb_row_update_columns( + ovsdb_txn_row_modify(ur->txn, row), + ur->row, ur->columns, false)); } return true; diff --git a/ovsdb/replication.c b/ovsdb/replication.c index b755976b0..d8b56d813 100644 --- a/ovsdb/replication.c +++ b/ovsdb/replication.c @@ -677,7 +677,7 @@ process_table_update(struct json *table_update, const char *table_name, struct ovsdb_error *error; error = (!new ? ovsdb_table_execute_delete(txn, &uuid, table) : !old ? ovsdb_table_execute_insert(txn, &uuid, table, new) - : ovsdb_table_execute_update(txn, &uuid, table, new)); + : ovsdb_table_execute_update(txn, &uuid, table, new, false)); if (error) { if (!strcmp(ovsdb_error_get_tag(error), "consistency violation")) { ovsdb_error_assert(error); diff --git a/ovsdb/row.c b/ovsdb/row.c index 755ab91a8..65a054621 100644 --- a/ovsdb/row.c +++ b/ovsdb/row.c @@ -163,20 +163,40 @@ ovsdb_row_equal_columns(const struct ovsdb_row *a, return true; } -void +struct ovsdb_error * ovsdb_row_update_columns(struct ovsdb_row *dst, const struct ovsdb_row *src, - const struct ovsdb_column_set *columns) + const struct ovsdb_column_set *columns, + bool xor) { size_t i; for (i = 0; i < columns->n_columns; i++) { const struct ovsdb_column *column = columns->columns[i]; + struct ovsdb_datum xor_datum; + struct ovsdb_error *error; + + if (xor) { + error = ovsdb_datum_apply_diff(&xor_datum, + &dst->fields[column->index], + &src->fields[column->index], + &column->type); + if (error) { + return error; + } + } + ovsdb_datum_destroy(&dst->fields[column->index], &column->type); - ovsdb_datum_clone(&dst->fields[column->index], - &src->fields[column->index], - &column->type); + + if (xor) { + ovsdb_datum_swap(&dst->fields[column->index], &xor_datum); + } else { + ovsdb_datum_clone(&dst->fields[column->index], + &src->fields[column->index], + &column->type); + } } + return NULL; } /* Appends the string form of the value in 'row' of each of the columns in diff --git a/ovsdb/row.h b/ovsdb/row.h index 2c441b5a4..394ac8eb4 100644 --- a/ovsdb/row.h +++ b/ovsdb/row.h @@ -82,8 +82,10 @@ bool ovsdb_row_equal_columns(const struct ovsdb_row *, int ovsdb_row_compare_columns_3way(const struct ovsdb_row *, const struct ovsdb_row *, const struct ovsdb_column_set *); -void ovsdb_row_update_columns(struct ovsdb_row *, const struct ovsdb_row *, - const struct ovsdb_column_set *); +struct ovsdb_error *ovsdb_row_update_columns(struct ovsdb_row *, + const struct ovsdb_row *, + const struct ovsdb_column_set *, + bool xor); void ovsdb_row_columns_to_string(const struct ovsdb_row *, const struct ovsdb_column_set *, struct ds *); struct ovsdb_error *ovsdb_row_from_json(struct ovsdb_row *, diff --git a/ovsdb/table.c b/ovsdb/table.c index 2935bd897..455a3663f 100644 --- a/ovsdb/table.c +++ b/ovsdb/table.c @@ -384,7 +384,8 @@ ovsdb_table_execute_delete(struct ovsdb_txn *txn, const struct uuid *row_uuid, struct ovsdb_error * ovsdb_table_execute_update(struct ovsdb_txn *txn, const struct uuid *row_uuid, - struct ovsdb_table *table, struct json *json_row) + struct ovsdb_table *table, struct json *json_row, + bool xor) { const struct ovsdb_row *row = ovsdb_table_get_row(table, row_uuid); if (!row) { @@ -399,9 +400,9 @@ ovsdb_table_execute_update(struct ovsdb_txn *txn, const struct uuid *row_uuid, struct ovsdb_error *error = ovsdb_row_from_json(update, json_row, NULL, &columns); - if (!error && !ovsdb_row_equal_columns(row, update, &columns)) { - ovsdb_row_update_columns(ovsdb_txn_row_modify(txn, row), - update, &columns); + if (!error && (xor || !ovsdb_row_equal_columns(row, update, &columns))) { + error = ovsdb_row_update_columns(ovsdb_txn_row_modify(txn, row), + update, &columns, xor); } ovsdb_column_set_destroy(&columns); diff --git a/ovsdb/table.h b/ovsdb/table.h index e21ec7b31..ce69a5d13 100644 --- a/ovsdb/table.h +++ b/ovsdb/table.h @@ -82,6 +82,6 @@ struct ovsdb_error *ovsdb_table_execute_delete(struct ovsdb_txn *txn, struct ovsdb_error *ovsdb_table_execute_update(struct ovsdb_txn *txn, const struct uuid *row_uuid, struct ovsdb_table *table, - struct json *new); + struct json *new, bool xor); #endif /* ovsdb/table.h */ From patchwork Sat Jun 12 02:00:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491228 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21C80SfVz9sVm for ; Sat, 12 Jun 2021 12:00:52 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A2CDE4159D; Sat, 12 Jun 2021 02:00:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eh9ofm4YHyWQ; Sat, 12 Jun 2021 02:00:47 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id E9824415B0; Sat, 12 Jun 2021 02:00:45 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B9DC5C000E; Sat, 12 Jun 2021 02:00:45 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 10B2BC000B for ; Sat, 12 Jun 2021 02:00:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 785D6415F6 for ; Sat, 12 Jun 2021 02:00:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bVPxdk3H1TuT for ; Sat, 12 Jun 2021 02:00:29 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp4.osuosl.org (Postfix) with ESMTPS id CFF0E415A3 for ; Sat, 12 Jun 2021 02:00:28 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 1A90D1BF206; Sat, 12 Jun 2021 02:00:25 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:04 +0200 Message-Id: <20210612020008.3944088-6-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 5/9] ovsdb: New ovsdb 'relay' service model. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" New database service model 'relay' that is needed to scale out read-mostly database access, e.g. ovn-controller connections to OVN_Southbound. In this service model ovsdb-server connects to existing OVSDB server and maintains in-memory copy of the database. It serves read-only transactions and monitor requests by its own, but forwards write transactions to the relay source. Key differences from the active-backup replication: - support for "write" transactions (next commit). - no on-disk storage. (probably, faster operation) - support for multiple remotes (connect to the clustered db). - doesn't try to keep connection as long as possible, but faster reconnects to other remotes to avoid missing updates. - No need to know the complete database schema beforehand, only the schema name. - can be used along with other standalone and clustered databases by the same ovsdb-server process. (doesn't turn the whole jsonrpc server to read-only mode) - supports modern version of monitors (monitor_cond_since), because based on ovsdb-cs. - could be chained, i.e. multiple relays could be connected one to another in a row or in a tree-like form. - doesn't increase availability. - cannot be converted to other service models or become a main active server. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- ovsdb/_server.ovsschema | 7 +- ovsdb/_server.xml | 16 +- ovsdb/automake.mk | 2 + ovsdb/execution.c | 5 + ovsdb/ovsdb-server.c | 97 ++++++++---- ovsdb/ovsdb.c | 2 + ovsdb/ovsdb.h | 3 + ovsdb/relay.c | 339 ++++++++++++++++++++++++++++++++++++++++ ovsdb/relay.h | 34 ++++ 9 files changed, 464 insertions(+), 41 deletions(-) create mode 100644 ovsdb/relay.c create mode 100644 ovsdb/relay.h diff --git a/ovsdb/_server.ovsschema b/ovsdb/_server.ovsschema index a867e5cbf..e3d9d893b 100644 --- a/ovsdb/_server.ovsschema +++ b/ovsdb/_server.ovsschema @@ -1,13 +1,14 @@ {"name": "_Server", - "version": "1.1.0", - "cksum": "3236486585 698", + "version": "1.2.0", + "cksum": "3009684573 744", "tables": { "Database": { "columns": { "name": {"type": "string"}, "model": { "type": {"key": {"type": "string", - "enum": ["set", ["standalone", "clustered"]]}}}, + "enum": ["set", + ["standalone", "clustered", "relay"]]}}}, "connected": {"type": "boolean"}, "leader": {"type": "boolean"}, "schema": { diff --git a/ovsdb/_server.xml b/ovsdb/_server.xml index 70cd22db7..414be6715 100644 --- a/ovsdb/_server.xml +++ b/ovsdb/_server.xml @@ -60,12 +60,14 @@ The storage model: standalone for a standalone or - active-backup database, clustered for a clustered database. + active-backup database, clustered for a clustered database, + relay for a relay database. The database schema, as a JSON string. In the case of a clustered - database, this is empty until it finishes joining its cluster. + database, this is empty until it finishes joining its cluster. In the + case of a relay database - until it connects to the relay source. @@ -85,20 +87,20 @@ True if the database is the leader in its cluster. For a standalone or - active-backup database, this is always true. + active-backup database, this is always true. Always false for relay. The cluster ID for this database, which is the same for all of the - servers that host this particular clustered database. For a standalone - or active-backup database, this is empty. + servers that host this particular clustered database. For a + standalone, active-backup or relay database, this is empty. The server ID for this database, different for each server that hosts a particular clustered database. A server that hosts more than one clustered database will have a different sid in each one. - For a standalone or active-backup database, this is empty. + For a standalone, active-backup or relay database, this is empty. @@ -112,7 +114,7 @@

- For a standalone or active-backup database, this is empty. + For a standalone, active-backup or relay database, this is empty.

diff --git a/ovsdb/automake.mk b/ovsdb/automake.mk index 446d6c136..05c8ebbdf 100644 --- a/ovsdb/automake.mk +++ b/ovsdb/automake.mk @@ -34,6 +34,8 @@ ovsdb_libovsdb_la_SOURCES = \ ovsdb/rbac.h \ ovsdb/replication.c \ ovsdb/replication.h \ + ovsdb/relay.c \ + ovsdb/relay.h \ ovsdb/row.c \ ovsdb/row.h \ ovsdb/server.c \ diff --git a/ovsdb/execution.c b/ovsdb/execution.c index f6150e944..dd2569055 100644 --- a/ovsdb/execution.c +++ b/ovsdb/execution.c @@ -196,6 +196,11 @@ ovsdb_execute_compose(struct ovsdb *db, const struct ovsdb_session *session, "%s operation not allowed on " "table in reserved database %s", op_name, db->schema->name); + } else if (db->is_relay) { + error = ovsdb_error("not allowed", + "%s operation not allowed when " + "database server is in relay mode", + op_name); } } if (error) { diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index 23bd226a3..77b1fbe40 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -44,6 +44,7 @@ #include "openvswitch/poll-loop.h" #include "process.h" #include "replication.h" +#include "relay.h" #include "row.h" #include "simap.h" #include "openvswitch/shash.h" @@ -225,6 +226,8 @@ main_loop(struct server_config *config, } } + ovsdb_relay_run(); + struct shash_node *next; SHASH_FOR_EACH_SAFE (node, next, all_dbs) { struct db *db = node->data; @@ -273,6 +276,8 @@ main_loop(struct server_config *config, replication_wait(); } + ovsdb_relay_wait(); + ovsdb_jsonrpc_server_wait(jsonrpc); unixctl_server_wait(unixctl); SHASH_FOR_EACH(node, all_dbs) { @@ -546,6 +551,9 @@ close_db(struct server_config *config, struct db *db, char *comment) { if (db) { ovsdb_jsonrpc_server_remove_db(config->jsonrpc, db->db, comment); + if (db->db->is_relay) { + ovsdb_relay_del_db(db->db); + } ovsdb_destroy(db->db); free(db->filename); free(db); @@ -554,6 +562,28 @@ close_db(struct server_config *config, struct db *db, char *comment) } } +static void +update_schema(struct ovsdb *db, const struct ovsdb_schema *schema, void *aux) +{ + struct server_config *config = aux; + + if (!db->schema || strcmp(schema->version, db->schema->version)) { + ovsdb_jsonrpc_server_reconnect( + config->jsonrpc, false, + (db->schema + ? xasprintf("database %s schema changed", db->name) + : xasprintf("database %s connected to storage", db->name))); + } + + ovsdb_replace(db, ovsdb_create(ovsdb_schema_clone(schema), NULL)); + + /* Force update to schema in _Server database. */ + struct db *dbp = shash_find_data(config->all_dbs, db->name); + if (dbp) { + dbp->row_uuid = UUID_ZERO; + } +} + static struct ovsdb_error * OVS_WARN_UNUSED_RESULT parse_txn(struct server_config *config, struct db *db, const struct ovsdb_schema *schema, const struct json *txn_json, @@ -575,21 +605,7 @@ parse_txn(struct server_config *config, struct db *db, if (error) { return error; } - - if (!db->db->schema || - strcmp(schema->version, db->db->schema->version)) { - ovsdb_jsonrpc_server_reconnect( - config->jsonrpc, false, - (db->db->schema - ? xasprintf("database %s schema changed", db->db->name) - : xasprintf("database %s connected to storage", - db->db->name))); - } - - ovsdb_replace(db->db, ovsdb_create(ovsdb_schema_clone(schema), NULL)); - - /* Force update to schema in _Server database. */ - db->row_uuid = UUID_ZERO; + update_schema(db->db, schema, config); } if (txn_json) { @@ -660,27 +676,42 @@ add_db(struct server_config *config, struct db *db) static struct ovsdb_error * OVS_WARN_UNUSED_RESULT open_db(struct server_config *config, const char *filename) { + bool is_relay = !strncmp(filename, "relay:", 6); + const char *relay_remotes = NULL; + struct ovsdb_storage *storage; + struct ovsdb_error *error; struct db *db; + char *name; + + if (!is_relay) { + /* If we know that the file is already open, return a good error + * message. Otherwise, if the file is open, we'll fail later on with + * a harder to interpret file locking error. */ + if (is_already_open(config, filename)) { + return ovsdb_error(NULL, "%s: already open", filename); + } - /* If we know that the file is already open, return a good error message. - * Otherwise, if the file is open, we'll fail later on with a harder to - * interpret file locking error. */ - if (is_already_open(config, filename)) { - return ovsdb_error(NULL, "%s: already open", filename); - } + error = ovsdb_storage_open(filename, true, &storage); + if (error) { + return error; + } + name = xstrdup(filename); + } else { + /* Parsing the relay in format 'relay:DB_NAME:'*/ + relay_remotes = strchr(filename + 6, ':'); - struct ovsdb_storage *storage; - struct ovsdb_error *error; - error = ovsdb_storage_open(filename, true, &storage); - if (error) { - return error; + if (!relay_remotes || relay_remotes[0] == '\0') { + return ovsdb_error(NULL, "%s: invalid syntax", filename); + } + name = xmemdup0(filename, relay_remotes - filename); + storage = ovsdb_storage_create_unbacked(name + 6); + relay_remotes++; /* Skip the ':'. */ } - db = xzalloc(sizeof *db); - db->filename = xstrdup(filename); + db->filename = name; struct ovsdb_schema *schema; - if (ovsdb_storage_is_clustered(storage)) { + if (is_relay || ovsdb_storage_is_clustered(storage)) { schema = NULL; } else { struct json *txn_json; @@ -716,6 +747,10 @@ open_db(struct server_config *config, const char *filename) } add_db(config, db); + + if (is_relay) { + ovsdb_relay_add_db(db->db, relay_remotes, update_schema, config); + } return NULL; } @@ -1153,11 +1188,11 @@ update_database_status(struct ovsdb_row *row, struct db *db) { ovsdb_util_write_string_column(row, "name", db->db->name); ovsdb_util_write_string_column(row, "model", - ovsdb_storage_get_model(db->db->storage)); + db->db->is_relay ? "relay" : ovsdb_storage_get_model(db->db->storage)); ovsdb_util_write_bool_column(row, "connected", ovsdb_storage_is_connected(db->db->storage)); ovsdb_util_write_bool_column(row, "leader", - ovsdb_storage_is_leader(db->db->storage)); + db->db->is_relay ? false : ovsdb_storage_is_leader(db->db->storage)); ovsdb_util_write_uuid_column(row, "cid", ovsdb_storage_get_cid(db->db->storage)); ovsdb_util_write_uuid_column(row, "sid", diff --git a/ovsdb/ovsdb.c b/ovsdb/ovsdb.c index e019631e9..999cd0d75 100644 --- a/ovsdb/ovsdb.c +++ b/ovsdb/ovsdb.c @@ -421,6 +421,8 @@ ovsdb_create(struct ovsdb_schema *schema, struct ovsdb_storage *storage) ovs_list_init(&db->triggers); db->run_triggers_now = db->run_triggers = false; + db->is_relay = false; + shash_init(&db->tables); if (schema) { SHASH_FOR_EACH (node, &schema->tables) { diff --git a/ovsdb/ovsdb.h b/ovsdb/ovsdb.h index 72e127c84..16bd5f5ec 100644 --- a/ovsdb/ovsdb.h +++ b/ovsdb/ovsdb.h @@ -91,6 +91,9 @@ struct ovsdb { bool need_txn_history; /* Need to maintain history of transactions. */ unsigned int n_txn_history; /* Current number of history transactions. */ struct ovs_list txn_history; /* Contains "struct ovsdb_txn_history_node. */ + + /* Relay mode. */ + bool is_relay; }; struct ovsdb *ovsdb_create(struct ovsdb_schema *, struct ovsdb_storage *); diff --git a/ovsdb/relay.c b/ovsdb/relay.c new file mode 100644 index 000000000..5f423a0b9 --- /dev/null +++ b/ovsdb/relay.c @@ -0,0 +1,339 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "relay.h" + +#include "coverage.h" +#include "jsonrpc.h" +#include "openvswitch/hmap.h" +#include "openvswitch/json.h" +#include "openvswitch/list.h" +#include "openvswitch/poll-loop.h" +#include "openvswitch/shash.h" +#include "openvswitch/vlog.h" +#include "ovsdb.h" +#include "ovsdb-cs.h" +#include "ovsdb-error.h" +#include "row.h" +#include "table.h" +#include "transaction.h" +#include "util.h" + +VLOG_DEFINE_THIS_MODULE(relay); + +static struct shash relay_dbs = SHASH_INITIALIZER(&relay_dbs); + +struct relay_ctx { + struct ovsdb *db; + struct ovsdb_cs *cs; + + /* Schema updates. */ + struct ovsdb_schema *new_schema; + schema_change_callback schema_change_cb; + void *schema_change_aux; +}; + +static struct json * +ovsdb_relay_compose_monitor_request(const struct json *schema_json, void *ctx_) +{ + struct json *monitor_request = json_object_create(); + struct relay_ctx *ctx = ctx_; + struct ovsdb_schema *schema; + struct ovsdb *db = ctx->db; + struct ovsdb_error *error; + + error = ovsdb_schema_from_json(schema_json, &schema); + if (error) { + char *msg = ovsdb_error_to_string_free(error); + VLOG_WARN("%s: Failed to parse db schema: %s", db->name, msg); + free(msg); + /* There is nothing we can really do here. */ + return monitor_request; + } + + const struct shash_node *node; + SHASH_FOR_EACH (node, &schema->tables) { + struct json *monitor_request_array = json_array_create_empty(); + struct ovsdb_table_schema *table = node->data; + + json_array_add(monitor_request_array, json_object_create()); + json_object_put(monitor_request, table->name, monitor_request_array); + } + + if (!db->schema || ovsdb_schema_equal(schema, db->schema)) { + VLOG_DBG("database %s schema changed.", db->name); + if (ctx->new_schema) { + ovsdb_schema_destroy(ctx->new_schema); + } + /* We will update the schema later when we will receive actual data + * from the mointor in order to avoid sitting with an empty database + * until the monitor reply. */ + ctx->new_schema = schema; + } else { + ovsdb_schema_destroy(schema); + } + return monitor_request; +} + +static struct ovsdb_cs_ops relay_cs_ops = { + .compose_monitor_requests = ovsdb_relay_compose_monitor_request, +}; + +void +ovsdb_relay_add_db(struct ovsdb *db, const char *remote, + schema_change_callback schema_change_cb, + void *schema_change_aux) +{ + struct relay_ctx *ctx; + + if (!db || !remote) { + return; + } + + ctx = shash_find_data(&relay_dbs, db->name); + if (ctx) { + ovsdb_cs_set_remote(ctx->cs, remote, true); + return; + } + + db->is_relay = true; + ctx = xzalloc(sizeof *ctx); + ctx->schema_change_cb = schema_change_cb; + ctx->schema_change_aux = schema_change_aux; + ctx->db = db; + ctx->cs = ovsdb_cs_create(db->name, 3, &relay_cs_ops, ctx); + shash_add(&relay_dbs, db->name, ctx); + ovsdb_cs_set_leader_only(ctx->cs, false); + ovsdb_cs_set_remote(ctx->cs, remote, true); + + VLOG_DBG("added database: %s, %s", db->name, remote); +} + +void +ovsdb_relay_del_db(struct ovsdb *db) +{ + struct relay_ctx *ctx; + + if (!db) { + return; + } + + ctx = shash_find_and_delete(&relay_dbs, db->name); + if (!ctx) { + return; + } + + VLOG_DBG("removed database: %s", db->name); + + db->is_relay = false; + ovsdb_cs_destroy(ctx->cs); + free(ctx); +} + +static struct ovsdb_error * +ovsdb_relay_process_row_update(struct ovsdb_table *table, + const struct ovsdb_cs_row_update *ru, + struct ovsdb_txn *txn) +{ + const struct uuid *uuid = &ru->row_uuid; + struct ovsdb_error * error = NULL; + + /* XXX: ovsdb-cs module returns shash which was previously part of a json + * structure and we need json row format in order to use ovsdb_row* + * functions. Creating a json object out of shash. */ + struct json *json_row = json_object_create(); + struct shash *obj = json_row->object; + json_row->object = CONST_CAST(struct shash *, ru->columns); + + switch (ru->type) { + case OVSDB_CS_ROW_DELETE: + error = ovsdb_table_execute_delete(txn, uuid, table); + break; + + case OVSDB_CS_ROW_INSERT: + error = ovsdb_table_execute_insert(txn, uuid, table, json_row); + break; + + case OVSDB_CS_ROW_UPDATE: + error = ovsdb_table_execute_update(txn, uuid, table, json_row, false); + break; + + case OVSDB_CS_ROW_XOR: + error = ovsdb_table_execute_update(txn, uuid, table, json_row, true); + break; + + default: + OVS_NOT_REACHED(); + } + + json_row->object = obj; + json_destroy(json_row); + + return error; +} + +static struct ovsdb_error * +ovsdb_relay_parse_update__(struct ovsdb *db, + const struct ovsdb_cs_db_update *du) +{ + struct ovsdb_error *error = NULL; + struct ovsdb_txn *txn; + + txn = ovsdb_txn_create(db); + + for (size_t i = 0; i < du->n; i++) { + const struct ovsdb_cs_table_update *tu = &du->table_updates[i]; + struct ovsdb_table *table = ovsdb_get_table(db, tu->table_name); + + if (!table) { + return ovsdb_error("unknown table", "unknown table %s", + tu->table_name); + } + + for (size_t j = 0; j < tu->n; j++) { + const struct ovsdb_cs_row_update *ru = &tu->row_updates[j]; + + error = ovsdb_relay_process_row_update(table, ru, txn); + if (error) { + return error; + } + } + } + + if (error) { + ovsdb_txn_abort(txn); + return error; + } else { + /* Commit transaction. */ + error = ovsdb_txn_propose_commit_block(txn, false); + } + + return error; +} + +static struct ovsdb_error * +ovsdb_relay_clear(struct ovsdb *db) +{ + struct ovsdb_txn *txn = ovsdb_txn_create(db); + struct shash_node *table_node; + + SHASH_FOR_EACH (table_node, &db->tables) { + struct ovsdb_table *table = table_node->data; + struct ovsdb_row *row, *next; + + HMAP_FOR_EACH_SAFE (row, next, hmap_node, &table->rows) { + ovsdb_txn_row_delete(txn, row); + } + } + + return ovsdb_txn_propose_commit_block(txn, false); +} + +static void +ovsdb_relay_parse_update(struct relay_ctx *ctx, + const struct ovsdb_cs_update_event *update) +{ + if (!ctx->db) { + return; + } + + if (update->monitor_reply && ctx->new_schema) { + /* There was a schema change. Updating a database with a new schema + * before processing monitor reply with the new data. */ + ctx->schema_change_cb(ctx->db, ctx->new_schema, + ctx->schema_change_aux); + ovsdb_schema_destroy(ctx->new_schema); + ctx->new_schema = NULL; + } + + struct ovsdb_cs_db_update *du; + struct ovsdb_error *error = ovsdb_cs_parse_db_update(update->table_updates, + update->version, &du); + if (!error) { + if (update->clear) { + error = ovsdb_relay_clear(ctx->db); + } + if (!error) { + error = ovsdb_relay_parse_update__(ctx->db, du); + } + } + ovsdb_cs_db_update_destroy(du); + if (error) { + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); + if (!VLOG_DROP_WARN(&rl)) { + char *s = ovsdb_error_to_string(error); + VLOG_WARN_RL(&rl, "%s", s); + free(s); + } + /* Something bad happened. Try to recover. */ + if (!strcmp(ovsdb_error_get_tag(error), "consistency violation")) { + ovsdb_cs_flag_inconsistency(ctx->cs); + } else { + ovsdb_cs_force_reconnect(ctx->cs); + } + ovsdb_error_destroy(error); + } +} + +void +ovsdb_relay_run(void) +{ + struct shash_node *node; + SHASH_FOR_EACH (node, &relay_dbs) { + struct relay_ctx *ctx = node->data; + struct ovs_list events; + + ovsdb_cs_run(ctx->cs, &events); + + struct ovsdb_cs_event *event; + LIST_FOR_EACH_POP (event, list_node, &events) { + if (!ctx->db) { + ovsdb_cs_event_destroy(event); + continue; + } + + switch (event->type) { + case OVSDB_CS_EVENT_TYPE_RECONNECT: + /* Nothing to do. */ + break; + + case OVSDB_CS_EVENT_TYPE_UPDATE: + ovsdb_relay_parse_update(ctx, &event->update); + break; + + case OVSDB_CS_EVENT_TYPE_TXN_REPLY: + case OVSDB_CS_EVENT_TYPE_LOCKED: + /* Not expected. */ + break; + } + ovsdb_cs_event_destroy(event); + } + } +} + +void +ovsdb_relay_wait(void) +{ + struct shash_node *node; + + SHASH_FOR_EACH (node, &relay_dbs) { + struct relay_ctx *ctx = node->data; + + ovsdb_cs_wait(ctx->cs); + } +} diff --git a/ovsdb/relay.h b/ovsdb/relay.h new file mode 100644 index 000000000..68586e9db --- /dev/null +++ b/ovsdb/relay.h @@ -0,0 +1,34 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef OVSDB_RELAY_H +#define OVSDB_RELAY_H 1 + +struct json; +struct ovsdb; +struct ovsdb_schema; + +typedef void (*schema_change_callback)(struct ovsdb *, + const struct ovsdb_schema *, void *aux); + +void ovsdb_relay_add_db(struct ovsdb *, const char *remote, + schema_change_callback schema_change_cb, + void *schema_change_aux); +void ovsdb_relay_del_db(struct ovsdb *); +void ovsdb_relay_run(void); +void ovsdb_relay_wait(void); + +#endif /* OVSDB_RELAY_H */ From patchwork Sat Jun 12 02:00:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491231 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21CQ5lH1z9sVm for ; Sat, 12 Jun 2021 12:01:06 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 745F9841E1; Sat, 12 Jun 2021 02:01:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MAHulPv8k2Pm; Sat, 12 Jun 2021 02:01:00 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 33432842ED; Sat, 12 Jun 2021 02:00:59 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F02D5C000E; Sat, 12 Jun 2021 02:00:58 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 12690C000B for ; Sat, 12 Jun 2021 02:00:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3E1A360659 for ; Sat, 12 Jun 2021 02:00:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ue20Yd1mhFYX for ; Sat, 12 Jun 2021 02:00:32 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp3.osuosl.org (Postfix) with ESMTPS id 2024A60AA4 for ; Sat, 12 Jun 2021 02:00:30 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 83FBD1BF203; Sat, 12 Jun 2021 02:00:28 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:05 +0200 Message-Id: <20210612020008.3944088-7-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 6/9] ovsdb: relay: Add support for transaction forwarding. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Current version of ovsdb relay allows to scale out read-only access to the primary database. However, many clients are not read-only but read-mostly. For example, ovn-controller. In order to scale out database access for this case ovsdb-server need to process transactions that are not read-only. Relay is not allowed to do that, i.e. not allowed to modify the database, but it can act like a proxy and forward transactions that includes database modifications to the primary server and forward replies back to a client. At the same time it may serve read-only transactions and monitor requests by itself greatly reducing the load on primary server. This configuration will slightly increase transaction latency, but it's not very important for read-mostly use cases. Implementation details: With this change instead of creating a trigger to commit the transaction, ovsdb-server will create a trigger for transaction forwarding. Later, ovsdb_relay_run() will send all new transactions to the relay source. Once transaction reply received from the relay source, ovsdb-relay module will update the state of the transaction forwarding with the reply. After that, trigger_run() will complete the trigger and jsonrpc_server_run() will send the reply back to the client. Since transaction reply from the relay source will be received after all the updates, client will receive all the updates before receiving the transaction reply as it is in a normal scenario with other database models. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- ovsdb/automake.mk | 2 + ovsdb/execution.c | 18 ++-- ovsdb/ovsdb.c | 9 ++ ovsdb/ovsdb.h | 8 +- ovsdb/relay.c | 10 +- ovsdb/transaction-forward.c | 182 ++++++++++++++++++++++++++++++++++++ ovsdb/transaction-forward.h | 44 +++++++++ ovsdb/trigger.c | 48 ++++++++-- ovsdb/trigger.h | 41 ++++---- tests/ovsdb-server.at | 85 ++++++++++++++++- 10 files changed, 409 insertions(+), 38 deletions(-) create mode 100644 ovsdb/transaction-forward.c create mode 100644 ovsdb/transaction-forward.h diff --git a/ovsdb/automake.mk b/ovsdb/automake.mk index 05c8ebbdf..62cc02686 100644 --- a/ovsdb/automake.mk +++ b/ovsdb/automake.mk @@ -48,6 +48,8 @@ ovsdb_libovsdb_la_SOURCES = \ ovsdb/trigger.h \ ovsdb/transaction.c \ ovsdb/transaction.h \ + ovsdb/transaction-forward.c \ + ovsdb/transaction-forward.h \ ovsdb/ovsdb-util.c \ ovsdb/ovsdb-util.h ovsdb_libovsdb_la_CFLAGS = $(AM_CFLAGS) diff --git a/ovsdb/execution.c b/ovsdb/execution.c index dd2569055..f9b8067d0 100644 --- a/ovsdb/execution.c +++ b/ovsdb/execution.c @@ -99,7 +99,8 @@ lookup_executor(const char *name, bool *read_only) } /* On success, returns a transaction and stores the results to return to the - * client in '*resultsp'. + * client in '*resultsp'. If 'forwarding_needed' is nonnull and transaction + * needs to be forwarded (in relay mode), sets '*forwarding_needed' to true. * * On failure, returns NULL. If '*resultsp' is nonnull, then it is the results * to return to the client. If '*resultsp' is null, then the execution failed @@ -111,7 +112,8 @@ ovsdb_execute_compose(struct ovsdb *db, const struct ovsdb_session *session, const struct json *params, bool read_only, const char *role, const char *id, long long int elapsed_msec, long long int *timeout_msec, - bool *durable, struct json **resultsp) + bool *durable, bool *forwarding_needed, + struct json **resultsp) { struct ovsdb_execution x; struct ovsdb_error *error; @@ -120,6 +122,9 @@ ovsdb_execute_compose(struct ovsdb *db, const struct ovsdb_session *session, size_t i; *durable = false; + if (forwarding_needed) { + *forwarding_needed = false; + } if (params->type != JSON_ARRAY || !params->array.n || params->array.elems[0]->type != JSON_STRING @@ -196,11 +201,8 @@ ovsdb_execute_compose(struct ovsdb *db, const struct ovsdb_session *session, "%s operation not allowed on " "table in reserved database %s", op_name, db->schema->name); - } else if (db->is_relay) { - error = ovsdb_error("not allowed", - "%s operation not allowed when " - "database server is in relay mode", - op_name); + } else if (db->is_relay && forwarding_needed) { + *forwarding_needed = true; } } if (error) { @@ -245,7 +247,7 @@ ovsdb_execute(struct ovsdb *db, const struct ovsdb_session *session, struct json *results; struct ovsdb_txn *txn = ovsdb_execute_compose( db, session, params, read_only, role, id, elapsed_msec, timeout_msec, - &durable, &results); + &durable, NULL, &results); if (!txn) { return results; } diff --git a/ovsdb/ovsdb.c b/ovsdb/ovsdb.c index 999cd0d75..126d16a2f 100644 --- a/ovsdb/ovsdb.c +++ b/ovsdb/ovsdb.c @@ -33,6 +33,7 @@ #include "table.h" #include "timeval.h" #include "transaction.h" +#include "transaction-forward.h" #include "trigger.h" #include "openvswitch/vlog.h" @@ -422,6 +423,8 @@ ovsdb_create(struct ovsdb_schema *schema, struct ovsdb_storage *storage) db->run_triggers_now = db->run_triggers = false; db->is_relay = false; + ovs_list_init(&db->txn_forward_new); + hmap_init(&db->txn_forward_sent); shash_init(&db->tables); if (schema) { @@ -465,6 +468,12 @@ ovsdb_destroy(struct ovsdb *db) /* Destroy txn history. */ ovsdb_txn_history_destroy(db); + /* Cancell all the forwarded transactions. There should not be + * any as all triggers should be already cancelled. */ + ovsdb_txn_forward_cancel_all(db, false); + ovs_assert(hmap_is_empty(&db->txn_forward_sent)); + hmap_destroy(&db->txn_forward_sent); + /* The caller must ensure that no triggers remain. */ ovs_assert(ovs_list_is_empty(&db->triggers)); diff --git a/ovsdb/ovsdb.h b/ovsdb/ovsdb.h index 16bd5f5ec..4a7bd0f0e 100644 --- a/ovsdb/ovsdb.h +++ b/ovsdb/ovsdb.h @@ -93,7 +93,11 @@ struct ovsdb { struct ovs_list txn_history; /* Contains "struct ovsdb_txn_history_node. */ /* Relay mode. */ - bool is_relay; + bool is_relay; /* True, if database is in relay mode. */ + /* List that holds transactions waiting to be forwarded to the server. */ + struct ovs_list txn_forward_new; + /* Hash map for transactions that are already sent and waits for reply. */ + struct hmap txn_forward_sent; }; struct ovsdb *ovsdb_create(struct ovsdb_schema *, struct ovsdb_storage *); @@ -107,7 +111,7 @@ struct ovsdb_txn *ovsdb_execute_compose( struct ovsdb *, const struct ovsdb_session *, const struct json *params, bool read_only, const char *role, const char *id, long long int elapsed_msec, long long int *timeout_msec, - bool *durable, struct json **); + bool *durable, bool *forwarding_needed, struct json **); struct json *ovsdb_execute(struct ovsdb *, const struct ovsdb_session *, const struct json *params, bool read_only, diff --git a/ovsdb/relay.c b/ovsdb/relay.c index 5f423a0b9..ef689c649 100644 --- a/ovsdb/relay.c +++ b/ovsdb/relay.c @@ -32,6 +32,7 @@ #include "row.h" #include "table.h" #include "transaction.h" +#include "transaction-forward.h" #include "util.h" VLOG_DEFINE_THIS_MODULE(relay); @@ -298,6 +299,7 @@ ovsdb_relay_run(void) struct relay_ctx *ctx = node->data; struct ovs_list events; + ovsdb_txn_forward_run(ctx->db, ctx->cs); ovsdb_cs_run(ctx->cs, &events); struct ovsdb_cs_event *event; @@ -309,7 +311,9 @@ ovsdb_relay_run(void) switch (event->type) { case OVSDB_CS_EVENT_TYPE_RECONNECT: - /* Nothing to do. */ + /* Cancelling all the transactions that was already sent but + * not replied yet as they might be lost. */ + ovsdb_txn_forward_cancel_all(ctx->db, true); break; case OVSDB_CS_EVENT_TYPE_UPDATE: @@ -317,6 +321,9 @@ ovsdb_relay_run(void) break; case OVSDB_CS_EVENT_TYPE_TXN_REPLY: + ovsdb_txn_forward_complete(ctx->db, event->txn_reply); + break; + case OVSDB_CS_EVENT_TYPE_LOCKED: /* Not expected. */ break; @@ -335,5 +342,6 @@ ovsdb_relay_wait(void) struct relay_ctx *ctx = node->data; ovsdb_cs_wait(ctx->cs); + ovsdb_txn_forward_wait(ctx->db, ctx->cs); } } diff --git a/ovsdb/transaction-forward.c b/ovsdb/transaction-forward.c new file mode 100644 index 000000000..8ff12ef4b --- /dev/null +++ b/ovsdb/transaction-forward.c @@ -0,0 +1,182 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "transaction-forward.h" + +#include "coverage.h" +#include "jsonrpc.h" +#include "openvswitch/hmap.h" +#include "openvswitch/json.h" +#include "openvswitch/list.h" +#include "openvswitch/poll-loop.h" +#include "openvswitch/vlog.h" +#include "ovsdb.h" +#include "ovsdb-cs.h" +#include "util.h" + +VLOG_DEFINE_THIS_MODULE(transaction_forward); + +COVERAGE_DEFINE(txn_forward_cancel); +COVERAGE_DEFINE(txn_forward_complete); +COVERAGE_DEFINE(txn_forward_create); +COVERAGE_DEFINE(txn_forward_sent); + +struct ovsdb_txn_forward { + struct ovs_list new_node; /* In 'txn_forward_new' of struct ovsdb. */ + struct hmap_node sent_node; /* In 'txn_forward_sent' of struct ovsdb. */ + struct json *id; /* 'id' of the forwarded transaction. */ + struct jsonrpc_msg *request; /* Original request. */ + struct jsonrpc_msg *reply; /* Reply from the server. */ +}; + +struct ovsdb_txn_forward * +ovsdb_txn_forward_create(struct ovsdb *db, const struct jsonrpc_msg *request) +{ + struct ovsdb_txn_forward *txn_fwd = xzalloc(sizeof *txn_fwd); + + COVERAGE_INC(txn_forward_create); + txn_fwd->request = jsonrpc_msg_clone(request); + ovs_list_push_back(&db->txn_forward_new, &txn_fwd->new_node); + + return txn_fwd; +} + +static void +ovsdb_txn_forward_unlist(struct ovsdb *db, struct ovsdb_txn_forward *txn_fwd) +{ + if (!ovs_list_is_empty(&txn_fwd->new_node)) { + ovs_list_remove(&txn_fwd->new_node); + ovs_list_init(&txn_fwd->new_node); + } + if (!hmap_node_is_null(&txn_fwd->sent_node)) { + hmap_remove(&db->txn_forward_sent, &txn_fwd->sent_node); + hmap_node_nullify(&txn_fwd->sent_node); + } +} + +void +ovsdb_txn_forward_destroy(struct ovsdb *db, struct ovsdb_txn_forward *txn_fwd) +{ + if (!txn_fwd) { + return; + } + + ovsdb_txn_forward_unlist(db, txn_fwd); + json_destroy(txn_fwd->id); + jsonrpc_msg_destroy(txn_fwd->request); + jsonrpc_msg_destroy(txn_fwd->reply); + free(txn_fwd); +} + +bool +ovsdb_txn_forward_is_complete(const struct ovsdb_txn_forward *txn_fwd) +{ + return txn_fwd->reply != NULL; +} + +void +ovsdb_txn_forward_complete(struct ovsdb *db, const struct jsonrpc_msg *reply) +{ + struct ovsdb_txn_forward *t; + size_t hash = json_hash(reply->id, 0); + + HMAP_FOR_EACH_WITH_HASH (t, sent_node, hash, &db->txn_forward_sent) { + if (json_equal(reply->id, t->id)) { + COVERAGE_INC(txn_forward_complete); + t->reply = jsonrpc_msg_clone(reply); + + /* Replacing id with the id of the original request. */ + json_destroy(t->reply->id); + t->reply->id = json_clone(t->request->id); + + hmap_remove(&db->txn_forward_sent, &t->sent_node); + hmap_node_nullify(&t->sent_node); + + db->run_triggers_now = db->run_triggers = true; + return; + } + } +} + +struct jsonrpc_msg * +ovsdb_txn_forward_steal_reply(struct ovsdb_txn_forward *txn_fwd) +{ + struct jsonrpc_msg *reply = txn_fwd->reply; + + txn_fwd->reply = NULL; + return reply; +} + +void +ovsdb_txn_forward_run(struct ovsdb *db, struct ovsdb_cs *cs) +{ + struct ovsdb_txn_forward *t, *next; + + /* Send all transactions that needs to be forwarded. */ + LIST_FOR_EACH_SAFE (t, next, new_node, &db->txn_forward_new) { + if (!ovsdb_cs_may_send_transaction(cs)) { + break; + } + ovs_assert(!strcmp(t->request->method, "transact")); + t->id = ovsdb_cs_send_transaction(cs, json_clone(t->request->params)); + if (t->id) { + COVERAGE_INC(txn_forward_sent); + ovs_list_remove(&t->new_node); + ovs_list_init(&t->new_node); + hmap_insert(&db->txn_forward_sent, &t->sent_node, + json_hash(t->id, 0)); + } + } +} + +void +ovsdb_txn_forward_wait(struct ovsdb *db, struct ovsdb_cs *cs) +{ + if (ovsdb_cs_may_send_transaction(cs) + && !ovs_list_is_empty(&db->txn_forward_new)) { + poll_immediate_wake(); + } +} + +void +ovsdb_txn_forward_cancel(struct ovsdb *db, struct ovsdb_txn_forward *txn_fwd) +{ + COVERAGE_INC(txn_forward_cancel); + jsonrpc_msg_destroy(txn_fwd->reply); + txn_fwd->reply = jsonrpc_create_error(json_string_create("canceled"), + txn_fwd->request->id); + ovsdb_txn_forward_unlist(db, txn_fwd); +} + +void +ovsdb_txn_forward_cancel_all(struct ovsdb *db, bool sent_only) +{ + struct ovsdb_txn_forward *t, *next; + + HMAP_FOR_EACH_SAFE (t, next, sent_node, &db->txn_forward_sent) { + ovsdb_txn_forward_cancel(db, t); + } + + if (sent_only) { + return; + } + + LIST_FOR_EACH_SAFE (t, next, new_node, &db->txn_forward_new) { + ovsdb_txn_forward_cancel(db, t); + } +} diff --git a/ovsdb/transaction-forward.h b/ovsdb/transaction-forward.h new file mode 100644 index 000000000..6788d3824 --- /dev/null +++ b/ovsdb/transaction-forward.h @@ -0,0 +1,44 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef OVSDB_TXN_FORWARD_H +#define OVSDB_TXN_FORWARD_H 1 + +#include + +struct ovsdb; +struct ovsdb_cs; +struct ovsdb_txn_forward; +struct jsonrpc_session; +struct jsonrpc_msg; + +struct ovsdb_txn_forward *ovsdb_txn_forward_create( + struct ovsdb *, const struct jsonrpc_msg *request); +void ovsdb_txn_forward_destroy(struct ovsdb *, struct ovsdb_txn_forward *); + +bool ovsdb_txn_forward_is_complete(const struct ovsdb_txn_forward *); +void ovsdb_txn_forward_complete(struct ovsdb *, + const struct jsonrpc_msg *reply); + +struct jsonrpc_msg *ovsdb_txn_forward_steal_reply(struct ovsdb_txn_forward *); + +void ovsdb_txn_forward_run(struct ovsdb *, struct ovsdb_cs *); +void ovsdb_txn_forward_wait(struct ovsdb *, struct ovsdb_cs *); + +void ovsdb_txn_forward_cancel(struct ovsdb *, struct ovsdb_txn_forward *); +void ovsdb_txn_forward_cancel_all(struct ovsdb *, bool sent_only); + +#endif /* OVSDB_TXN_FORWARD_H */ diff --git a/ovsdb/trigger.c b/ovsdb/trigger.c index 0372302af..1c38a94c1 100644 --- a/ovsdb/trigger.c +++ b/ovsdb/trigger.c @@ -28,6 +28,7 @@ #include "openvswitch/poll-loop.h" #include "server.h" #include "transaction.h" +#include "transaction-forward.h" #include "openvswitch/vlog.h" #include "util.h" @@ -53,6 +54,7 @@ ovsdb_trigger_init(struct ovsdb_session *session, struct ovsdb *db, trigger->request = request; trigger->reply = NULL; trigger->progress = NULL; + trigger->txn_forward = NULL; trigger->created = now; trigger->timeout_msec = LLONG_MAX; trigger->read_only = read_only; @@ -65,6 +67,7 @@ void ovsdb_trigger_destroy(struct ovsdb_trigger *trigger) { ovsdb_txn_progress_destroy(trigger->progress); + ovsdb_txn_forward_destroy(trigger->db, trigger->txn_forward); ovs_list_remove(&trigger->node); jsonrpc_msg_destroy(trigger->request); jsonrpc_msg_destroy(trigger->reply); @@ -75,7 +78,7 @@ ovsdb_trigger_destroy(struct ovsdb_trigger *trigger) bool ovsdb_trigger_is_complete(const struct ovsdb_trigger *trigger) { - return trigger->reply && !trigger->progress; + return trigger->reply && !trigger->progress && !trigger->txn_forward; } struct jsonrpc_msg * @@ -98,6 +101,11 @@ ovsdb_trigger_cancel(struct ovsdb_trigger *trigger, const char *reason) trigger->progress = NULL; } + if (trigger->txn_forward) { + ovsdb_txn_forward_destroy(trigger->db, trigger->txn_forward); + trigger->txn_forward = NULL; + } + jsonrpc_msg_destroy(trigger->reply); trigger->reply = NULL; @@ -148,7 +156,7 @@ ovsdb_trigger_run(struct ovsdb *db, long long int now) LIST_FOR_EACH_SAFE (t, next, node, &db->triggers) { if (run_triggers || now - t->created >= t->timeout_msec - || t->progress) { + || t->progress || t->txn_forward) { if (ovsdb_trigger_try(t, now)) { disconnect_all = true; } @@ -188,7 +196,7 @@ static bool ovsdb_trigger_try(struct ovsdb_trigger *t, long long int now) { /* Handle "initialized" state. */ - if (!t->reply) { + if (!t->reply && !t->txn_forward) { ovs_assert(!t->progress); struct ovsdb_txn *txn = NULL; @@ -198,13 +206,14 @@ ovsdb_trigger_try(struct ovsdb_trigger *t, long long int now) return false; } - bool durable; + bool durable, forwarding_needed; struct json *result; + /* Trying to compose transaction. */ txn = ovsdb_execute_compose( t->db, t->session, t->request->params, t->read_only, t->role, t->id, now - t->created, &t->timeout_msec, - &durable, &result); + &durable, &forwarding_needed, &result); if (!txn) { if (result) { /* Complete. There was an error but we still represent it @@ -217,9 +226,20 @@ ovsdb_trigger_try(struct ovsdb_trigger *t, long long int now) return false; } - /* Transition to "committing" state. */ - t->reply = jsonrpc_create_reply(result, t->request->id); - t->progress = ovsdb_txn_propose_commit(txn, durable); + if (forwarding_needed) { + /* Transaction is good, but we don't need it. */ + ovsdb_txn_abort(txn); + json_destroy(result); + /* Transition to "forwarding" state. */ + t->txn_forward = ovsdb_txn_forward_create(t->db, t->request); + /* Forward will not be completed immediately. Will check + * next time. */ + return false; + } else { + /* Transition to "committing" state. */ + t->reply = jsonrpc_create_reply(result, t->request->id); + t->progress = ovsdb_txn_propose_commit(txn, durable); + } } else if (!strcmp(t->request->method, "convert")) { /* Permission check. */ if (t->role && *t->role) { @@ -348,6 +368,18 @@ ovsdb_trigger_try(struct ovsdb_trigger *t, long long int now) ovsdb_trigger_complete(t); } + return false; + } else if (t->txn_forward) { + /* Handle "forwarding" state. */ + if (!ovsdb_txn_forward_is_complete(t->txn_forward)) { + return false; + } + + /* Transition to "complete". */ + t->reply = ovsdb_txn_forward_steal_reply(t->txn_forward); + ovsdb_txn_forward_destroy(t->db, t->txn_forward); + t->txn_forward = NULL; + ovsdb_trigger_complete(t); return false; } diff --git a/ovsdb/trigger.h b/ovsdb/trigger.h index 79af7f6be..d060c72e5 100644 --- a/ovsdb/trigger.h +++ b/ovsdb/trigger.h @@ -22,26 +22,34 @@ struct ovsdb; /* Triggers have the following states: * - * - Initialized (reply == NULL, progress == NULL): Executing the trigger - * can keep it in the initialized state, if it has a "wait" condition that - * isn't met. Executing the trigger can also yield an error, in which - * case it transitions to "complete". Otherwise, execution yields a - * transaction, which the database attempts to commit. If the transaction - * completes immediately and synchronously, then the trigger transitions - * to the "complete" state. If the transaction requires some time to - * complete, it transitions to the "committing" state. + * - Initialized (reply == NULL, progress == NULL, txn_forward == NULL): + * Executing the trigger can keep it in the initialized state, if it has a + * "wait" condition that isn't met. Executing the trigger can also yield + * an error, in which case it transitions to "complete". Otherwise, + * execution yields a transaction, which the database attempts to commit. + * If the transaction completes immediately and synchronously, then the + * trigger transitions to the "complete" state. If the transaction + * requires some time to complete, it transitions to the "committing" + * state. If the transaction can not be completed locally due to + * read-only restrictions and transaction forwarding is enabled, starts + * forwarding and transitions to the "forwarding" state. * - * - Committing (reply != NULL, progress != NULL): The transaction is - * committing. If it succeeds, or if it fails permanently, then the - * trigger transitions to "complete". If it fails temporarily - * (e.g. because someone else committed to cluster-based storage before we - * did), then we transition back to "initialized" to try again. + * - Committing (reply != NULL, progress != NULL, txn_forward == NULL): + * The transaction is committing. If it succeeds, or if it fails + * permanently, then the trigger transitions to "complete". If it fails + * temporarily (e.g. because someone else committed to cluster-based + * storage before we did), then we transition back to "initialized" to + * try again. * - * - Complete (reply != NULL, progress == NULL): The transaction is done - * and either succeeded or failed. + * - Forwarding (reply == NULL, progress == NULL, txn_forward != NULL): + * Transaction is forwarded. Either it succeeds or it fails, the trigger + * transitions to "complete". + * + * - Complete (reply != NULL, progress == NULL, txn_forward == NULL): + * The transaction is done and either succeeded or failed. */ struct ovsdb_trigger { - /* In "initialized" or "committing" state, in db->triggers. + /* In "initialized", "committing" or "forwarding" state, in db->triggers. * In "complete", in session->completions. */ struct ovs_list node; struct ovsdb_session *session; /* Session that owns this trigger. */ @@ -49,6 +57,7 @@ struct ovsdb_trigger { struct jsonrpc_msg *request; /* Database request. */ struct jsonrpc_msg *reply; /* Result (null if none yet). */ struct ovsdb_txn_progress *progress; + struct ovsdb_txn_forward *txn_forward; /* Tracks transaction forwarding. */ long long int created; /* Time created. */ long long int timeout_msec; /* Max wait duration. */ bool read_only; /* Database is in read only mode. */ diff --git a/tests/ovsdb-server.at b/tests/ovsdb-server.at index ba1b369c1..ac243d6a7 100644 --- a/tests/ovsdb-server.at +++ b/tests/ovsdb-server.at @@ -3,10 +3,13 @@ AT_BANNER([OVSDB -- ovsdb-server transactions (Unix sockets)]) m4_define([OVSDB_SERVER_SHUTDOWN], [OVS_APP_EXIT_AND_WAIT_BY_TARGET([ovsdb-server], [ovsdb-server.pid])]) +m4_define([OVSDB_SERVER_SHUTDOWN_N], + [cp pid$1 savepid$1 + AT_CHECK([ovs-appctl -t "`pwd`"/unixctl$1 -e exit], [0], [ignore], [ignore]) + OVS_WAIT_WHILE([kill -0 `cat savepid$1`], [kill `cat savepid$1`])]) + m4_define([OVSDB_SERVER_SHUTDOWN2], - [cp pid2 savepid2 - AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 -e exit], [0], [ignore], [ignore]) - OVS_WAIT_WHILE([kill -0 `cat savepid2`], [kill `cat savepid2`])]) + [OVSDB_SERVER_SHUTDOWN_N([2])]) # OVSDB_CHECK_EXECUTION(TITLE, SCHEMA, TRANSACTIONS, OUTPUT, [KEYWORDS]) # @@ -1412,6 +1415,82 @@ m4_define([OVSDB_CHECK_EXECUTION], EXECUTION_EXAMPLES +AT_BANNER([OVSDB -- ovsdb-server relay]) + +# OVSDB_CHECK_EXECUTION(TITLE, SCHEMA, TRANSACTIONS, OUTPUT, [KEYWORDS]) +# +# Creates a database with the given SCHEMA and starts an ovsdb-server on +# it. Also starts a daisy chain of ovsdb-servers in relay mode where the +# first relay server is connected to the main non-relay ovsdb-server. +# +# Runs each of the TRANSACTIONS (which should be a quoted list of +# quoted strings) against one of relay servers in the middle with +# ovsdb-client one at a time. The server executes read-only transactions +# and forwards rest of them to the previous ovsdb-server in a chain. +# The main ovsdb-server executes 'write' transactions. Transaction +# reply with data updates propagates back through the chain to all +# the servers and the client. +# +# main relay relay relay relay relay +# server1 <-- server2 <-- server3 <-- server4 <-- server5 <-- server6 +# ^ +# | +# ovsdb-client +# +# Checks that the overall output is OUTPUT, but UUIDs in the output +# are replaced by markers of the form where N is a number. The +# first unique UUID is replaced by <0>, the next by <1>, and so on. +# If a given UUID appears more than once it is always replaced by the +# same marker. +# +# Checks that the dump of all databases is the same. +# +# TITLE is provided to AT_SETUP and KEYWORDS to AT_KEYWORDS. +m4_define([OVSDB_CHECK_EXECUTION], + [AT_SETUP([$1]) + AT_KEYWORDS([ovsdb server tcp relay $5]) + n_servers=6 + target=4 + $2 > schema + schema_name=`ovsdb-tool schema-name schema` + AT_CHECK([ovsdb-tool create db1 schema], [0], [stdout], [ignore]) + + on_exit 'kill `cat *.pid`' + AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server1.log dnl + --pidfile --remote=punix:db1.sock db1 + ], [0], [ignore], [ignore]) + + for i in $(seq 2 ${n_servers}); do + AT_CHECK([ovsdb-server --detach --no-chdir dnl + --log-file=ovsdb-server$i.log dnl + --pidfile=${i}.pid --remote=punix:db${i}.sock dnl + --unixctl=unixctl${i} -vjsonrpc:file:dbg dnl + relay:${schema_name}:unix:db$((i-1)).sock + ], [0], [ignore], [ignore]) + done + + m4_foreach([txn], [$3], + [AT_CHECK([ovsdb-client transact unix:db${target}.sock 'txn'], [0], + [stdout], [ignore]) + cat stdout >> output + ]) + + AT_CHECK([uuidfilt output], [0], [$4], [ignore]) + + AT_CHECK([ovsdb-client dump unix:db1.sock], [0], [stdout], [ignore]) + for i in $(seq 2 ${n_servers}); do + OVS_WAIT_UNTIL([ovsdb-client dump unix:db${i}.sock > dump${i}; dnl + diff stdout dump${i}]) + done + + OVSDB_SERVER_SHUTDOWN + for i in $(seq 2 ${n_servers}); do + OVSDB_SERVER_SHUTDOWN_N([$i]) + done + AT_CLEANUP]) + +EXECUTION_EXAMPLES + AT_BANNER([OVSDB -- ovsdb-server replication]) # OVSDB_CHECK_EXECUTION(TITLE, SCHEMA, TRANSACTIONS, OUTPUT, [KEYWORDS]) From patchwork Sat Jun 12 02:00:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491230 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21CK4YFMz9sVm for ; Sat, 12 Jun 2021 12:01:01 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 4B014415D8; Sat, 12 Jun 2021 02:00:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O3n52T5rebkl; Sat, 12 Jun 2021 02:00:57 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 9CB73421D3; Sat, 12 Jun 2021 02:00:56 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C350AC0026; Sat, 12 Jun 2021 02:00:54 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3C772C000E for ; Sat, 12 Jun 2021 02:00:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 47A3C60B4B for ; Sat, 12 Jun 2021 02:00:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 99gYjjwS2E06 for ; Sat, 12 Jun 2021 02:00:33 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4661B60B4D for ; Sat, 12 Jun 2021 02:00:33 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id D96D41BF204; Sat, 12 Jun 2021 02:00:30 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:06 +0200 Message-Id: <20210612020008.3944088-8-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 7/9] ovsdb: relay: Reflect connection status in _Server database. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" It might be important for clients to know that relay lost connection with the relay remote, so they could re-connect to other relay. Signed-off-by: Ilya Maximets --- ovsdb/_server.xml | 17 +++++++++-------- ovsdb/ovsdb-server.c | 3 ++- ovsdb/relay.c | 34 ++++++++++++++++++++++++++++++++++ ovsdb/relay.h | 4 ++++ 4 files changed, 49 insertions(+), 9 deletions(-) diff --git a/ovsdb/_server.xml b/ovsdb/_server.xml index 414be6715..b4606b25b 100644 --- a/ovsdb/_server.xml +++ b/ovsdb/_server.xml @@ -70,6 +70,15 @@ case of a relay database - until it connects to the relay source. + + True if the database is connected to its storage. A standalone database + is always connected. A clustered database is connected if the server is + in contact with a majority of its cluster. A relay database is connected + if the server is in contact with the relay source, i.e. is connected to + the server it syncs from. An unconnected database cannot be modified and + its data might be unavailable or stale. + +

These columns are most interesting and in some cases only relevant for @@ -77,14 +86,6 @@ column is clustered.

- - True if the database is connected to its storage. A standalone or - active-backup database is always connected. A clustered database is - connected if the server is in contact with a majority of its cluster. - An unconnected database cannot be modified and its data might be - unavailable or stale. - - True if the database is the leader in its cluster. For a standalone or active-backup database, this is always true. Always false for relay. diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index 77b1fbe40..cdd6cf7fd 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -1190,7 +1190,8 @@ update_database_status(struct ovsdb_row *row, struct db *db) ovsdb_util_write_string_column(row, "model", db->db->is_relay ? "relay" : ovsdb_storage_get_model(db->db->storage)); ovsdb_util_write_bool_column(row, "connected", - ovsdb_storage_is_connected(db->db->storage)); + db->db->is_relay ? ovsdb_relay_is_connected(db->db) + : ovsdb_storage_is_connected(db->db->storage)); ovsdb_util_write_bool_column(row, "leader", db->db->is_relay ? false : ovsdb_storage_is_leader(db->db->storage)); ovsdb_util_write_uuid_column(row, "cid", diff --git a/ovsdb/relay.c b/ovsdb/relay.c index ef689c649..4a8f5c206 100644 --- a/ovsdb/relay.c +++ b/ovsdb/relay.c @@ -31,6 +31,7 @@ #include "ovsdb-error.h" #include "row.h" #include "table.h" +#include "timeval.h" #include "transaction.h" #include "transaction-forward.h" #include "util.h" @@ -47,8 +48,36 @@ struct relay_ctx { struct ovsdb_schema *new_schema; schema_change_callback schema_change_cb; void *schema_change_aux; + + long long int last_connected; }; +#define RELAY_MAX_RECONNECTION_MS 30000 + +/* Reports if the database connected to the relay source and functional, i.e. + * it actively monitors the source and is able to forward transactions. */ +bool +ovsdb_relay_is_connected(struct ovsdb *db) +{ + struct relay_ctx *ctx = shash_find_data(&relay_dbs, db->name); + + if (!ctx || !ovsdb_cs_is_alive(ctx->cs)) { + return false; + } + + if (ovsdb_cs_may_send_transaction(ctx->cs)) { + return true; + } + + /* Trying to avoid connection state flapping by delaying report for + * upper layer and giving ovsdb-cs some time to reconnect. */ + if (time_msec() - ctx->last_connected < RELAY_MAX_RECONNECTION_MS) { + return true; + } + + return false; +} + static struct json * ovsdb_relay_compose_monitor_request(const struct json *schema_json, void *ctx_) { @@ -118,6 +147,7 @@ ovsdb_relay_add_db(struct ovsdb *db, const char *remote, ctx->schema_change_aux = schema_change_aux; ctx->db = db; ctx->cs = ovsdb_cs_create(db->name, 3, &relay_cs_ops, ctx); + ctx->last_connected = 0; shash_add(&relay_dbs, db->name, ctx); ovsdb_cs_set_leader_only(ctx->cs, false); ovsdb_cs_set_remote(ctx->cs, remote, true); @@ -302,6 +332,10 @@ ovsdb_relay_run(void) ovsdb_txn_forward_run(ctx->db, ctx->cs); ovsdb_cs_run(ctx->cs, &events); + if (ovsdb_cs_may_send_transaction(ctx->cs)) { + ctx->last_connected = time_msec(); + } + struct ovsdb_cs_event *event; LIST_FOR_EACH_POP (event, list_node, &events) { if (!ctx->db) { diff --git a/ovsdb/relay.h b/ovsdb/relay.h index 68586e9db..390ea70c8 100644 --- a/ovsdb/relay.h +++ b/ovsdb/relay.h @@ -17,6 +17,8 @@ #ifndef OVSDB_RELAY_H #define OVSDB_RELAY_H 1 +#include + struct json; struct ovsdb; struct ovsdb_schema; @@ -31,4 +33,6 @@ void ovsdb_relay_del_db(struct ovsdb *); void ovsdb_relay_run(void); void ovsdb_relay_wait(void); +bool ovsdb_relay_is_connected(struct ovsdb *); + #endif /* OVSDB_RELAY_H */ From patchwork Sat Jun 12 02:00:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491229 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21CF67Cqz9sVm for ; Sat, 12 Jun 2021 12:00:57 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E7D4341CB4; Sat, 12 Jun 2021 02:00:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tXJNNmCIegve; Sat, 12 Jun 2021 02:00:55 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2D235419FC; Sat, 12 Jun 2021 02:00:54 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C9F99C000E; Sat, 12 Jun 2021 02:00:53 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 10F53C000B for ; Sat, 12 Jun 2021 02:00:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 3C92F41A02 for ; Sat, 12 Jun 2021 02:00:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9JHbyxK1F9Vi for ; Sat, 12 Jun 2021 02:00:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp2.osuosl.org (Postfix) with ESMTPS id 269AF41490 for ; Sat, 12 Jun 2021 02:00:35 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id A4A771BF203; Sat, 12 Jun 2021 02:00:33 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:07 +0200 Message-Id: <20210612020008.3944088-9-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 8/9] ovsdb: Make clients aware of relay service model. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Clients needs to re-connect from the relay that has no connection with the database source. Also, relay acts similarly to the follower from a clustered model from the consistency point of view, so it's not suitable for leader-only connections. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara Acked-by: Mark D. Gray --- lib/ovsdb-cs.c | 15 ++++++++++++++- ovsdb/ovsdb-client.c | 2 +- python/ovs/db/idl.py | 16 ++++++++++++++++ 3 files changed, 31 insertions(+), 2 deletions(-) diff --git a/lib/ovsdb-cs.c b/lib/ovsdb-cs.c index 911b71dd4..fb3fdd006 100644 --- a/lib/ovsdb-cs.c +++ b/lib/ovsdb-cs.c @@ -1897,8 +1897,8 @@ ovsdb_cs_check_server_db__(struct ovsdb_cs *cs) bool ok = false; const char *model = server_column_get_string(db_row, COL_MODEL, ""); const char *schema = server_column_get_string(db_row, COL_SCHEMA, NULL); + bool connected = server_column_get_bool(db_row, COL_CONNECTED, false); if (!strcmp(model, "clustered")) { - bool connected = server_column_get_bool(db_row, COL_CONNECTED, false); bool leader = server_column_get_bool(db_row, COL_LEADER, false); uint64_t index = server_column_get_int(db_row, COL_INDEX, 0); @@ -1918,6 +1918,19 @@ ovsdb_cs_check_server_db__(struct ovsdb_cs *cs) cs->min_index = index; ok = true; } + } else if (!strcmp(model, "relay")) { + if (!schema) { + VLOG_INFO("%s: relay database server has not yet connected to the " + "relay source; trying another server", server_name); + } else if (!connected) { + VLOG_INFO("%s: relay database server is disconnected from the " + "relay source; trying another server", server_name); + } else if (cs->leader_only) { + VLOG_INFO("%s: relay database server cannot be a leader; " + "trying another server", server_name); + } else { + ok = true; + } } else { if (!schema) { VLOG_INFO("%s: missing database schema", server_name); diff --git a/ovsdb/ovsdb-client.c b/ovsdb/ovsdb-client.c index ffa8f8df2..f1b8d6491 100644 --- a/ovsdb/ovsdb-client.c +++ b/ovsdb/ovsdb-client.c @@ -716,7 +716,7 @@ should_stay_connected(const char *server, const char *database, return false; } - if (strcmp(parse_string_column(row, "model"), "clustered")) { + if (!strcmp(parse_string_column(row, "model"), "standalone")) { /* Always accept standalone databases. */ return true; } diff --git a/python/ovs/db/idl.py b/python/ovs/db/idl.py index 889cf3431..94bf13b1c 100644 --- a/python/ovs/db/idl.py +++ b/python/ovs/db/idl.py @@ -38,6 +38,7 @@ OVSDB_UPDATE = 0 OVSDB_UPDATE2 = 1 CLUSTERED = "clustered" +RELAY = "relay" Notice = collections.namedtuple('Notice', ('event', 'row', 'updates')) @@ -797,6 +798,21 @@ class Idl(object): 'trying another server' % session_name) return False self._min_index = database.index[0] + elif database.model == RELAY: + if not database.schema: + vlog.info('%s: relay database server has not yet connected ' + 'to the relay source; trying another server' + % session_name) + return False + if not database.connected: + vlog.info('%s: relay database server is disconnected ' + 'from the relay source; trying another server' + % session_name) + return False + if self.leader_only: + vlog.info('%s: relay database server cannot be a leader; ' + 'trying another server' % session_name) + return False return True From patchwork Sat Jun 12 02:00:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1491232 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G21Ct6pXzz9sVm for ; Sat, 12 Jun 2021 12:01:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id A296940669; Sat, 12 Jun 2021 02:01:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ks-8C0WvdBWg; Sat, 12 Jun 2021 02:01:27 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 6FFD441AA2; Sat, 12 Jun 2021 02:01:26 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2AE0AC0011; Sat, 12 Jun 2021 02:01:26 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id D68A7C000B for ; Sat, 12 Jun 2021 02:01:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 31C82415D3 for ; Sat, 12 Jun 2021 02:00:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kIeNUrvJDIdJ for ; Sat, 12 Jun 2021 02:00:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by smtp4.osuosl.org (Postfix) with ESMTPS id EEDAC41622 for ; Sat, 12 Jun 2021 02:00:38 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 75B3A1BF204; Sat, 12 Jun 2021 02:00:36 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 12 Jun 2021 04:00:08 +0200 Message-Id: <20210612020008.3944088-10-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210612020008.3944088-1-i.maximets@ovn.org> References: <20210612020008.3944088-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH v2 9/9] docs: Add documentation for ovsdb relay mode. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Main documentation for the service model and tutorial with the use case and configuration examples. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- Documentation/automake.mk | 1 + Documentation/ref/ovsdb.7.rst | 62 ++++++++++++-- Documentation/topics/index.rst | 1 + Documentation/topics/ovsdb-relay.rst | 124 +++++++++++++++++++++++++++ NEWS | 3 + ovsdb/ovsdb-server.1.in | 27 +++--- 6 files changed, 200 insertions(+), 18 deletions(-) create mode 100644 Documentation/topics/ovsdb-relay.rst diff --git a/Documentation/automake.mk b/Documentation/automake.mk index bc30f94c5..213d9c867 100644 --- a/Documentation/automake.mk +++ b/Documentation/automake.mk @@ -52,6 +52,7 @@ DOC_SOURCE = \ Documentation/topics/networking-namespaces.rst \ Documentation/topics/openflow.rst \ Documentation/topics/ovs-extensions.rst \ + Documentation/topics/ovsdb-relay.rst \ Documentation/topics/ovsdb-replication.rst \ Documentation/topics/porting.rst \ Documentation/topics/record-replay.rst \ diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst index e4f1bf766..a5b8a9c33 100644 --- a/Documentation/ref/ovsdb.7.rst +++ b/Documentation/ref/ovsdb.7.rst @@ -121,13 +121,14 @@ schema checksum from a schema or database file, respectively. Service Models ============== -OVSDB supports three service models for databases: **standalone**, -**active-backup**, and **clustered**. The service models provide different -compromises among consistency, availability, and partition tolerance. They -also differ in the number of servers required and in terms of performance. The -standalone and active-backup database service models share one on-disk format, -and clustered databases use a different format, but the OVSDB programs work -with both formats. ``ovsdb(5)`` documents these file formats. +OVSDB supports four service models for databases: **standalone**, +**active-backup**, **relay** and **clustered**. The service models provide +different compromises among consistency, availability, and partition tolerance. +They also differ in the number of servers required and in terms of performance. +The standalone and active-backup database service models share one on-disk +format, and clustered databases use a different format, but the OVSDB programs +work with both formats. ``ovsdb(5)`` documents these file formats. Relay +databases has no on-disk storage. RFC 7047, which specifies the OVSDB protocol, does not mandate or specify any particular service model. @@ -406,6 +407,50 @@ following consequences: that the client previously read. The OVSDB client library in Open vSwitch uses this feature to avoid servers with stale data. +Relay Service Model +------------------- + +A **relay** database is a way to scale out read-mostly access to the +existing database working in any service model including relay. + +Relay database creates and maintains an OVSDB connection with other OVSDB +server. It uses this connection to maintain in-memory copy of the remote +database (a.k.a. the ``relay source``) keeping the copy up-to-date as the +database content changes on relay source in the real time. + +The purpose of relay server is to scale out the number of database clients. +Read-only transactions and monitor requests are fully handled by the relay +server itself. For the transactions that requests database modifications, +relay works as a proxy between the client and the relay source, i.e. it +forwards transactions and replies between them. + +Compared to a clustered and active-backup models, relay service model provides +read and write access to the database similarly to a clustered database (and +even more scalable), but with generally insignificant performance overhead of +an active-backup model. At the same time it doesn't increase availability that +needs to be covered by the service model of the relay source. + +Relay database has no on-disk storage and therefore cannot be converted to +any other service model. + +If there is already a database started in any service model, to start a relay +database server use ``ovsdb-server relay::``, where +```` is the database name as specified in the schema of the database +that existing server runs, and ```` is an OVSDB connection method +(see `Connection Methods`_ below) that connects to the existing database +server. ```` could contain a comma-separated list of connection +methods, e.g. to connect to any server of the clustered database. +Multiple relay servers could be started for the same relay source. + +Since the way how relay handles read and write transactions is very similar +to the clustered model where "cluster" means "set or relay servers connected +to the same relay source", "follower" means "relay server" and the "leader" +means "relay source", same consistency consequences as for the clustered +model applies to relay as well (See `Understanding Cluster Consistency`_ +above). + +Open vSwitch 2.16 introduced support for relay service model. + Database Replication ==================== @@ -414,7 +459,8 @@ Replication, in this context, means to make, and keep up-to-date, a read-only copy of the contents of a database (the ``replica``). One use of replication is to keep an up-to-date backup of a database. A replica used solely for backup would not need to support clients of its own. A set of replicas that do -serve clients could be used to scale out read access to the primary database. +serve clients could be used to scale out read access to the primary database, +however `Relay Service Model`_ is more suitable for that purpose. A database replica is set up in the same way as a backup server in an active-backup pair, with the difference that the replica is never promoted to diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst index 0036567eb..d8ccbd757 100644 --- a/Documentation/topics/index.rst +++ b/Documentation/topics/index.rst @@ -44,6 +44,7 @@ OVS openflow bonding networking-namespaces + ovsdb-relay ovsdb-replication dpdk/index windows diff --git a/Documentation/topics/ovsdb-relay.rst b/Documentation/topics/ovsdb-relay.rst new file mode 100644 index 000000000..40d294c55 --- /dev/null +++ b/Documentation/topics/ovsdb-relay.rst @@ -0,0 +1,124 @@ +.. + Copyright 2021, Red Hat, Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +=============================== +Scaling OVSDB Access With Relay +=============================== + +Open vSwitch 2.16 introduced support for OVSDB Relay mode with the goal to +increase database scalability for a big deployments. Mainly, OVN (Open Virtual +Network) Southbound Database deployments. This document describes the main +concept and provides the configuration examples. + +What is OVSDB Relay? +-------------------- + +Relay is a database service model in which one ``ovsdb-server`` (``relay``) +connects to another standalone or clustered database server +(``relay source``) and maintains in-memory copy of its data, receiving +all the updates via this OVSDB connection. Relay server handles all the +read-only requests (monitors and transactions) on its own and forwards all the +transactions that requires database modifications to the relay source. + +Why is this needed? +------------------- + +Some OVN deployment could have hundreds or even thousands nodes, on each of +these nodes there is an ovn-controller, which is connected to the +OVN_Southbound database that is served by a standalone or clustered OVSDB. +Standalone database is handled by a single ovsdb-server process and clustered +could consist of 3 to 5 ovsdb-server processes. For the clustered database, +higher number of servers may significantly increase transaction latency due +to necessity for these servers to reach consensus. So, in the end limited +number of ovsdb-server processes serves ever growing number of clients and this +leads to performance issues. + +Read-only access could be scaled up with OVSDB replication on top of +active-backup service model, but ovn-controller is a read-mostly client, not +a read-only, i.e. it needs to execute write transactions from time to time. +Here relay service model comes into play. + +2-Tier Deployment +----------------- + +Solution for the scaling issue could look like a 2-tier deployment, where +a set of relay servers is connected to the main database cluster +(OVN_Southbound) and clients (ovn-conrtoller) connected to these relay +servers:: + + 172.16.0.1 + +--------------------+ +----+ ovsdb-relay-1 +--+---+ client-1 + | | | | + | Clustered | | +---+ client-2 + | Database | | ... + | | | +---+ client-N + | 10.0.0.2 | | + | ovsdb-server-2 | | 172.16.0.2 + | + + | +----+ ovsdb-relay-2 +--+---+ client-N+1 + | | | | | | + | | + +---+ +---+ client-N+2 + | | 10.0.0.1 | | ... + | | ovsdb-server-1 | | +---+ client-2N + | | + | | + | | | | | + | + + | + ... ... ... ... ... + | ovsdb-server-3 | | + | 10.0.0.3 | | +---+ client-KN-1 + | | | 172.16.0.K | + +--------------------+ +----+ ovsdb-relay-K +--+---+ client-KN + +In practice, the picture might look a bit more complex, because all relay +servers might connect to any member of a main cluster and clients might +connect to any relay server of their choice. + +Assuming that servers of a main cluster started like this:: + + $ ovsdb-server --remote=ptcp:10.0.0.1:6642 ovn-sb-1.db + +The same for other two servers. In this case relay servers could be +started like this:: + + $ REMOTES=tcp:10.0.0.1:6642,tcp:10.0.0.2:6642,tcp:10.0.0.3:6642 + $ ovsdb-server --remote=ptcp:172.16.0.1:6642 relay:OVN_Southbound:$REMOTES + $ ... + $ ovsdb-server --remote=ptcp:172.16.0.K:6642 relay:OVN_Southbound:$REMOTES + +Every relay server could connect to any of the cluster members of their choice, +fairness of load distribution is achieved by shuffling remotes. + +For the actual clients, they could be configured to connect to any of the +relay servers. For ovn-controllers the configuration could look like this:: + + $ REMOTES=tcp:172.16.0.1:6642,...,tcp:172.16.0.K:6642 + $ ovs-vsctl set Open_vSwitch . external-ids:ovn-remote=$REMOTES + +Setup like this allows the system to serve ``K * N`` clients while having only +``K`` actual connections on the main clustered database keeping it in a +stable state. + +It's also possible to create multi-tier deployments by connecting one set +of relay servers to another (smaller) set of relay servers, or even create +tree-like structures by the cost of increased latency for write transactions, +because they will be forwarded multiple times. diff --git a/NEWS b/NEWS index ebba17b22..391b0abba 100644 --- a/NEWS +++ b/NEWS @@ -1,6 +1,9 @@ Post-v2.15.0 --------------------- - OVSDB: + * Introduced new database service model - "relay". Targeted to scale out + read-mostly access (ovn-controller) to existing databases. + For more information: ovsdb(7) and Documentation/topics/ovsdb-relay.rst * New command line options --record/--replay for ovsdb-server and ovsdb-client to record and replay all the incoming transactions, monitors, etc. More datails in Documentation/topics/record-replay.rst. diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in index fdd52e8f6..dac0f02cb 100644 --- a/ovsdb/ovsdb-server.1.in +++ b/ovsdb/ovsdb-server.1.in @@ -10,6 +10,7 @@ ovsdb\-server \- Open vSwitch database server .SH SYNOPSIS \fBovsdb\-server\fR [\fIdatabase\fR]\&... +[\fIrelay:schema_name:remote\fR]\&... [\fB\-\-remote=\fIremote\fR]\&... [\fB\-\-run=\fIcommand\fR] .so lib/daemon-syn.man @@ -35,12 +36,15 @@ For an introduction to OVSDB and its implementation in Open vSwitch, see \fBovsdb\fR(7). .PP Each OVSDB file may be specified on the command line as \fIdatabase\fR. -If none is specified, the default is \fB@DBDIR@/conf.db\fR. The database -files must already have been created and initialized using, for -example, \fBovsdb\-tool\fR's \fBcreate\fR, \fBcreate\-cluster\fR, or -\fBjoin\-cluster\fR command. +Relay databases may be specified on the command line as +\fIrelay:schema_name:remote\fR. For a detailed description of relay database +argument, see \fBovsdb\fR(7). +If none of database files or relay databases is specified, the default is +\fB@DBDIR@/conf.db\fR. The database files must already have been created and +initialized using, for example, \fBovsdb\-tool\fR's \fBcreate\fR, +\fBcreate\-cluster\fR, or \fBjoin\-cluster\fR command. .PP -This OVSDB implementation supports standalone, active-backup, and +This OVSDB implementation supports standalone, active-backup, relay and clustered database service models, as well as database replication. See the Service Models section of \fBovsdb\fR(7) for more information. .PP @@ -50,7 +54,9 @@ successfully join a cluster (if the database file is freshly created with \fBovsdb\-tool join\-cluster\fR) or connect to a cluster that it has already joined. Use \fBovsdb\-client wait\fR (see \fBovsdb\-client\fR(1)) to wait until the server has successfully -joined and connected to a cluster. +joined and connected to a cluster. The same is true for relay databases. +Same commands could be used to wait for a relay database to connect to +the relay source (remote). .PP In addition to user-specified databases, \fBovsdb\-server\fR version 2.9 and later also always hosts a built-in database named @@ -243,10 +249,11 @@ not list remotes added indirectly because they were read from the database by configuring a \fBdb:\fIdb\fB,\fItable\fB,\fIcolumn\fR remote. . -.IP "\fBovsdb\-server/add\-db \fIdatabase\fR" -Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR. The database -file must already have been created and initialized using, for example, -\fBovsdb\-tool create\fR. +.IP "\fBovsdb\-server/add\-db \fIdatabase\fR +Adds the \fIdatabase\fR to the running \fBovsdb\-server\fR. \fIdatabase\fR +could be a database file or a relay description in the following format: +\fIrelay:schema_name:remote\fR. The database file must already have been +created and initialized using, for example, \fBovsdb\-tool create\fR. . .IP "\fBovsdb\-server/remove\-db \fIdatabase\fR" Removes \fIdatabase\fR from the running \fBovsdb\-server\fR. \fIdatabase\fR