From patchwork Sat May 1 00:55:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1472586 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FX9m66H5vz9sSC for ; Sat, 1 May 2021 10:56:22 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 966184328D; Sat, 1 May 2021 00:56:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Jjq-H-JIdm1S; Sat, 1 May 2021 00:56:15 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id 404574026E; Sat, 1 May 2021 00:56:14 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 19C3EC000E; Sat, 1 May 2021 00:56:14 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 07CF6C000E for ; Sat, 1 May 2021 00:56:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A14E641CF8 for ; Sat, 1 May 2021 00:56:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H92DxMKYlLVU for ; Sat, 1 May 2021 00:56:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by smtp4.osuosl.org (Postfix) with ESMTPS id 13C5E41BB8 for ; Sat, 1 May 2021 00:56:08 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 3C31D240005; Sat, 1 May 2021 00:56:06 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 1 May 2021 02:55:42 +0200 Message-Id: <20210501005548.3071269-2-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210501005548.3071269-1-i.maximets@ovn.org> References: <20210501005548.3071269-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH 1/7] ovsdb: Add support for transaction forwarding to the replication mode. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Current version of ovsdb replication allows to scale out read-only access to the primary database. However, many clients are not read-only but read-mostly. For example, ovn-controller. In order to scale out database access for this case ovsdb-server need to process transactions that are not read-only. Replica is not allowed to do that, i.e. not allowed to modify the database, but it can act like a proxy and forward transactions that includes database modifications to the primary server and forward replies back to client. At the same time it may serve read-only transactions and monitor requests by itself greatly reducing the load on primary server. To support that new command line option '--enable-txn-forward' added to ovsdb-server. This configuration will slightly increase transaction latency, but it's not very important for read-mostly use cases. With this change instead of creating a trigger to commit the transaction, ovsdb-server will create a trigger for transaction forwarding. Later, replication_run() will send all new transactions to the replication source. Once transaction reply received from the replication source, replication module will update the state of the transaction forwarding with the reply. After that, trigger_run() will complete the trigger and jsonrpc_server_run() will send the reply back to the client. Since transaction reply from the replication source will be received after all the updates, client will receive all the updates before receiving the transaction reply as it is in a normal scenario without transaction forwarding. Signed-off-by: Ilya Maximets --- Documentation/ref/ovsdb.7.rst | 8 ++ NEWS | 4 + ovsdb/automake.mk | 2 + ovsdb/execution.c | 14 ++- ovsdb/jsonrpc-server.c | 61 +++++++++-- ovsdb/jsonrpc-server.h | 6 +- ovsdb/ovsdb-server.c | 54 +++++++--- ovsdb/ovsdb.h | 2 +- ovsdb/replication.c | 34 ++++++- ovsdb/transaction-forward.c | 187 ++++++++++++++++++++++++++++++++++ ovsdb/transaction-forward.h | 42 ++++++++ ovsdb/trigger.c | 62 +++++++++-- ovsdb/trigger.h | 45 ++++---- tests/ovsdb-server.at | 64 +++++++++++- tests/test-ovsdb.c | 2 +- 15 files changed, 526 insertions(+), 61 deletions(-) create mode 100644 ovsdb/transaction-forward.c create mode 100644 ovsdb/transaction-forward.h diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst index e4f1bf766..164f40b7b 100644 --- a/Documentation/ref/ovsdb.7.rst +++ b/Documentation/ref/ovsdb.7.rst @@ -424,6 +424,14 @@ A database can have multiple replicas. Open vSwitch 2.6 introduced support for database replication. +Open vSwitch 2.16 introduced transaction forwarding support for database +replication. It can be enabled by passing ``--enable-txn-forward`` command +line argument to ``ovsdb-server``. A replica with enabled transaction +forwarding can be used to scale out read-mostly access to the primary database. +In this case replica will work as a proxy between the client and the primary +database for transactions that needs to modify the database, while executing +read-only transactions and serving monitor requests by itself. + Connection Methods ================== diff --git a/NEWS b/NEWS index 95cf922aa..790f93af9 100644 --- a/NEWS +++ b/NEWS @@ -1,5 +1,9 @@ Post-v2.15.0 --------------------- + - OVSDB: + * New command line argument '--enable-txn-forward' for ovsdb-server in + replication mode that allows to forward transactions that includes + database modifications to the primary database server. - In ovs-vsctl and vtep-ctl, the "find" command now accept new operators {in} and {not-in}. - Userspace datapath: diff --git a/ovsdb/automake.mk b/ovsdb/automake.mk index 446d6c136..5f5da26d4 100644 --- a/ovsdb/automake.mk +++ b/ovsdb/automake.mk @@ -46,6 +46,8 @@ ovsdb_libovsdb_la_SOURCES = \ ovsdb/trigger.h \ ovsdb/transaction.c \ ovsdb/transaction.h \ + ovsdb/transaction-forward.c \ + ovsdb/transaction-forward.h \ ovsdb/ovsdb-util.c \ ovsdb/ovsdb-util.h ovsdb_libovsdb_la_CFLAGS = $(AM_CFLAGS) diff --git a/ovsdb/execution.c b/ovsdb/execution.c index 3a0dad5d0..fb94c2071 100644 --- a/ovsdb/execution.c +++ b/ovsdb/execution.c @@ -99,7 +99,8 @@ lookup_executor(const char *name, bool *read_only) } /* On success, returns a transaction and stores the results to return to the - * client in '*resultsp'. + * client in '*resultsp'. If 'all_ops_read_only' is nonnull and all oparations + * in transaction are read-only operations, sets '*all_ops_read_only' to true. * * On failure, returns NULL. If '*resultsp' is nonnull, then it is the results * to return to the client. If '*resultsp' is null, then the execution failed @@ -111,7 +112,8 @@ ovsdb_execute_compose(struct ovsdb *db, const struct ovsdb_session *session, const struct json *params, bool read_only, const char *role, const char *id, long long int elapsed_msec, long long int *timeout_msec, - bool *durable, struct json **resultsp) + bool *durable, bool *all_ops_read_only, + struct json **resultsp) { struct ovsdb_execution x; struct ovsdb_error *error; @@ -120,6 +122,9 @@ ovsdb_execute_compose(struct ovsdb *db, const struct ovsdb_session *session, size_t i; *durable = false; + if (all_ops_read_only) { + *all_ops_read_only = true; + } if (params->type != JSON_ARRAY || !params->array.n || params->array.elems[0]->type != JSON_STRING @@ -210,6 +215,9 @@ ovsdb_execute_compose(struct ovsdb *db, const struct ovsdb_session *session, } break; } + if (!ro && all_ops_read_only) { + *all_ops_read_only = false; + } json_array_add(results, result); } while (json_array(results)->n < n_operations) { @@ -240,7 +248,7 @@ ovsdb_execute(struct ovsdb *db, const struct ovsdb_session *session, struct json *results; struct ovsdb_txn *txn = ovsdb_execute_compose( db, session, params, read_only, role, id, elapsed_msec, timeout_msec, - &durable, &results); + &durable, NULL, &results); if (!txn) { return results; } diff --git a/ovsdb/jsonrpc-server.c b/ovsdb/jsonrpc-server.c index 4e2dfc3d7..0ff4f7c32 100644 --- a/ovsdb/jsonrpc-server.c +++ b/ovsdb/jsonrpc-server.c @@ -57,7 +57,7 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); /* Sessions. */ static struct ovsdb_jsonrpc_session *ovsdb_jsonrpc_session_create( - struct ovsdb_jsonrpc_remote *, struct jsonrpc_session *, bool); + struct ovsdb_jsonrpc_remote *, struct jsonrpc_session *, bool, bool); static void ovsdb_jsonrpc_session_preremove_db(struct ovsdb_jsonrpc_remote *, struct ovsdb *); static void ovsdb_jsonrpc_session_run_all(struct ovsdb_jsonrpc_remote *); @@ -82,6 +82,8 @@ static void ovsdb_jsonrpc_session_send(struct ovsdb_jsonrpc_session *, struct jsonrpc_msg *); static void ovsdb_jsonrpc_session_set_readonly_all( struct ovsdb_jsonrpc_remote *remote, bool read_only); +static void ovsdb_jsonrpc_session_set_txn_forward_all( + struct ovsdb_jsonrpc_remote *remote, bool txn_forward); /* Triggers. */ static void ovsdb_jsonrpc_trigger_create(struct ovsdb_jsonrpc_session *, @@ -127,6 +129,8 @@ struct ovsdb_jsonrpc_server { unsigned int n_sessions; bool read_only; /* This server is does not accept any transactions that can modify the database. */ + bool txn_forward; /* This server is able to forward transactions + to another server. */ struct shash remotes; /* Contains "struct ovsdb_jsonrpc_remote *"s. */ }; @@ -139,6 +143,7 @@ struct ovsdb_jsonrpc_remote { struct ovs_list sessions; /* List of "struct ovsdb_jsonrpc_session"s. */ uint8_t dscp; bool read_only; + bool txn_forward; char *role; }; @@ -153,12 +158,13 @@ static void ovsdb_jsonrpc_server_del_remote(struct shash_node *); * The caller must call ovsdb_jsonrpc_server_add_db() for each database to * which 'server' should provide access. */ struct ovsdb_jsonrpc_server * -ovsdb_jsonrpc_server_create(bool read_only) +ovsdb_jsonrpc_server_create(bool read_only, bool txn_forward) { struct ovsdb_jsonrpc_server *server = xzalloc(sizeof *server); ovsdb_server_init(&server->up); shash_init(&server->remotes); server->read_only = read_only; + server->txn_forward = txn_forward; return server; } @@ -215,6 +221,7 @@ ovsdb_jsonrpc_default_options(const char *target) options->probe_interval = (stream_or_pstream_needs_probes(target) ? RECONNECT_DEFAULT_PROBE_INTERVAL : 0); + options->txn_forward = true; return options; } @@ -278,12 +285,14 @@ ovsdb_jsonrpc_server_add_remote(struct ovsdb_jsonrpc_server *svr, ovs_list_init(&remote->sessions); remote->dscp = options->dscp; remote->read_only = options->read_only; + remote->txn_forward = options->txn_forward; remote->role = nullable_xstrdup(options->role); shash_add(&svr->remotes, name, remote); if (!listener) { ovsdb_jsonrpc_session_create(remote, jsonrpc_session_open(name, true), - svr->read_only || remote->read_only); + svr->read_only || remote->read_only, + svr->txn_forward && remote->txn_forward); } return remote; } @@ -377,6 +386,22 @@ ovsdb_jsonrpc_server_set_read_only(struct ovsdb_jsonrpc_server *svr, } } +void +ovsdb_jsonrpc_server_set_txn_forward(struct ovsdb_jsonrpc_server *svr, + bool txn_forward) +{ + if (svr->txn_forward != txn_forward) { + svr->txn_forward = txn_forward; + + struct shash_node *node; + SHASH_FOR_EACH (node, &svr->remotes) { + struct ovsdb_jsonrpc_remote *remote = node->data; + + ovsdb_jsonrpc_session_set_txn_forward_all(remote, txn_forward); + } + } +} + void ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *svr) { @@ -394,8 +419,9 @@ ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *svr) struct jsonrpc_session *js; js = jsonrpc_session_open_unreliably(jsonrpc_open(stream), remote->dscp); - ovsdb_jsonrpc_session_create(remote, js, svr->read_only || - remote->read_only); + ovsdb_jsonrpc_session_create(remote, js, + svr->read_only || remote->read_only, + svr->txn_forward && remote->txn_forward); } else if (error != EAGAIN) { VLOG_WARN_RL(&rl, "%s: accept failed: %s", pstream_get_name(remote->listener), @@ -473,6 +499,10 @@ struct ovsdb_jsonrpc_session { /* Read only. */ bool read_only; /* When true, not allow to modify the database. */ + + /* Transaction forwarding. */ + bool txn_forward; /* When true, allow to forward incoming + transactions. */ }; static void ovsdb_jsonrpc_session_close(struct ovsdb_jsonrpc_session *); @@ -487,7 +517,8 @@ static void ovsdb_jsonrpc_session_got_notify(struct ovsdb_jsonrpc_session *, static struct ovsdb_jsonrpc_session * ovsdb_jsonrpc_session_create(struct ovsdb_jsonrpc_remote *remote, - struct jsonrpc_session *js, bool read_only) + struct jsonrpc_session *js, + bool read_only, bool txn_forward) { struct ovsdb_jsonrpc_session *s; @@ -500,6 +531,7 @@ ovsdb_jsonrpc_session_create(struct ovsdb_jsonrpc_remote *remote, s->js = js; s->js_seqno = jsonrpc_session_get_seqno(js); s->read_only = read_only; + s->txn_forward = txn_forward; remote->server->n_sessions++; @@ -599,8 +631,10 @@ static void ovsdb_jsonrpc_session_wait(struct ovsdb_jsonrpc_session *s) { jsonrpc_session_wait(s->js); + if (!jsonrpc_session_get_backlog(s->js)) { - if (ovsdb_jsonrpc_monitor_needs_flush(s)) { + if (ovsdb_jsonrpc_monitor_needs_flush(s) + || !ovs_list_is_empty(&s->up.completions)) { poll_immediate_wake(); } else { jsonrpc_session_recv_wait(s->js); @@ -686,6 +720,17 @@ ovsdb_jsonrpc_session_set_readonly_all(struct ovsdb_jsonrpc_remote *remote, } } +static void +ovsdb_jsonrpc_session_set_txn_forward_all(struct ovsdb_jsonrpc_remote *remote, + bool txn_forward) +{ + struct ovsdb_jsonrpc_session *s; + + LIST_FOR_EACH (s, node, &remote->sessions) { + s->txn_forward = txn_forward; + } +} + /* Sets the options for all of the JSON-RPC sessions managed by 'remote' to * 'options'. * @@ -1134,7 +1179,7 @@ ovsdb_jsonrpc_trigger_create(struct ovsdb_jsonrpc_session *s, struct ovsdb *db, t = xmalloc(sizeof *t); bool disconnect_all = ovsdb_trigger_init( &s->up, db, &t->trigger, request, time_msec(), s->read_only, - s->remote->role, jsonrpc_session_get_id(s->js)); + s->txn_forward, s->remote->role, jsonrpc_session_get_id(s->js)); t->id = json_clone(request->id); hmap_insert(&s->triggers, &t->hmap_node, hash); diff --git a/ovsdb/jsonrpc-server.h b/ovsdb/jsonrpc-server.h index e0653aa39..0a8ec63b1 100644 --- a/ovsdb/jsonrpc-server.h +++ b/ovsdb/jsonrpc-server.h @@ -24,7 +24,8 @@ struct shash; struct simap; struct uuid; -struct ovsdb_jsonrpc_server *ovsdb_jsonrpc_server_create(bool read_only); +struct ovsdb_jsonrpc_server *ovsdb_jsonrpc_server_create(bool read_only, + bool txn_forward); bool ovsdb_jsonrpc_server_add_db(struct ovsdb_jsonrpc_server *, struct ovsdb *); void ovsdb_jsonrpc_server_remove_db(struct ovsdb_jsonrpc_server *, @@ -36,6 +37,7 @@ struct ovsdb_jsonrpc_options { int max_backoff; /* Maximum reconnection backoff, in msec. */ int probe_interval; /* Max idle time before probing, in msec. */ bool read_only; /* Only read-only transactions are allowed. */ + bool txn_forward; /* Transaction forwarding is allowed. */ int dscp; /* Dscp value for manager connections */ char *role; /* Role, for role-based access controls */ }; @@ -72,6 +74,8 @@ void ovsdb_jsonrpc_server_wait(struct ovsdb_jsonrpc_server *); void ovsdb_jsonrpc_server_set_read_only(struct ovsdb_jsonrpc_server *, bool read_only); +void ovsdb_jsonrpc_server_set_txn_forward(struct ovsdb_jsonrpc_server *, + bool txn_forward); void ovsdb_jsonrpc_server_get_memory_usage(const struct ovsdb_jsonrpc_server *, struct simap *usage); diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index 29a2bace8..c9d21af1c 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -103,6 +103,7 @@ struct server_config { char **sync_from; char **sync_exclude; bool *is_backup; + bool *txn_forward; int *replication_probe_interval; struct ovsdb_jsonrpc_server *jsonrpc; }; @@ -126,7 +127,7 @@ static void parse_options(int argc, char *argvp[], struct sset *db_filenames, struct sset *remotes, char **unixctl_pathp, char **run_command, char **sync_from, char **sync_exclude, - bool *is_backup); + bool *is_backup, bool *txn_forward); OVS_NO_RETURN static void usage(void); static char *reconfigure_remotes(struct ovsdb_jsonrpc_server *, @@ -143,11 +144,12 @@ static void update_server_status(struct shash *all_dbs); static void save_config__(FILE *config_file, const struct sset *remotes, const struct sset *db_filenames, const char *sync_from, const char *sync_exclude, - bool is_backup); + bool is_backup, bool txn_forward); static void save_config(struct server_config *); static void load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames, char **sync_from, - char **sync_exclude, bool *is_backup); + char **sync_exclude, bool *is_backup, + bool *txn_forward); static void ovsdb_replication_init(const char *sync_from, const char *exclude, @@ -178,7 +180,8 @@ static void main_loop(struct server_config *config, struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs, struct unixctl_server *unixctl, struct sset *remotes, - struct process *run_process, bool *exiting, bool *is_backup) + struct process *run_process, bool *exiting, bool *is_backup, + bool *txn_forward) { char *remotes_error, *ssl_error; struct shash_node *node; @@ -209,6 +212,7 @@ main_loop(struct server_config *config, unixctl_server_run(unixctl); ovsdb_jsonrpc_server_set_read_only(jsonrpc, *is_backup); + ovsdb_jsonrpc_server_set_txn_forward(jsonrpc, *txn_forward); report_error_if_changed( reconfigure_remotes(jsonrpc, all_dbs, remotes), @@ -321,8 +325,10 @@ main(int argc, char *argv[]) process_init(); bool active = false; + bool txn_forward = false; parse_options(argc, argv, &db_filenames, &remotes, &unixctl_path, - &run_command, &sync_from, &sync_exclude, &active); + &run_command, &sync_from, &sync_exclude, &active, + &txn_forward); is_backup = sync_from && !active; daemon_become_new_user(false); @@ -341,18 +347,18 @@ main(int argc, char *argv[]) server_config.config_tmpfile = config_tmpfile; save_config__(config_tmpfile, &remotes, &db_filenames, sync_from, - sync_exclude, is_backup); + sync_exclude, is_backup, txn_forward); daemonize_start(false); /* Load the saved config. */ load_config(config_tmpfile, &remotes, &db_filenames, &sync_from, - &sync_exclude, &is_backup); + &sync_exclude, &is_backup, &txn_forward); /* Start ovsdb jsonrpc server. When running as a backup server, * jsonrpc connections are read only. Otherwise, both read * and write transactions are allowed. */ - jsonrpc = ovsdb_jsonrpc_server_create(is_backup); + jsonrpc = ovsdb_jsonrpc_server_create(is_backup, txn_forward); shash_init(&all_dbs); server_config.all_dbs = &all_dbs; @@ -360,6 +366,7 @@ main(int argc, char *argv[]) server_config.sync_from = &sync_from; server_config.sync_exclude = &sync_exclude; server_config.is_backup = &is_backup; + server_config.txn_forward = &txn_forward; server_config.replication_probe_interval = &replication_probe_interval; perf_counters_init(); @@ -478,7 +485,7 @@ main(int argc, char *argv[]) } main_loop(&server_config, jsonrpc, &all_dbs, unixctl, &remotes, - run_process, &exiting, &is_backup); + run_process, &exiting, &is_backup, &txn_forward); SHASH_FOR_EACH_SAFE(node, next, &all_dbs) { struct db *db = node->data; @@ -905,7 +912,7 @@ add_manager_options(struct shash *remotes, const struct ovsdb_row *row) static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); struct ovsdb_jsonrpc_options *options; long long int max_backoff, probe_interval; - bool read_only; + bool read_only, txn_forward; const char *target, *dscp_string, *role; if (!ovsdb_util_read_string_column(row, "target", &target) || !target) { @@ -925,6 +932,9 @@ add_manager_options(struct shash *remotes, const struct ovsdb_row *row) if (ovsdb_util_read_bool_column(row, "read_only", &read_only)) { options->read_only = read_only; } + if (ovsdb_util_read_bool_column(row, "txn_forward", &txn_forward)) { + options->txn_forward = txn_forward; + } free(options->role); options->role = NULL; @@ -1733,7 +1743,8 @@ ovsdb_server_get_sync_status(struct unixctl_conn *conn, int argc OVS_UNUSED, struct ds ds = DS_EMPTY_INITIALIZER; ds_put_format(&ds, "state: %s\n", is_backup ? "backup" : "active"); - + ds_put_format(&ds, "transaction forwarding: %s\n", + *config->txn_forward ? "enabled" : "disabled"); if (is_backup) { ds_put_and_free_cstr(&ds, replication_status()); } @@ -1781,7 +1792,8 @@ static void parse_options(int argc, char *argv[], struct sset *db_filenames, struct sset *remotes, char **unixctl_pathp, char **run_command, - char **sync_from, char **sync_exclude, bool *active) + char **sync_from, char **sync_exclude, + bool *active, bool *txn_forward) { enum { OPT_REMOTE = UCHAR_MAX + 1, @@ -1791,6 +1803,7 @@ parse_options(int argc, char *argv[], OPT_PEER_CA_CERT, OPT_SYNC_FROM, OPT_SYNC_EXCLUDE, + OPT_TXN_FORWARD, OPT_ACTIVE, OPT_NO_DBS, OPT_FILE_COLUMN_DIFF, @@ -1814,6 +1827,7 @@ parse_options(int argc, char *argv[], STREAM_SSL_LONG_OPTIONS, {"sync-from", required_argument, NULL, OPT_SYNC_FROM}, {"sync-exclude-tables", required_argument, NULL, OPT_SYNC_EXCLUDE}, + {"enable-txn-forward", no_argument, NULL, OPT_TXN_FORWARD}, {"active", no_argument, NULL, OPT_ACTIVE}, {"no-dbs", no_argument, NULL, OPT_NO_DBS}, {"disable-file-column-diff", no_argument, NULL, OPT_FILE_COLUMN_DIFF}, @@ -1899,6 +1913,11 @@ parse_options(int argc, char *argv[], *sync_exclude = xstrdup(optarg); break; } + + case OPT_TXN_FORWARD: + *txn_forward = true; + break; + case OPT_ACTIVE: *active = true; break; @@ -1973,7 +1992,7 @@ sset_to_json(const struct sset *sset) static void save_config__(FILE *config_file, const struct sset *remotes, const struct sset *db_filenames, const char *sync_from, - const char *sync_exclude, bool is_backup) + const char *sync_exclude, bool is_backup, bool txn_forward) { struct json *obj; char *s; @@ -1994,6 +2013,7 @@ save_config__(FILE *config_file, const struct sset *remotes, json_string_create(sync_exclude)); } json_object_put(obj, "is_backup", json_boolean_create(is_backup)); + json_object_put(obj, "txn_forward", json_boolean_create(txn_forward)); s = json_to_string(obj, 0); json_destroy(obj); @@ -2024,7 +2044,7 @@ save_config(struct server_config *config) save_config__(config->config_tmpfile, config->remotes, &db_filenames, *config->sync_from, *config->sync_exclude, - *config->is_backup); + *config->is_backup, *config->txn_forward); sset_destroy(&db_filenames); } @@ -2047,7 +2067,8 @@ sset_from_json(struct sset *sset, const struct json *array) * 'config_file', which must have been previously written by save_config(). */ static void load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames, - char **sync_from, char **sync_exclude, bool *is_backup) + char **sync_from, char **sync_exclude, bool *is_backup, + bool *txn_forward) { struct json *json; @@ -2074,6 +2095,7 @@ load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames, *sync_exclude = string ? xstrdup(json_string(string)) : NULL; *is_backup = json_boolean(shash_find_data(json_object(json), "is_backup")); - + *txn_forward = json_boolean(shash_find_data(json_object(json), + "txn_forward")); json_destroy(json); } diff --git a/ovsdb/ovsdb.h b/ovsdb/ovsdb.h index 72e127c84..db86f6da5 100644 --- a/ovsdb/ovsdb.h +++ b/ovsdb/ovsdb.h @@ -104,7 +104,7 @@ struct ovsdb_txn *ovsdb_execute_compose( struct ovsdb *, const struct ovsdb_session *, const struct json *params, bool read_only, const char *role, const char *id, long long int elapsed_msec, long long int *timeout_msec, - bool *durable, struct json **); + bool *durable, bool *all_ops_read_only, struct json **); struct json *ovsdb_execute(struct ovsdb *, const struct ovsdb_session *, const struct json *params, bool read_only, diff --git a/ovsdb/replication.c b/ovsdb/replication.c index bb1bd4250..543881843 100644 --- a/ovsdb/replication.c +++ b/ovsdb/replication.c @@ -34,6 +34,7 @@ #include "svec.h" #include "table.h" #include "transaction.h" +#include "transaction-forward.h" #include "uuid.h" VLOG_DEFINE_THIS_MODULE(replication); @@ -140,6 +141,9 @@ replication_init(const char *sync_from_, const char *exclude_tables, jsonrpc_session_close(session); } + /* Cancel all pending transactions. */ + ovsdb_txn_forward_cancel_all(false); + session = jsonrpc_session_open(sync_from, true); session_seqno = UINT_MAX; @@ -198,6 +202,9 @@ replication_run(void) if (seqno != session_seqno || state == RPL_S_INIT) { session_seqno = seqno; request_ids_clear(); + /* Canceling all the transactions that have already been forwarded, + * as they might be lost. */ + ovsdb_txn_forward_cancel_all(true); struct jsonrpc_msg *request; request = jsonrpc_create_request("get_server_id", json_array_create_empty(), NULL); @@ -208,6 +215,12 @@ replication_run(void) VLOG_DBG("send server ID request."); } + if (state == RPL_S_REPLICATING) { + /* Replication is active. Trying to forward client transactions + * to the replication source. */ + ovsdb_txn_forward_run(session); + } + msg = jsonrpc_session_recv(session); if (!msg) { continue; @@ -233,7 +246,8 @@ replication_run(void) } else if (msg->type == JSONRPC_REPLY) { struct replication_db *rdb; struct ovsdb *db; - if (!request_ids_lookup_and_free(msg->id, &db)) { + if (!request_ids_lookup_and_free(msg->id, &db) + && state != RPL_S_REPLICATING) { VLOG_WARN("received unexpected reply"); goto next; } @@ -383,12 +397,17 @@ replication_run(void) break; } + case RPL_S_REPLICATING: + /* We're not expecting any replies in this state. Assuming + * this is a reply for forwarded transaction. */ + ovsdb_txn_forward_complete(msg); + break; + case RPL_S_ERR: /* Ignore all messages */ break; case RPL_S_INIT: - case RPL_S_REPLICATING: default: OVS_NOT_REACHED(); } @@ -404,6 +423,11 @@ replication_wait(void) if (session) { jsonrpc_session_wait(session); jsonrpc_session_recv_wait(session); + + if (jsonrpc_session_is_connected(session) + && state == RPL_S_REPLICATING) { + ovsdb_txn_forward_wait(); + } } } @@ -526,6 +550,8 @@ disconnect_active_server(void) { jsonrpc_session_close(session); session = NULL; + /* Cancel all pending transactions. */ + ovsdb_txn_forward_cancel_all(false); } void @@ -542,6 +568,8 @@ replication_destroy(void) request_ids_destroy(); replication_dbs_destroy(); + ovsdb_txn_forward_cancel_all(false); + shash_destroy(&local_dbs); } @@ -998,5 +1026,7 @@ Syncing options:\n\ backup mode (except with --active)\n\ --sync-exclude-tables=DB:TABLE,...\n\ exclude the TABLE in DB from syncing\n\ + --enable-txn-forward allow trnasaction forwarding to the\n\ + replication source.\n\ --active with --sync-from, start in active mode\n"); } diff --git a/ovsdb/transaction-forward.c b/ovsdb/transaction-forward.c new file mode 100644 index 000000000..f865ff066 --- /dev/null +++ b/ovsdb/transaction-forward.c @@ -0,0 +1,187 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "transaction-forward.h" + +#include "coverage.h" +#include "jsonrpc.h" +#include "openvswitch/hmap.h" +#include "openvswitch/json.h" +#include "openvswitch/list.h" +#include "openvswitch/poll-loop.h" +#include "openvswitch/vlog.h" +#include "ovsdb.h" +#include "util.h" + +VLOG_DEFINE_THIS_MODULE(transaction_forward); + +COVERAGE_DEFINE(txn_forward_cancel); +COVERAGE_DEFINE(txn_forward_complete); +COVERAGE_DEFINE(txn_forward_create); +COVERAGE_DEFINE(txn_forward_sent); + +struct ovsdb_txn_forward { + struct ovs_list new_node; /* In 'new_transactions'. */ + struct hmap_node sent_node; /* In 'sent_transactions'. */ + struct ovsdb *db; /* Database of this transaction. */ + struct json *id; /* 'id' of the forwarded transaction. */ + struct jsonrpc_msg *request; /* Original request. */ + struct jsonrpc_msg *reply; /* Reply from the server. */ +}; + +/* List that holds transactions waiting to be forwarded to the server. */ +static struct ovs_list new_transactions = + OVS_LIST_INITIALIZER(&new_transactions); +/* Hash map for transactions that are already sent and waits for reply. */ +static struct hmap sent_transactions = HMAP_INITIALIZER(&sent_transactions); + +struct ovsdb_txn_forward * +ovsdb_txn_forward_create(struct ovsdb *db, const struct jsonrpc_msg *request) +{ + struct ovsdb_txn_forward *txn_fwd = xzalloc(sizeof *txn_fwd); + + COVERAGE_INC(txn_forward_create); + txn_fwd->db = db; + txn_fwd->request = jsonrpc_msg_clone(request); + ovs_list_push_back(&new_transactions, &txn_fwd->new_node); + + return txn_fwd; +} + +static void +ovsdb_txn_forward_unlist(struct ovsdb_txn_forward *txn_fwd) +{ + if (!ovs_list_is_empty(&txn_fwd->new_node)) { + ovs_list_remove(&txn_fwd->new_node); + ovs_list_init(&txn_fwd->new_node); + } + if (!hmap_node_is_null(&txn_fwd->sent_node)) { + hmap_remove(&sent_transactions, &txn_fwd->sent_node); + hmap_node_nullify(&txn_fwd->sent_node); + } +} + +void +ovsdb_txn_forward_destroy(struct ovsdb_txn_forward *txn_fwd) +{ + if (!txn_fwd) { + return; + } + + ovsdb_txn_forward_unlist(txn_fwd); + json_destroy(txn_fwd->id); + jsonrpc_msg_destroy(txn_fwd->request); + jsonrpc_msg_destroy(txn_fwd->reply); + free(txn_fwd); +} + +bool +ovsdb_txn_forward_is_complete(const struct ovsdb_txn_forward *txn_fwd) +{ + return txn_fwd->reply != NULL; +} + +void +ovsdb_txn_forward_complete(const struct jsonrpc_msg *reply) +{ + struct ovsdb_txn_forward *t; + size_t hash = json_hash(reply->id, 0); + + HMAP_FOR_EACH_WITH_HASH (t, sent_node, hash, &sent_transactions) { + if (json_equal(reply->id, t->id)) { + COVERAGE_INC(txn_forward_complete); + t->reply = jsonrpc_msg_clone(reply); + + /* Replacing id with the id of the original request. */ + json_destroy(t->reply->id); + t->reply->id = json_clone(t->request->id); + + hmap_remove(&sent_transactions, &t->sent_node); + hmap_node_nullify(&t->sent_node); + + t->db->run_triggers_now = t->db->run_triggers = true; + return; + } + } +} + +struct jsonrpc_msg * +ovsdb_txn_forward_steal_reply(struct ovsdb_txn_forward *txn_fwd) +{ + struct jsonrpc_msg *reply = txn_fwd->reply; + + txn_fwd->reply = NULL; + return reply; +} + +void +ovsdb_txn_forward_run(struct jsonrpc_session *session) +{ + struct ovsdb_txn_forward *t, *next; + struct jsonrpc_msg *request; + + /* Send all transactions that needs to be forwarded. */ + LIST_FOR_EACH_SAFE (t, next, new_node, &new_transactions) { + request = jsonrpc_create_request(t->request->method, + json_clone(t->request->params), + &t->id); + if (!jsonrpc_session_send(session, request)) { + COVERAGE_INC(txn_forward_sent); + ovs_list_remove(&t->new_node); + ovs_list_init(&t->new_node); + hmap_insert(&sent_transactions, &t->sent_node, + json_hash(t->id, 0)); + } + } +} + +void +ovsdb_txn_forward_wait(void) +{ + if (!ovs_list_is_empty(&new_transactions)) { + poll_immediate_wake(); + } +} + +void +ovsdb_txn_forward_cancel(struct ovsdb_txn_forward *txn_fwd) +{ + COVERAGE_INC(txn_forward_cancel); + jsonrpc_msg_destroy(txn_fwd->reply); + txn_fwd->reply = jsonrpc_create_error(json_string_create("canceled"), + txn_fwd->request->id); + ovsdb_txn_forward_unlist(txn_fwd); +} + +void +ovsdb_txn_forward_cancel_all(bool sent_only) +{ + struct ovsdb_txn_forward *t, *next; + + HMAP_FOR_EACH_SAFE (t, next, sent_node, &sent_transactions) { + ovsdb_txn_forward_cancel(t); + } + + if (sent_only) { + return; + } + + LIST_FOR_EACH_SAFE (t, next, new_node, &new_transactions) { + ovsdb_txn_forward_cancel(t); + } +} diff --git a/ovsdb/transaction-forward.h b/ovsdb/transaction-forward.h new file mode 100644 index 000000000..6bd0aed64 --- /dev/null +++ b/ovsdb/transaction-forward.h @@ -0,0 +1,42 @@ +/* + * Copyright (c) 2021, Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef OVSDB_TXN_FORWARD_H +#define OVSDB_TXN_FORWARD_H 1 + +#include + +struct ovsdb; +struct ovsdb_txn_forward; +struct jsonrpc_session; +struct jsonrpc_msg; + +struct ovsdb_txn_forward *ovsdb_txn_forward_create( + struct ovsdb *, const struct jsonrpc_msg *request); +void ovsdb_txn_forward_destroy(struct ovsdb_txn_forward *); + +bool ovsdb_txn_forward_is_complete(const struct ovsdb_txn_forward *); +void ovsdb_txn_forward_complete(const struct jsonrpc_msg *reply); + +struct jsonrpc_msg *ovsdb_txn_forward_steal_reply(struct ovsdb_txn_forward *); + +void ovsdb_txn_forward_run(struct jsonrpc_session *); +void ovsdb_txn_forward_wait(void); + +void ovsdb_txn_forward_cancel(struct ovsdb_txn_forward *); +void ovsdb_txn_forward_cancel_all(bool sent_only); + +#endif /* OVSDB_TXN_FORWARD_H */ diff --git a/ovsdb/trigger.c b/ovsdb/trigger.c index 0372302af..7067f84cc 100644 --- a/ovsdb/trigger.c +++ b/ovsdb/trigger.c @@ -28,6 +28,7 @@ #include "openvswitch/poll-loop.h" #include "server.h" #include "transaction.h" +#include "transaction-forward.h" #include "openvswitch/vlog.h" #include "util.h" @@ -43,7 +44,8 @@ bool ovsdb_trigger_init(struct ovsdb_session *session, struct ovsdb *db, struct ovsdb_trigger *trigger, struct jsonrpc_msg *request, long long int now, - bool read_only, const char *role, const char *id) + bool read_only, bool txn_forward_enabled, + const char *role, const char *id) { ovs_assert(!strcmp(request->method, "transact") || !strcmp(request->method, "convert")); @@ -53,9 +55,11 @@ ovsdb_trigger_init(struct ovsdb_session *session, struct ovsdb *db, trigger->request = request; trigger->reply = NULL; trigger->progress = NULL; + trigger->txn_forward = NULL; trigger->created = now; trigger->timeout_msec = LLONG_MAX; trigger->read_only = read_only; + trigger->txn_forward_enabled = txn_forward_enabled; trigger->role = nullable_xstrdup(role); trigger->id = nullable_xstrdup(id); return ovsdb_trigger_try(trigger, now); @@ -65,6 +69,7 @@ void ovsdb_trigger_destroy(struct ovsdb_trigger *trigger) { ovsdb_txn_progress_destroy(trigger->progress); + ovsdb_txn_forward_destroy(trigger->txn_forward); ovs_list_remove(&trigger->node); jsonrpc_msg_destroy(trigger->request); jsonrpc_msg_destroy(trigger->reply); @@ -75,7 +80,7 @@ ovsdb_trigger_destroy(struct ovsdb_trigger *trigger) bool ovsdb_trigger_is_complete(const struct ovsdb_trigger *trigger) { - return trigger->reply && !trigger->progress; + return trigger->reply && !trigger->progress && !trigger->txn_forward; } struct jsonrpc_msg * @@ -98,6 +103,11 @@ ovsdb_trigger_cancel(struct ovsdb_trigger *trigger, const char *reason) trigger->progress = NULL; } + if (trigger->txn_forward) { + ovsdb_txn_forward_destroy(trigger->txn_forward); + trigger->txn_forward = NULL; + } + jsonrpc_msg_destroy(trigger->reply); trigger->reply = NULL; @@ -148,7 +158,7 @@ ovsdb_trigger_run(struct ovsdb *db, long long int now) LIST_FOR_EACH_SAFE (t, next, node, &db->triggers) { if (run_triggers || now - t->created >= t->timeout_msec - || t->progress) { + || t->progress || t->txn_forward) { if (ovsdb_trigger_try(t, now)) { disconnect_all = true; } @@ -188,23 +198,32 @@ static bool ovsdb_trigger_try(struct ovsdb_trigger *t, long long int now) { /* Handle "initialized" state. */ - if (!t->reply) { + if (!t->reply && !t->txn_forward) { ovs_assert(!t->progress); struct ovsdb_txn *txn = NULL; struct ovsdb *newdb = NULL; + bool txn_forwarding_needed = (t->read_only && t->txn_forward_enabled); if (!strcmp(t->request->method, "transact")) { if (!ovsdb_txn_precheck_prereq(t->db)) { return false; } - bool durable; + bool durable, all_ops_read_only; struct json *result; + /* Trying to compose transaction. If transaction forwarding is + * needed, composing it as if there is no read-only restriction. + * This allows us to avoid forwarding of invalid requests. It's + * also a security check, because we will forward transactions from + * this server and we have to be sure that client passes the RBAC + * check. Transaction will be executed locally if all operations + * are read-only. */ txn = ovsdb_execute_compose( - t->db, t->session, t->request->params, t->read_only, + t->db, t->session, t->request->params, + txn_forwarding_needed ? false : t->read_only, t->role, t->id, now - t->created, &t->timeout_msec, - &durable, &result); + &durable, &all_ops_read_only, &result); if (!txn) { if (result) { /* Complete. There was an error but we still represent it @@ -217,9 +236,20 @@ ovsdb_trigger_try(struct ovsdb_trigger *t, long long int now) return false; } - /* Transition to "committing" state. */ - t->reply = jsonrpc_create_reply(result, t->request->id); - t->progress = ovsdb_txn_propose_commit(txn, durable); + if (txn_forwarding_needed && !all_ops_read_only) { + /* Transaction is good, but we don't need it. */ + ovsdb_txn_abort(txn); + json_destroy(result); + /* Transition to "forwarding" state. */ + t->txn_forward = ovsdb_txn_forward_create(t->db, t->request); + /* Forward will not be completed immediately. Will check + * next time. */ + return false; + } else { + /* Transition to "committing" state. */ + t->reply = jsonrpc_create_reply(result, t->request->id); + t->progress = ovsdb_txn_propose_commit(txn, durable); + } } else if (!strcmp(t->request->method, "convert")) { /* Permission check. */ if (t->role && *t->role) { @@ -348,6 +378,18 @@ ovsdb_trigger_try(struct ovsdb_trigger *t, long long int now) ovsdb_trigger_complete(t); } + return false; + } else if (t->txn_forward) { + /* Handle "forwarding" state. */ + if (!ovsdb_txn_forward_is_complete(t->txn_forward)) { + return false; + } + + /* Transition to "complete". */ + t->reply = ovsdb_txn_forward_steal_reply(t->txn_forward); + ovsdb_txn_forward_destroy(t->txn_forward); + t->txn_forward = NULL; + ovsdb_trigger_complete(t); return false; } diff --git a/ovsdb/trigger.h b/ovsdb/trigger.h index 79af7f6be..4da0ff2e1 100644 --- a/ovsdb/trigger.h +++ b/ovsdb/trigger.h @@ -22,26 +22,34 @@ struct ovsdb; /* Triggers have the following states: * - * - Initialized (reply == NULL, progress == NULL): Executing the trigger - * can keep it in the initialized state, if it has a "wait" condition that - * isn't met. Executing the trigger can also yield an error, in which - * case it transitions to "complete". Otherwise, execution yields a - * transaction, which the database attempts to commit. If the transaction - * completes immediately and synchronously, then the trigger transitions - * to the "complete" state. If the transaction requires some time to - * complete, it transitions to the "committing" state. + * - Initialized (reply == NULL, progress == NULL, txn_forward == NULL): + * Executing the trigger can keep it in the initialized state, if it has a + * "wait" condition that isn't met. Executing the trigger can also yield + * an error, in which case it transitions to "complete". Otherwise, + * execution yields a transaction, which the database attempts to commit. + * If the transaction completes immediately and synchronously, then the + * trigger transitions to the "complete" state. If the transaction + * requires some time to complete, it transitions to the "committing" + * state. If the transaction can not be completed locally due to + * read-only restrictions and transaction forwarding is enabled, starts + * forwarding and transitions to the "forwarding" state. * - * - Committing (reply != NULL, progress != NULL): The transaction is - * committing. If it succeeds, or if it fails permanently, then the - * trigger transitions to "complete". If it fails temporarily - * (e.g. because someone else committed to cluster-based storage before we - * did), then we transition back to "initialized" to try again. + * - Committing (reply != NULL, progress != NULL, txn_forward == NULL): + * The transaction is committing. If it succeeds, or if it fails + * permanently, then the trigger transitions to "complete". If it fails + * temporarily (e.g. because someone else committed to cluster-based + * storage before we did), then we transition back to "initialized" to + * try again. * - * - Complete (reply != NULL, progress == NULL): The transaction is done - * and either succeeded or failed. + * - Forwarding (reply == NULL, progress == NULL, txn_forward != NULL): + * Transaction is forwarded. Either it succeeds or it fails, the trigger + * transitions to "complete". + * + * - Complete (reply != NULL, progress == NULL, txn_forward == NULL): + * The transaction is done and either succeeded or failed. */ struct ovsdb_trigger { - /* In "initialized" or "committing" state, in db->triggers. + /* In "initialized", "committing" or "forwarding" state, in db->triggers. * In "complete", in session->completions. */ struct ovs_list node; struct ovsdb_session *session; /* Session that owns this trigger. */ @@ -49,9 +57,11 @@ struct ovsdb_trigger { struct jsonrpc_msg *request; /* Database request. */ struct jsonrpc_msg *reply; /* Result (null if none yet). */ struct ovsdb_txn_progress *progress; + struct ovsdb_txn_forward *txn_forward; /* Tracks transaction forwarding. */ long long int created; /* Time created. */ long long int timeout_msec; /* Max wait duration. */ bool read_only; /* Database is in read only mode. */ + bool txn_forward_enabled; /* Server configured to forward transactions.*/ char *role; /* Role, for role-based access controls. */ char *id; /* ID, for role-based access controls. */ }; @@ -59,7 +69,8 @@ struct ovsdb_trigger { bool ovsdb_trigger_init(struct ovsdb_session *, struct ovsdb *, struct ovsdb_trigger *, struct jsonrpc_msg *request, long long int now, - bool read_only, const char *role, const char *id); + bool read_only, bool txn_forward_enabled, + const char *role, const char *id); void ovsdb_trigger_destroy(struct ovsdb_trigger *); bool ovsdb_trigger_is_complete(const struct ovsdb_trigger *); diff --git a/tests/ovsdb-server.at b/tests/ovsdb-server.at index 926abce3a..c718f7e2a 100644 --- a/tests/ovsdb-server.at +++ b/tests/ovsdb-server.at @@ -1452,6 +1452,62 @@ m4_define([OVSDB_CHECK_EXECUTION], EXECUTION_EXAMPLES +AT_BANNER([OVSDB -- ovsdb-server replication with transaction forwarding]) + +# OVSDB_CHECK_EXECUTION(TITLE, SCHEMA, TRANSACTIONS, OUTPUT, [KEYWORDS]) +# +# Creates two databases with the given SCHEMA, and starts an ovsdb-server on +# each database. +# Runs each of the TRANSACTIONS (which should be a quoted list of +# quoted strings) against the replication server with ovsdb-client one at a +# time. The server executes read-only transactions and forwards rest of them +# to the other ovsdb-server. +# +# Checks that the overall output is OUTPUT, but UUIDs in the output +# are replaced by markers of the form where N is a number. The +# first unique UUID is replaced by <0>, the next by <1>, and so on. +# If a given UUID appears more than once it is always replaced by the +# same marker. +# +# Checks that the dump of both databases are the same. +# +# TITLE is provided to AT_SETUP and KEYWORDS to AT_KEYWORDS. +m4_define([OVSDB_CHECK_EXECUTION], + [AT_SETUP([$1]) + AT_KEYWORDS([ovsdb server tcp replication transaction forward $5]) + $2 > schema + AT_CHECK([ovsdb-tool create db1 schema], [0], [stdout], [ignore]) + AT_CHECK([ovsdb-tool create db2 schema], [0], [stdout], [ignore]) + + on_exit 'kill `cat *.pid`' + AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server1.log dnl + --pidfile --remote=punix:db.sock db1 + ], [0], [ignore], [ignore]) + + AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server2.log dnl + --pidfile=pid2 --remote=punix:db2.sock dnl + --unixctl=unixctl2 -vjsonrpc:file:dbg dnl + --sync-from=unix:db.sock --enable-txn-forward db2 + ], [0], [ignore], [ignore]) + on_exit 'test ! -e pid2 || kill `cat pid2`' + + m4_foreach([txn], [$3], + [AT_CHECK([ovsdb-client transact unix:db2.sock 'txn'], [0], + [stdout], [ignore]) + cat stdout >> output + ]) + + AT_CHECK([uuidfilt output], [0], [$4], [ignore]) + + AT_CHECK([ovsdb-client dump], [0], [stdout], [ignore]) + OVS_WAIT_UNTIL([ ovsdb-client dump unix:db2.sock > dump2; diff stdout dump2]) + + OVSDB_SERVER_SHUTDOWN + OVSDB_SERVER_SHUTDOWN2 + AT_CLEANUP]) + +EXECUTION_EXAMPLES + AT_BANNER([OVSDB -- ovsdb-server replication table-exclusion]) # OVSDB_CHECK_REPLICATION(TITLE, SCHEMA, TRANSACTIONS, OUTPUT, [KEYWORDS]) @@ -1762,7 +1818,9 @@ OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/sync-status |grep re dnl Switch the 'db1' to active AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/disconnect-active-ovsdb-server]) -AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status], [0], [state: active +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status], [0], [dnl +state: active +transaction forwarding: disabled ]) dnl Issue a transaction to 'db1' @@ -1781,7 +1839,9 @@ AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/connect-active-ovsdb-server dnl Verify the change happend OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status |grep replicating]) -AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/sync-status], [0], [state: active +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/sync-status], [0], [dnl +state: active +transaction forwarding: disabled ]) dnl Issue an transaction to 'db2' which is now active. diff --git a/tests/test-ovsdb.c b/tests/test-ovsdb.c index a886f971e..7dc634d3e 100644 --- a/tests/test-ovsdb.c +++ b/tests/test-ovsdb.c @@ -1573,7 +1573,7 @@ do_trigger(struct ovs_cmdl_context *ctx) ovsdb_trigger_init(&session, db, &t->trigger, jsonrpc_create_request("transact", params, NULL), - now, false, NULL, NULL); + now, false, false, NULL, NULL); t->number = number++; if (ovsdb_trigger_is_complete(&t->trigger)) { do_trigger_dump(t, now, "immediate"); From patchwork Sat May 1 00:55:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1472587 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FX9mC0bXWz9sSC for ; Sat, 1 May 2021 10:56:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 31A9343289; Sat, 1 May 2021 00:56:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y81I1D0YqkDC; Sat, 1 May 2021 00:56:18 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id 96D7943290; Sat, 1 May 2021 00:56:16 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D5799C0026; Sat, 1 May 2021 00:56:14 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 95F8BC0001 for ; Sat, 1 May 2021 00:56:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 85C2143281 for ; Sat, 1 May 2021 00:56:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bGWhtDGGSisB for ; Sat, 1 May 2021 00:56:12 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by smtp2.osuosl.org (Postfix) with ESMTPS id 83C8943283 for ; Sat, 1 May 2021 00:56:11 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 2DB26240008; Sat, 1 May 2021 00:56:09 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 1 May 2021 02:55:43 +0200 Message-Id: <20210501005548.3071269-3-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210501005548.3071269-1-i.maximets@ovn.org> References: <20210501005548.3071269-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH 2/7] ovsdb: Add extra internal tables to databases for replication purposes. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" New flag for ovsdb table schema 'copyForReplication'. It's needed to enable replication of a _Server database in later commits. With this option ovsdb-server will create a new _synced_ table where it will store data received from the replication source while keeping the original table for data of the local server. This way ovsdb-server will be able to keep state of local databases while replicating state of databases from the active server. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- Documentation/ref/ovsdb-server.7.rst | 7 +++++++ ovsdb/ovsdb-doc | 3 ++- ovsdb/ovsdb.c | 19 ++++++++++++++++++- ovsdb/table.c | 20 ++++++++++++++++---- ovsdb/table.h | 4 +++- python/ovs/db/schema.py | 20 +++++++++++++++++--- 6 files changed, 63 insertions(+), 10 deletions(-) diff --git a/Documentation/ref/ovsdb-server.7.rst b/Documentation/ref/ovsdb-server.7.rst index 04414350a..717f62d81 100644 --- a/Documentation/ref/ovsdb-server.7.rst +++ b/Documentation/ref/ovsdb-server.7.rst @@ -104,6 +104,13 @@ configuration and the following columns: The set of columns and column:key pairs for which authorized update and mutate operations should be permitted. +Since version 2.16, database table could be copied for replication purposes +by setting ``copyForReplication`` flag to ``true``. For each table marked +with this flag, ``ovsdb-server`` will create one more table with the same +name and ``_synced_`` prefix (e.g., ``_synced_``). Server in a +backup role will keep its own content in the original table and will put +data, received from the active server, to this special table. + 4 Wire Protocol --------------- diff --git a/ovsdb/ovsdb-doc b/ovsdb/ovsdb-doc index 10d0c0c13..5513783c1 100755 --- a/ovsdb/ovsdb-doc +++ b/ovsdb/ovsdb-doc @@ -213,7 +213,8 @@ def docsToNroff(schemaFile, xmlFile, erFile, version=None): introNodes += [dbNode] documented_tables = set((name for (name, title) in summary)) - schema_tables = set(schema.tables.keys()) + schema_tables = set([name for name in schema.tables.keys() + if not name.startswith("_")]) undocumented_tables = schema_tables - documented_tables for table in undocumented_tables: raise error.Error("undocumented table %s" % table) diff --git a/ovsdb/ovsdb.c b/ovsdb/ovsdb.c index 9042658fa..f662a0238 100644 --- a/ovsdb/ovsdb.c +++ b/ovsdb/ovsdb.c @@ -238,6 +238,21 @@ ovsdb_schema_from_json(const struct json *json, struct ovsdb_schema **schemap) } shash_add(&schema->tables, table->name, table); + + if (table->copy_for_replication) { + /* Need to create a copy of the table for the case it will be + * synced from another server. */ + struct ovsdb_table_schema *synced_table; + + synced_table = ovsdb_table_schema_clone(table); + free(synced_table->name); + synced_table->name = xasprintf("_synced_%s", node->name); + /* Clearing 'copy' flag to avoid accidental further copying. */ + synced_table->copy_for_replication = false; + + shash_add(&schema->tables, synced_table->name, synced_table); + } + } /* "isRoot" was not part of the original schema definition. Before it was @@ -308,8 +323,10 @@ ovsdb_schema_to_json(const struct ovsdb_schema *schema) SHASH_FOR_EACH (node, &schema->tables) { struct ovsdb_table_schema *table = node->data; - json_object_put(tables, table->name, + if (node->name[0] != '_') { + json_object_put(tables, table->name, ovsdb_table_schema_to_json(table, default_is_root)); + } } json_object_put(json, "tables", tables); diff --git a/ovsdb/table.c b/ovsdb/table.c index 6cd2d886d..b46946072 100644 --- a/ovsdb/table.c +++ b/ovsdb/table.c @@ -36,7 +36,8 @@ add_column(struct ovsdb_table_schema *ts, struct ovsdb_column *column) struct ovsdb_table_schema * ovsdb_table_schema_create(const char *name, bool mutable, - unsigned int max_rows, bool is_root) + unsigned int max_rows, bool is_root, + bool copy_for_replication) { struct ovsdb_column *uuid, *version; struct ovsdb_table_schema *ts; @@ -47,6 +48,7 @@ ovsdb_table_schema_create(const char *name, bool mutable, shash_init(&ts->columns); ts->max_rows = max_rows; ts->is_root = is_root; + ts->copy_for_replication = copy_for_replication; uuid = ovsdb_column_create("_uuid", false, true, &ovsdb_type_uuid); add_column(ts, uuid); @@ -70,7 +72,8 @@ ovsdb_table_schema_clone(const struct ovsdb_table_schema *old) size_t i; new = ovsdb_table_schema_create(old->name, old->mutable, - old->max_rows, old->is_root); + old->max_rows, old->is_root, + old->copy_for_replication); SHASH_FOR_EACH (node, &old->columns) { const struct ovsdb_column *column = node->data; @@ -126,7 +129,8 @@ ovsdb_table_schema_from_json(const struct json *json, const char *name, struct ovsdb_table_schema **tsp) { struct ovsdb_table_schema *ts; - const struct json *columns, *mutable, *max_rows, *is_root, *indexes; + const struct json *columns, *mutable, *max_rows; + const struct json *is_root, *indexes, *copy_for_replication; struct shash_node *node; struct ovsdb_parser parser; struct ovsdb_error *error; @@ -141,6 +145,8 @@ ovsdb_table_schema_from_json(const struct json *json, const char *name, max_rows = ovsdb_parser_member(&parser, "maxRows", OP_INTEGER | OP_OPTIONAL); is_root = ovsdb_parser_member(&parser, "isRoot", OP_BOOLEAN | OP_OPTIONAL); + copy_for_replication = ovsdb_parser_member(&parser, "copyForReplication", + OP_BOOLEAN | OP_OPTIONAL); indexes = ovsdb_parser_member(&parser, "indexes", OP_ARRAY | OP_OPTIONAL); error = ovsdb_parser_finish(&parser); if (error) { @@ -165,7 +171,10 @@ ovsdb_table_schema_from_json(const struct json *json, const char *name, ts = ovsdb_table_schema_create(name, mutable ? json_boolean(mutable) : true, MIN(n_max_rows, UINT_MAX), - is_root ? json_boolean(is_root) : false); + is_root ? json_boolean(is_root) : false, + copy_for_replication + ? json_boolean(copy_for_replication) + : false); SHASH_FOR_EACH (node, json_object(columns)) { struct ovsdb_column *column; @@ -249,6 +258,9 @@ ovsdb_table_schema_to_json(const struct ovsdb_table_schema *ts, if (default_is_root != ts->is_root) { json_object_put(json, "isRoot", json_boolean_create(ts->is_root)); } + if (ts->copy_for_replication) { + json_object_put(json, "copyForReplication", json_boolean_create(true)); + } columns = json_object_create(); diff --git a/ovsdb/table.h b/ovsdb/table.h index 69dd649df..afd56f7a6 100644 --- a/ovsdb/table.h +++ b/ovsdb/table.h @@ -29,6 +29,7 @@ struct ovsdb_table_schema { char *name; bool mutable; bool is_root; /* Part of garbage collection root set? */ + bool copy_for_replication; /* '_synced_' copy of the table needed? */ unsigned int max_rows; /* Maximum number of rows. */ struct shash columns; /* Contains "struct ovsdb_column *"s. */ struct ovsdb_column_set *indexes; @@ -36,7 +37,8 @@ struct ovsdb_table_schema { }; struct ovsdb_table_schema *ovsdb_table_schema_create( - const char *name, bool mutable, unsigned int max_rows, bool is_root); + const char *name, bool mutable, unsigned int max_rows, + bool is_root, bool copy_for_replication); struct ovsdb_table_schema *ovsdb_table_schema_clone( const struct ovsdb_table_schema *); void ovsdb_table_schema_destroy(struct ovsdb_table_schema *); diff --git a/python/ovs/db/schema.py b/python/ovs/db/schema.py index 3ba844ae5..e58e81080 100644 --- a/python/ovs/db/schema.py +++ b/python/ovs/db/schema.py @@ -80,6 +80,13 @@ class DbSchema(object): _check_id(tableName, json) tables[tableName] = TableSchema.from_json(tableJson, tableName, allow_extensions) + if tables[tableName].copy_for_replication: + synced_table_name = "_synced_" + tableName + synced_table = TableSchema.from_json(tableJson, + synced_table_name, + allow_extensions) + synced_table.copy_for_replication = False + tables[synced_table_name] = synced_table return DbSchema(name, version, tables) @@ -92,7 +99,8 @@ class DbSchema(object): tables = {} for table in self.tables.values(): - tables[table.name] = table.to_json(default_is_root) + if not table.name.startswith("_"): + tables[table.name] = table.to_json(default_is_root) json = {"name": self.name, "tables": tables} if self.version: json["version"] = self.version @@ -172,7 +180,8 @@ def column_set_from_json(json, columns): class TableSchema(object): def __init__(self, name, columns, mutable=True, max_rows=sys.maxsize, - is_root=True, indexes=[], extensions={}): + is_root=True, indexes=[], extensions={}, + copy_for_replication=False): self.name = name self.columns = columns self.mutable = mutable @@ -180,6 +189,7 @@ class TableSchema(object): self.is_root = is_root self.indexes = indexes self.extensions = extensions + self.copy_for_replication = copy_for_replication @staticmethod def from_json(json, name, allow_extensions=False): @@ -188,6 +198,8 @@ class TableSchema(object): mutable = parser.get_optional("mutable", [bool], True) max_rows = parser.get_optional("maxRows", [int]) is_root = parser.get_optional("isRoot", [bool], False) + copy_for_replication = parser.get_optional("copyForReplication", + [bool], False) indexes_json = parser.get_optional("indexes", [list], []) if allow_extensions: extensions = parser.get_optional("extensions", [dict], {}) @@ -224,7 +236,7 @@ class TableSchema(object): indexes.append(index) return TableSchema(name, columns, mutable, max_rows, is_root, indexes, - extensions) + extensions, copy_for_replication) def to_json(self, default_is_root=False): """Returns this table schema serialized into JSON. @@ -243,6 +255,8 @@ class TableSchema(object): json["mutable"] = False if default_is_root != self.is_root: json["isRoot"] = self.is_root + if self.copy_for_replication: + json["copyForReplication"] = True json["columns"] = columns = {} for column in self.columns.values(): From patchwork Sat May 1 00:55:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1472589 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FX9mK5Pgwz9sWl for ; Sat, 1 May 2021 10:56:33 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id D8F476FAB9; Sat, 1 May 2021 00:56:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N9Lhf8s2YsjC; Sat, 1 May 2021 00:56:25 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id 4CBCA6FA8B; Sat, 1 May 2021 00:56:18 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 07A46C0025; Sat, 1 May 2021 00:56:18 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3A53EC002E for ; Sat, 1 May 2021 00:56:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 80A536FA75 for ; Sat, 1 May 2021 00:56:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HOztBDcoMWm6 for ; Sat, 1 May 2021 00:56:14 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by smtp3.osuosl.org (Postfix) with ESMTPS id 10ED960695 for ; Sat, 1 May 2021 00:56:13 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id B0047240003; Sat, 1 May 2021 00:56:11 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 1 May 2021 02:55:44 +0200 Message-Id: <20210501005548.3071269-4-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210501005548.3071269-1-i.maximets@ovn.org> References: <20210501005548.3071269-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH 3/7] replication: Allow replication of _Server database. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" If replication server is connected to a cluster member, clients would like to know the state of the cluster from that server. This will allow them to make a decision about re-connection. Marking 'Database' table with a 'copyForReplication' flag to have a new '_synced_Database' table where records from the active server will be stored. This way client will see state of local databases in 'Database' table and state of databases of the active server in '_synced_Database' table. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- Documentation/ref/ovsdb-server.7.rst | 3 +- NEWS | 4 ++ ovsdb/_server.ovsschema | 5 ++- ovsdb/_server.xml | 10 +++++ ovsdb/ovsdb-server.c | 2 +- ovsdb/replication.c | 66 +++++++++++++++++++++++++--- tests/ovsdb-server.at | 5 +++ 7 files changed, 86 insertions(+), 9 deletions(-) diff --git a/Documentation/ref/ovsdb-server.7.rst b/Documentation/ref/ovsdb-server.7.rst index 717f62d81..28686eb44 100644 --- a/Documentation/ref/ovsdb-server.7.rst +++ b/Documentation/ref/ovsdb-server.7.rst @@ -109,7 +109,8 @@ by setting ``copyForReplication`` flag to ``true``. For each table marked with this flag, ``ovsdb-server`` will create one more table with the same name and ``_synced_`` prefix (e.g., ``_synced_``). Server in a backup role will keep its own content in the original table and will put -data, received from the active server, to this special table. +data, received from the active server, to this special table. Currently +used to replicate ``Database`` table from a ``_Server`` database. 4 Wire Protocol --------------- diff --git a/NEWS b/NEWS index 790f93af9..d2b12b23c 100644 --- a/NEWS +++ b/NEWS @@ -4,6 +4,10 @@ Post-v2.15.0 * New command line argument '--enable-txn-forward' for ovsdb-server in replication mode that allows to forward transactions that includes database modifications to the primary database server. + * Replication (backup) server now replicates a _Server database from the + replication source (active server). Data from the 'Database' table of + the active server stored in '_synced_Database' table on backup and + available for monitoring. - In ovs-vsctl and vtep-ctl, the "find" command now accept new operators {in} and {not-in}. - Userspace datapath: diff --git a/ovsdb/_server.ovsschema b/ovsdb/_server.ovsschema index a867e5cbf..ad31a7f7e 100644 --- a/ovsdb/_server.ovsschema +++ b/ovsdb/_server.ovsschema @@ -1,6 +1,6 @@ {"name": "_Server", - "version": "1.1.0", - "cksum": "3236486585 698", + "version": "1.2.0", + "cksum": "88700080 731", "tables": { "Database": { "columns": { @@ -18,4 +18,5 @@ "type": {"key": {"type": "uuid"}, "min": 0, "max": 1}}, "index": { "type": {"key": {"type": "integer"}, "min": 0, "max": 1}}}, + "copyForReplication": true, "isRoot": true}}} diff --git a/ovsdb/_server.xml b/ovsdb/_server.xml index 70cd22db7..2e8b3393d 100644 --- a/ovsdb/_server.xml +++ b/ovsdb/_server.xml @@ -54,6 +54,16 @@ to the server causes its row _uuid to change.

+

+ This table is marked with copyForReplication flag. This + means that ovsdb-server will create one more table with + name _synced_Database for replication purposes. Backup + database in active-backup model will replicate content of this table + from the active database and store it in _synced_Database + table. This way clients are able to monitor the state of the active + server, e.g., check the status of clustered databases. +

+ The database's name, as specified in its schema. diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index c9d21af1c..89780dac8 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -160,7 +160,7 @@ ovsdb_replication_init(const char *sync_from, const char *exclude, struct shash_node *node; SHASH_FOR_EACH (node, all_dbs) { struct db *db = node->data; - if (node->name[0] != '_' && db->db) { + if (db->db) { replication_add_local_db(node->name, db->db); } } diff --git a/ovsdb/replication.c b/ovsdb/replication.c index 543881843..5aac325f2 100644 --- a/ovsdb/replication.c +++ b/ovsdb/replication.c @@ -49,6 +49,7 @@ static void add_monitored_table(struct ovsdb_table_schema *table, struct json *monitor_requests); static struct ovsdb_error *reset_database(struct ovsdb *db); +static void reset_internal_databases(void); static struct ovsdb_error *process_notification(struct json *, struct ovsdb *); static struct ovsdb_error *process_table_update(struct json *table_update, @@ -104,6 +105,7 @@ static enum ovsdb_replication_state state; struct replication_db { struct ovsdb *db; bool schema_version_higher; + bool internal; /* True if the database name starts with '_'. */ /* Points to the schema received from the active server if * the local db schema version is higher. NULL otherwise. */ struct ovsdb_schema *active_db_schema; @@ -118,6 +120,10 @@ static bool is_replication_possible(struct ovsdb_schema *local_db_schema, static struct shash local_dbs = SHASH_INITIALIZER(&local_dbs); static struct shash *replication_dbs; +/* Number of internal databases in 'replication_dbs', i.e. databases which + * name starts with '_'. */ +static int n_internal_dbs = 0; + static struct shash *replication_dbs_create(void); static void replication_dbs_destroy(void); /* Find 'struct ovsdb' by name within 'replication_dbs' */ @@ -134,6 +140,7 @@ replication_init(const char *sync_from_, const char *exclude_tables, * parseable. An error here is unexpected. */ ovs_assert(!set_excluded_tables(exclude_tables, false)); + reset_internal_databases(); replication_dbs_destroy(); shash_clear(&local_dbs); @@ -337,6 +344,9 @@ replication_run(void) if (r->active_db_schema) { ovsdb_schema_destroy(r->active_db_schema); } + if (r->internal) { + n_internal_dbs--; + } free(r); ovsdb_schema_destroy(schema); } @@ -349,8 +359,10 @@ replication_run(void) if (hmap_is_empty(&request_ids)) { struct shash_node *node; - if (shash_is_empty(replication_dbs)) { + if (shash_is_empty(replication_dbs) || + n_internal_dbs == shash_count(replication_dbs)) { VLOG_WARN("Nothing to replicate."); + replication_dbs_destroy(); state = RPL_S_ERR; } else { SHASH_FOR_EACH (node, replication_dbs) { @@ -552,6 +564,8 @@ disconnect_active_server(void) session = NULL; /* Cancel all pending transactions. */ ovsdb_txn_forward_cancel_all(false); + /* Clear data from '_synced_' tables. */ + reset_internal_databases(); } void @@ -586,9 +600,12 @@ reset_database(struct ovsdb *db) struct shash_node *table_node; SHASH_FOR_EACH (table_node, &db->tables) { - /* Delete all rows if the table is not excluded. */ - if (!excluded_tables_find(db->schema->name, table_node->name)) { - struct ovsdb_table *table = table_node->data; + struct ovsdb_table *table = table_node->data; + + /* Delete all rows if the table is not excluded and if it has no + * special '_synced_' copy. */ + if (!excluded_tables_find(db->schema->name, table_node->name) + && !table->schema->copy_for_replication) { struct ovsdb_row *row, *next; HMAP_FOR_EACH_SAFE (row, next, hmap_node, &table->rows) { ovsdb_txn_row_delete(txn, row); @@ -599,6 +616,26 @@ reset_database(struct ovsdb *db) return ovsdb_txn_propose_commit_block(txn, false); } +static void +reset_internal_databases(void) +{ + if (!replication_dbs) { + return; + } + + struct shash_node *node; + + SHASH_FOR_EACH (node, replication_dbs) { + struct replication_db *rdb = node->data; + + if (rdb->internal) { + /* Cleaning up data from '_synced_' tables of internal DBs. */ + VLOG_INFO("Resetting internal database %s", node->name); + ovsdb_error_assert(reset_database(rdb->db)); + } + } +} + /* Create a monitor request for 'db'. The monitor request will include * any tables from 'excluded_tables' * @@ -617,9 +654,11 @@ create_monitor_request(struct ovsdb_schema *schema) for (int j = 0; j < n; j++) { struct ovsdb_table_schema *table = nodes[j]->data; + /* Not monitoring data from replicated copies of tables. */ + bool skip = !strncmp(table->name, "_synced_", 8); /* Monitor all tables not excluded. */ - if (!excluded_tables_find(db_name, table->name)) { + if (!excluded_tables_find(db_name, table->name) && !skip) { add_monitored_table(table, monitor_request); } } @@ -690,6 +729,18 @@ process_table_update(struct json *table_update, const char *table_name, return ovsdb_error("unknown table", "unknown table %s", table_name); } + if (table->schema->copy_for_replication) { + /* Data from this table should go to special '_synced_' table. */ + char *name = xasprintf("_synced_%s", table_name); + + table = ovsdb_get_table(database, name); + free(name); + if (!table) { + return ovsdb_error("unknown table", + "unknown table _synced_%s", table_name); + } + } + if (table_update->type != JSON_OBJECT) { return ovsdb_error("Not a JSON object", " for table is not object"); @@ -853,7 +904,11 @@ replication_dbs_create(void) repl_db->db = node->data; repl_db->schema_version_higher = false; repl_db->active_db_schema = NULL; + repl_db->internal = (node->name[0] == '_'); shash_add(new, node->name, repl_db); + if (repl_db->internal) { + n_internal_dbs++; + } } return new; @@ -882,6 +937,7 @@ replication_dbs_destroy(void) hmap_destroy(&replication_dbs->map); free(replication_dbs); replication_dbs = NULL; + n_internal_dbs = 0; } /* Return true if replication just started or is ongoing. diff --git a/tests/ovsdb-server.at b/tests/ovsdb-server.at index c718f7e2a..378e07372 100644 --- a/tests/ovsdb-server.at +++ b/tests/ovsdb-server.at @@ -1502,6 +1502,11 @@ m4_define([OVSDB_CHECK_EXECUTION], AT_CHECK([ovsdb-client dump], [0], [stdout], [ignore]) OVS_WAIT_UNTIL([ ovsdb-client dump unix:db2.sock > dump2; diff stdout dump2]) + AT_CHECK([ovsdb-client dump unix:db2.sock _Server _synced_Database | dnl + sed 's/_synced_Database/Database/' > synced_Server ]) + AT_CHECK([ovsdb-client dump unix:db.sock _Server Database > Server]) + AT_CHECK([diff synced_Server Server]) + OVSDB_SERVER_SHUTDOWN OVSDB_SERVER_SHUTDOWN2 AT_CLEANUP]) From patchwork Sat May 1 00:55:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1472591 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FX9mR2Krcz9sWl for ; Sat, 1 May 2021 10:56:39 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id C71786FB0C; Sat, 1 May 2021 00:56:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DkVVc9oGRdq3; Sat, 1 May 2021 00:56:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id EC4786FA9F; Sat, 1 May 2021 00:56:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CF86CC0023; Sat, 1 May 2021 00:56:21 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 131F6C000E for ; Sat, 1 May 2021 00:56:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id E9C0D850E0 for ; Sat, 1 May 2021 00:56:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rB4DohYpRD7Z for ; Sat, 1 May 2021 00:56:16 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by smtp1.osuosl.org (Postfix) with ESMTPS id 73520850D9 for ; Sat, 1 May 2021 00:56:16 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 13918240005; Sat, 1 May 2021 00:56:13 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 1 May 2021 02:55:45 +0200 Message-Id: <20210501005548.3071269-5-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210501005548.3071269-1-i.maximets@ovn.org> References: <20210501005548.3071269-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH 4/7] ovsdb-cs: Monitor _synced_Database table. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Monitoring data from '_synced_Database' table to check the state of clustered databases from which this replica receives updates. Not allowing 'leader-only' connection to the replication server as this type of connections typically used for database locking. Otherwise making decision about re-connection as usual, but considering rows from both tables. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- lib/ovsdb-cs.c | 110 ++++++++++++++++++++++++++++++----------- tests/ovsdb-cluster.at | 37 +++++++++++++- 2 files changed, 116 insertions(+), 31 deletions(-) diff --git a/lib/ovsdb-cs.c b/lib/ovsdb-cs.c index 6f9f912ac..7e861d1f1 100644 --- a/lib/ovsdb-cs.c +++ b/lib/ovsdb-cs.c @@ -1639,6 +1639,7 @@ static const struct server_column server_columns[] = { struct server_row { struct hmap_node hmap_node; struct uuid uuid; + bool synced_table; struct ovsdb_datum data[N_SERVER_COLUMNS]; }; @@ -1673,11 +1674,13 @@ ovsdb_cs_delete_server_row(struct ovsdb_cs *cs, struct server_row *row) } static struct server_row * -ovsdb_cs_insert_server_row(struct ovsdb_cs *cs, const struct uuid *uuid) +ovsdb_cs_insert_server_row(struct ovsdb_cs *cs, const struct uuid *uuid, + bool synced_table) { struct server_row *row = xmalloc(sizeof *row); hmap_insert(&cs->server_rows, &row->hmap_node, uuid_hash(uuid)); row->uuid = *uuid; + row->synced_table = synced_table; for (size_t i = 0; i < N_SERVER_COLUMNS; i++) { ovsdb_datum_init_default(&row->data[i], &server_columns[i].type); } @@ -1758,9 +1761,13 @@ ovsdb_cs_process_server_event(struct ovsdb_cs *cs, ovsdb_cs_clear_server_rows(cs); } - const struct ovsdb_cs_table_update *tu = ovsdb_cs_db_update_find_table( - du, "Database"); - if (tu) { + const char *tables[] = {"Database", "_synced_Database"}; + for (size_t idx = 0; idx < ARRAY_SIZE(tables); idx++) { + const struct ovsdb_cs_table_update *tu = ovsdb_cs_db_update_find_table( + du, tables[idx]); + if (!tu) { + continue; + } for (size_t i = 0; i < tu->n; i++) { const struct ovsdb_cs_row_update *ru = &tu->row_updates[i]; struct server_row *row @@ -1769,14 +1776,13 @@ ovsdb_cs_process_server_event(struct ovsdb_cs *cs, ovsdb_cs_delete_server_row(cs, row); } else { if (!row) { - row = ovsdb_cs_insert_server_row(cs, &ru->row_uuid); + row = ovsdb_cs_insert_server_row(cs, &ru->row_uuid, idx); } ovsdb_cs_update_server_row(row, ru->columns, ru->type == OVSDB_CS_ROW_XOR); } } } - ovsdb_cs_db_update_destroy(du); } @@ -1820,20 +1826,26 @@ server_column_get_uuid(const struct server_row *row, return d->n == 1 ? &d->keys[0].uuid : default_value; } -static const struct server_row * -ovsdb_find_server_row(struct ovsdb_cs *cs) +static size_t +ovsdb_find_server_rows(struct ovsdb_cs *cs, + const struct server_row *rows[], size_t max_rows) { const struct server_row *row; + size_t n = 0; + HMAP_FOR_EACH (row, hmap_node, &cs->server_rows) { const struct uuid *cid = server_column_get_uuid(row, COL_CID, NULL); const char *name = server_column_get_string(row, COL_NAME, NULL); if (uuid_is_zero(&cs->cid) ? (name && !strcmp(cs->data.db_name, name)) : (cid && uuid_equals(cid, &cs->cid))) { - return row; + rows[n++] = row; + if (n == max_rows) { + break; + } } } - return NULL; + return n; } static void OVS_UNUSED @@ -1858,27 +1870,18 @@ ovsdb_log_server_rows(const struct ovsdb_cs *cs) } static bool -ovsdb_cs_check_server_db__(struct ovsdb_cs *cs) +ovsdb_cs_check_server_db_row(struct ovsdb_cs *cs, + const struct server_row *db_row) { - struct ovsdb_cs_event *event; - LIST_FOR_EACH_POP (event, list_node, &cs->server.events) { - ovsdb_cs_process_server_event(cs, event); - ovsdb_cs_event_destroy(event); - } - - const struct server_row *db_row = ovsdb_find_server_row(cs); - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 5); const char *server_name = jsonrpc_session_get_name(cs->session); - if (!db_row) { - VLOG_INFO_RL(&rl, "%s: server does not have %s database", - server_name, cs->data.db_name); - return false; - } - - bool ok = false; const char *model = server_column_get_string(db_row, COL_MODEL, ""); const char *schema = server_column_get_string(db_row, COL_SCHEMA, NULL); - if (!strcmp(model, "clustered")) { + bool ok = false; + + if (cs->leader_only && db_row->synced_table) { + VLOG_INFO("%s is a replication server and therefore not a leader; " + "trying another server", server_name); + } else if (!strcmp(model, "clustered")) { bool connected = server_column_get_bool(db_row, COL_CONNECTED, false); bool leader = server_column_get_bool(db_row, COL_LEADER, false); uint64_t index = server_column_get_int(db_row, COL_INDEX, 0); @@ -1906,7 +1909,38 @@ ovsdb_cs_check_server_db__(struct ovsdb_cs *cs) ok = true; } } - if (!ok) { + return ok; +} + +static bool +ovsdb_cs_check_server_db__(struct ovsdb_cs *cs) +{ + struct ovsdb_cs_event *event; + LIST_FOR_EACH_POP (event, list_node, &cs->server.events) { + ovsdb_cs_process_server_event(cs, event); + ovsdb_cs_event_destroy(event); + } + + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 5); + const char *server_name = jsonrpc_session_get_name(cs->session); + const struct server_row *db_row[2]; + size_t n_rows = ovsdb_find_server_rows(cs, db_row, ARRAY_SIZE(db_row)); + const char *schema = NULL; + + for (size_t i = 0; i < n_rows; i++) { + if (!ovsdb_cs_check_server_db_row(cs, db_row[i])) { + return false; + } + /* Getting the schema from the original Database table, because that + * is what we actually will monitor. */ + if (!db_row[i]->synced_table) { + schema = server_column_get_string(db_row[i], COL_SCHEMA, NULL); + } + } + + if (!n_rows || !schema) { + VLOG_INFO_RL(&rl, "%s: server does not have %s database", + server_name, cs->data.db_name); return false; } @@ -1946,6 +1980,7 @@ ovsdb_cs_compose_server_monitor_request(const struct json *schema_json, struct json *monitor_requests = json_object_create(); const char *table_name = "Database"; + const char *synced_table_name = "_synced_Database"; const struct sset *table_schema = schema ? shash_find_data(schema, table_name) : NULL; if (!table_schema) { @@ -1972,6 +2007,12 @@ ovsdb_cs_compose_server_monitor_request(const struct json *schema_json, json_object_put(monitor_request, "columns", columns); json_object_put(monitor_requests, table_name, json_array_create_1(monitor_request)); + if (shash_find_data(schema, synced_table_name)) { + struct json *synced_request = json_deep_clone(monitor_request); + + json_object_put(monitor_requests, synced_table_name, + json_array_create_1(synced_request)); + } } ovsdb_cs_free_schema(schema); @@ -2022,11 +2063,14 @@ ovsdb_cs_parse_schema(const struct json *schema_json) SHASH_FOR_EACH (node, json_object(tables_json)) { const char *table_name = node->name; const struct json *json = node->data; - const struct json *columns_json; + const struct json *columns_json, *copy_for_replication; ovsdb_parser_init(&parser, json, "table schema for table %s", table_name); columns_json = ovsdb_parser_member(&parser, "columns", OP_OBJECT); + copy_for_replication = ovsdb_parser_member(&parser, + "copyForReplication", + OP_BOOLEAN | OP_OPTIONAL); error = ovsdb_parser_destroy(&parser); if (error) { log_error(error); @@ -2043,6 +2087,14 @@ ovsdb_cs_parse_schema(const struct json *schema_json) sset_add(columns, column_name); } shash_add(schema, table_name, columns); + + if (copy_for_replication && json_boolean(copy_for_replication)) { + struct sset *synced_columns = xmalloc(sizeof *synced_columns); + + sset_clone(synced_columns, columns); + shash_add_nocopy(schema, xasprintf("_synced_%s", table_name), + synced_columns); + } } return schema; } diff --git a/tests/ovsdb-cluster.at b/tests/ovsdb-cluster.at index cf43e9cf8..abf57a6b9 100644 --- a/tests/ovsdb-cluster.at +++ b/tests/ovsdb-cluster.at @@ -66,16 +66,19 @@ EXECUTION_EXAMPLES AT_BANNER([OVSDB - disconnect from cluster]) OVS_START_SHELL_HELPERS -# ovsdb_test_cluster_disconnect N_SERVERS LEADER_OR_FOLLOWER [CHECK_FLAPPING] +# ovsdb_test_cluster_disconnect N_SERVERS LEADER_OR_FOLLOWER [CHECK_FLAPPING [REPLICATION]] # Test server disconnected from the cluster. # N_SERVERS: Number of servers in the cluster. # LEADER_OR_FOLLOWER: The role of the server that is disconnected from the # cluster: "leader" or "follower". # CHECK_FLAPPING: Whether to check if is_disconnected flapped. "yes", "no". +# REPLICATION: Whether to start replication server and connect client to it: +# "yes" or "no" ovsdb_test_cluster_disconnect () { n=$1 leader_or_follower=$2 check_flapping=$3 + replication=$4 schema_name=`ovsdb-tool schema-name $abs_srcdir/idltest.ovsschema` ordinal_schema > schema AT_CHECK([ovsdb-tool '-vPATTERN:console:%c|%p|%m' create-cluster s1.db $abs_srcdir/idltest.ovsschema unix:s1.raft], [0], [], [stderr]) @@ -119,7 +122,22 @@ ovsdb_test_cluster_disconnect () { # Connect to $target. Use "wait" to trigger a non-op transaction so # that test-ovsdb will not quit. - test-ovsdb '-vPATTERN:console:test-ovsdb|%c|%m' -v -t10 idl unix:s$target.ovsdb '[["idltest", + client_target=$target + if test X$replication == X"yes"; then + AT_CHECK([ovsdb-tool create s${target}_backup.db $abs_srcdir/idltest.ovsschema]) + AT_CHECK([ovsdb-server -v -vconsole:off -vsyslog:off --detach dnl + --no-chdir --log-file=s${target}_backup.log dnl + --pidfile=s${target}_backup.pid dnl + --unixctl=s${target}_backup dnl + --sync-from=unix:s$target.ovsdb dnl + --remote=punix:s${target}_backup.ovsdb dnl + s${target}_backup.db]) + AT_CHECK([ovsdb_client_wait unix:s${target}_backup.ovsdb $schema_name connected]) + client_target=${target}_backup + cleanup="$cleanup ${target}_backup" + fi + + test-ovsdb '-vPATTERN:console:test-ovsdb|%c|%m' -v -t10 idl unix:s${client_target}.ovsdb '[["idltest", {"op": "wait", "table": "simple", "where": [["i", "==", 1]], @@ -179,6 +197,21 @@ AT_KEYWORDS([ovsdb server negative unix cluster disconnect]) ovsdb_test_cluster_disconnect 5 leader yes AT_CLEANUP +AT_SETUP([OVSDB cluster - follower disconnect from cluster, single remote, replication]) +AT_KEYWORDS([ovsdb server negative unix cluster disconnect replication]) +ovsdb_test_cluster_disconnect 3 follower no yes +AT_CLEANUP + +AT_SETUP([OVSDB cluster - leader disconnect from cluster, single remote, replication]) +AT_KEYWORDS([ovsdb server negative unix cluster disconnect replication]) +ovsdb_test_cluster_disconnect 3 leader no yes +AT_CLEANUP + +AT_SETUP([OVSDB cluster - leader disconnect from cluster, check flapping, replication]) +AT_KEYWORDS([ovsdb server negative unix cluster disconnect replication]) +ovsdb_test_cluster_disconnect 5 leader yes yes +AT_CLEANUP + AT_SETUP([OVSDB cluster - initial status should be disconnected]) AT_KEYWORDS([ovsdb server negative unix cluster disconnect]) From patchwork Sat May 1 00:55:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1472588 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FX9mK1N3pz9sSC for ; Sat, 1 May 2021 10:56:33 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 8492E432AD; Sat, 1 May 2021 00:56:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4tJQClWRnZ_U; Sat, 1 May 2021 00:56:27 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id 0C6D2432BA; Sat, 1 May 2021 00:56:26 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E040DC000E; Sat, 1 May 2021 00:56:25 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 29530C0026 for ; Sat, 1 May 2021 00:56:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 11E21850E6 for ; Sat, 1 May 2021 00:56:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7DBhtEqMomQh for ; Sat, 1 May 2021 00:56:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by smtp1.osuosl.org (Postfix) with ESMTPS id D767F84F96 for ; Sat, 1 May 2021 00:56:18 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 5B6F4240008; Sat, 1 May 2021 00:56:16 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 1 May 2021 02:55:46 +0200 Message-Id: <20210501005548.3071269-6-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210501005548.3071269-1-i.maximets@ovn.org> References: <20210501005548.3071269-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH 5/7] python: idl: Monitor _synced_Database table. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The same change as for ovsdb-cs module. Monitoring data from '_synced_Database' table to check the state of clustered databases from which this replica receives updates. Not allowing 'leader-only' connection to the replication server as this type of connections typically used for database locking. Otherwise making decision about re-connection as usual, but considering rows from both tables. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- python/ovs/db/idl.py | 65 +++++++++++++++++++++++++++++++------------- 1 file changed, 46 insertions(+), 19 deletions(-) diff --git a/python/ovs/db/idl.py b/python/ovs/db/idl.py index 4226d1cb2..0d581a110 100644 --- a/python/ovs/db/idl.py +++ b/python/ovs/db/idl.py @@ -587,16 +587,18 @@ class Idl(object): def __send_server_monitor_request(self): self.state = self.IDL_S_SERVER_MONITOR_REQUESTED monitor_requests = {} - table = self.server_tables[self._server_db_table] - columns = [column for column in table.columns.keys()] - for column in table.columns.values(): - if not hasattr(column, 'alert'): - column.alert = True - table.rows = custom_index.IndexedRows(table) - table.need_table = False - table.idl = self - monitor_request = {"columns": columns} - monitor_requests[table.name] = [monitor_request] + + for table in self.server_tables.values(): + columns = [column for column in table.columns.keys()] + for column in table.columns.values(): + if not hasattr(column, 'alert'): + column.alert = True + table.rows = custom_index.IndexedRows(table) + table.need_table = False + table.idl = self + monitor_request = {"columns": columns} + monitor_requests[table.name] = [monitor_request] + msg = ovs.jsonrpc.Message.create_request( 'monitor', [self._server_db.name, str(self.server_monitor_uuid), @@ -749,17 +751,12 @@ class Idl(object): return Notice(op, row, Row.from_json(self, table, uuid, old)) return False - def __check_server_db(self): - """Returns True if this is a valid server database, False otherwise.""" + def __check_server_db_table(self, table_name): + """Returns True if this is a valid server database table, + False otherwise.""" session_name = self.session_name() - if self._server_db_table not in self.server_tables: - vlog.info("%s: server does not have %s table in its %s database" - % (session_name, self._server_db_table, - self._server_db_name)) - return False - - rows = self.server_tables[self._server_db_table].rows + rows = self.server_tables[table_name].rows database = None for row in rows.values(): @@ -801,6 +798,30 @@ class Idl(object): return True + def __check_server_db(self): + """Returns True if this is a valid server database, False otherwise.""" + session_name = self.session_name() + + if self._server_db_table not in self.server_tables: + vlog.info("%s: server does not have %s table in its %s database" + % (session_name, self._server_db_table, + self._server_db_name)) + return False + + if not self.__check_server_db_table(self._server_db_table): + return False + + synced_table_name = '_synced_' + self._server_db_table + if (synced_table_name in self.server_tables and + len(self.server_tables[synced_table_name].rows) > 0): + if self.leader_only: + vlog.info('%s is a replication server and therefore not a ' + 'leader; trying another server' % session_name) + if not self.__check_server_db_table(synced_table_name): + return False + + return True + def __column_name(self, column): if column.type.key.type == ovs.db.types.UuidType: return ovs.ovsuuid.to_json(column.type.key.type.default) @@ -2019,6 +2040,12 @@ class SchemaHelper(object): schema_tables[table] = ( self._keep_table_columns(schema, table, columns)) + if schema_tables[table].copy_for_replication: + synced_table_name = '_synced_' + table + schema_tables[synced_table_name] = ( + self._keep_table_columns(schema, synced_table_name, + columns)) + schema.tables = schema_tables schema.readonly = self._readonly return schema From patchwork Sat May 1 00:55:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1472590 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FX9mM32Xsz9sSC for ; Sat, 1 May 2021 10:56:35 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 1A4FE8511A; Sat, 1 May 2021 00:56:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c5HkVpiMLN_p; Sat, 1 May 2021 00:56:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTP id CBF9284D23; Sat, 1 May 2021 00:56:29 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 97620C000E; Sat, 1 May 2021 00:56:29 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 405BDC0001 for ; Sat, 1 May 2021 00:56:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 665BC850D5 for ; Sat, 1 May 2021 00:56:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id k60gf2CDCWhq for ; Sat, 1 May 2021 00:56:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1A2E2850ED for ; Sat, 1 May 2021 00:56:20 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 9E06C240003; Sat, 1 May 2021 00:56:18 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 1 May 2021 02:55:47 +0200 Message-Id: <20210501005548.3071269-7-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210501005548.3071269-1-i.maximets@ovn.org> References: <20210501005548.3071269-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH 6/7] ovsdb: Report connection state for replication server. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Clients may want to know if current replica actually has an active connection with the replication source. For example, if one of the replicas lost connection, clients may want to re-connect to another replica in order to receive all the up-to-date database content. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- NEWS | 3 +++ lib/ovsdb-cs.c | 6 +++++- ovsdb/_server.xml | 18 ++++++++++-------- ovsdb/ovsdb-server.c | 16 +++++++++------- ovsdb/replication.c | 8 ++++++++ ovsdb/replication.h | 7 ++++--- python/ovs/db/idl.py | 9 +++++++-- 7 files changed, 46 insertions(+), 21 deletions(-) diff --git a/NEWS b/NEWS index d2b12b23c..fe25f2fc0 100644 --- a/NEWS +++ b/NEWS @@ -8,6 +8,9 @@ Post-v2.15.0 replication source (active server). Data from the 'Database' table of the active server stored in '_synced_Database' table on backup and available for monitoring. + * Replication (backup) server now reports 'connected: false' in Database + table of _Server database if it's not currently connected to the + replication source (active server). - In ovs-vsctl and vtep-ctl, the "find" command now accept new operators {in} and {not-in}. - Userspace datapath: diff --git a/lib/ovsdb-cs.c b/lib/ovsdb-cs.c index 7e861d1f1..4d6c41030 100644 --- a/lib/ovsdb-cs.c +++ b/lib/ovsdb-cs.c @@ -1876,13 +1876,13 @@ ovsdb_cs_check_server_db_row(struct ovsdb_cs *cs, const char *server_name = jsonrpc_session_get_name(cs->session); const char *model = server_column_get_string(db_row, COL_MODEL, ""); const char *schema = server_column_get_string(db_row, COL_SCHEMA, NULL); + bool connected = server_column_get_bool(db_row, COL_CONNECTED, false); bool ok = false; if (cs->leader_only && db_row->synced_table) { VLOG_INFO("%s is a replication server and therefore not a leader; " "trying another server", server_name); } else if (!strcmp(model, "clustered")) { - bool connected = server_column_get_bool(db_row, COL_CONNECTED, false); bool leader = server_column_get_bool(db_row, COL_LEADER, false); uint64_t index = server_column_get_int(db_row, COL_INDEX, 0); @@ -1905,6 +1905,10 @@ ovsdb_cs_check_server_db_row(struct ovsdb_cs *cs, } else { if (!schema) { VLOG_INFO("%s: missing database schema", server_name); + } else if (!connected) { + VLOG_INFO("%s: replication server is disconnected from the " + "replication source; trying another server", + server_name); } else { ok = true; } diff --git a/ovsdb/_server.xml b/ovsdb/_server.xml index 2e8b3393d..cf1b8b61b 100644 --- a/ovsdb/_server.xml +++ b/ovsdb/_server.xml @@ -78,6 +78,16 @@ database, this is empty until it finishes joining its cluster. + + True if the database is connected to its storage. A standalone database + is always connected. A clustered database is connected if the server is + in contact with a majority of its cluster. An active-backup database + is connected if the server is in contact with the replication source, + i.e. is connected to the server it syncs from. + An unconnected database cannot be modified and its data might be + unavailable or stale. + +

These columns are most interesting and in some cases only relevant for @@ -85,14 +95,6 @@ column is clustered.

- - True if the database is connected to its storage. A standalone or - active-backup database is always connected. A clustered database is - connected if the server is in contact with a majority of its cluster. - An unconnected database cannot be modified and its data might be - unavailable or stale. - - True if the database is the leader in its cluster. For a standalone or active-backup database, this is always true. diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index 89780dac8..93de2d1be 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -139,7 +139,7 @@ static void report_error_if_changed(char *error, char **last_errorp); static void update_remote_status(const struct ovsdb_jsonrpc_server *jsonrpc, const struct sset *remotes, struct shash *all_dbs); -static void update_server_status(struct shash *all_dbs); +static void update_server_status(struct shash *all_dbs, bool is_backup); static void save_config__(FILE *config_file, const struct sset *remotes, const struct sset *db_filenames, @@ -267,7 +267,7 @@ main_loop(struct server_config *config, update_remote_status(jsonrpc, remotes, all_dbs); } - update_server_status(all_dbs); + update_server_status(all_dbs, *is_backup); memory_wait(); if (*is_backup) { @@ -1154,13 +1154,14 @@ update_remote_status(const struct ovsdb_jsonrpc_server *jsonrpc, /* Updates 'row', a row in the _Server database's Database table, to match * 'db'. */ static void -update_database_status(struct ovsdb_row *row, struct db *db) +update_database_status(struct ovsdb_row *row, struct db *db, bool is_backup) { ovsdb_util_write_string_column(row, "name", db->db->name); ovsdb_util_write_string_column(row, "model", ovsdb_storage_get_model(db->db->storage)); ovsdb_util_write_bool_column(row, "connected", - ovsdb_storage_is_connected(db->db->storage)); + is_backup ? replication_is_connected() + : ovsdb_storage_is_connected(db->db->storage)); ovsdb_util_write_bool_column(row, "leader", ovsdb_storage_is_leader(db->db->storage)); ovsdb_util_write_uuid_column(row, "cid", @@ -1195,7 +1196,7 @@ update_database_status(struct ovsdb_row *row, struct db *db) /* Updates the Database table in the _Server database. */ static void -update_server_status(struct shash *all_dbs) +update_server_status(struct shash *all_dbs, bool is_backup) { struct db *server_db = shash_find_data(all_dbs, "_Server"); struct ovsdb_table *database_table = shash_find_data( @@ -1212,7 +1213,8 @@ update_server_status(struct shash *all_dbs) if (!db || !db->db) { ovsdb_txn_row_delete(txn, row); } else { - update_database_status(ovsdb_txn_row_modify(txn, row), db); + update_database_status(ovsdb_txn_row_modify(txn, row), + db, is_backup); } } @@ -1238,7 +1240,7 @@ update_server_status(struct shash *all_dbs) /* Add row. */ struct ovsdb_row *new_row = ovsdb_row_create(database_table); uuid_generate(ovsdb_row_get_uuid_rw(new_row)); - update_database_status(new_row, db); + update_database_status(new_row, db, is_backup); ovsdb_txn_row_insert(txn, new_row); next:; diff --git a/ovsdb/replication.c b/ovsdb/replication.c index 5aac325f2..a02f2514d 100644 --- a/ovsdb/replication.c +++ b/ovsdb/replication.c @@ -952,6 +952,14 @@ replication_is_alive(void) return false; } +/* Returns 'true' if there is an active established connection with a + * replication source. */ +bool +replication_is_connected(void) +{ + return session ? jsonrpc_session_is_connected(session) : false; +} + /* Return the last error reported on a connection by 'session'. The * return value is 0 if replication is not currently running, or * if replication session has not encountered any error. diff --git a/ovsdb/replication.h b/ovsdb/replication.h index 6d1be820f..bfef4f32a 100644 --- a/ovsdb/replication.h +++ b/ovsdb/replication.h @@ -35,9 +35,9 @@ struct ovsdb; * - replication_destroy() should be called when OVSDB server shutdown to * reclaim resources. * - * - replication_run(), replication_wait(), replication_is_alive() and - * replication_get_last_error() should be call within the main loop - * whenever OVSDB server runs in the backup mode. + * - replication_run(), replication_wait(), replication_is_alive(), + * replication_is_connected() and replication_get_last_error() should be + * called within the main loop whenever OVSDB server runs in the backup mode. * * - set_excluded_tables(), get_excluded_tables(), disconnect_active_server() * and replication_usage() are support functions used mainly by unixctl @@ -54,6 +54,7 @@ void replication_destroy(void); void replication_usage(void); void replication_add_local_db(const char *databse, struct ovsdb *db); bool replication_is_alive(void); +bool replication_is_connected(void); int replication_get_last_error(void); char *replication_status(void); void replication_set_probe_interval(int); diff --git a/python/ovs/db/idl.py b/python/ovs/db/idl.py index 0d581a110..f8474d8de 100644 --- a/python/ovs/db/idl.py +++ b/python/ovs/db/idl.py @@ -774,8 +774,13 @@ class Idl(object): % (session_name, self._db.name)) return False - if (database.model == CLUSTERED and - self._session.get_num_of_remotes() > 1): + if database.model != CLUSTERED: + if not database.connected: + vlog.info('%s: replication server is disconnected from the ' + 'replication source; trying another server' + % session_name) + elif (database.model == CLUSTERED and + self._session.get_num_of_remotes() > 1): if not database.schema: vlog.info('%s: clustered database server has not yet joined ' 'cluster; trying another server' % session_name) From patchwork Sat May 1 00:55:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1472592 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FX9mp4l0Gz9sSC for ; Sat, 1 May 2021 10:56:58 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 92095432ED; Sat, 1 May 2021 00:56:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c4-YyfMr_lRP; Sat, 1 May 2021 00:56:55 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id 91D65432D8; Sat, 1 May 2021 00:56:54 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 750FBC000E; Sat, 1 May 2021 00:56:54 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 18E0AC000E for ; Sat, 1 May 2021 00:56:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 76BDB6FA76 for ; Sat, 1 May 2021 00:56:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IvM_xitBym00 for ; Sat, 1 May 2021 00:56:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6295B60825 for ; Sat, 1 May 2021 00:56:23 +0000 (UTC) X-Originating-IP: 78.45.89.65 Received: from im-t490s.redhat.com (ip-78-45-89-65.net.upcbroadband.cz [78.45.89.65]) (Authenticated sender: i.maximets@ovn.org) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id F32E2240005; Sat, 1 May 2021 00:56:20 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Sat, 1 May 2021 02:55:48 +0200 Message-Id: <20210501005548.3071269-8-i.maximets@ovn.org> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210501005548.3071269-1-i.maximets@ovn.org> References: <20210501005548.3071269-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH 7/7] python: idl: Allow retry even when using a single remote. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" As described in commit [1], it's possible that remote IP is backed by a load-balancer and re-connection to this same IP will lead to connection to a different server. This case is supported for C version of IDL and should be supported in a same way for python implementation. [1] ca367fa5f8bb ("ovsdb-idl.c: Allows retry even when using a single remote.") Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- python/ovs/db/idl.py | 3 +-- tests/ovsdb-cluster.at | 14 ++++++++++++-- tests/test-ovsdb.py | 2 +- 3 files changed, 14 insertions(+), 5 deletions(-) diff --git a/python/ovs/db/idl.py b/python/ovs/db/idl.py index f8474d8de..65c1c787b 100644 --- a/python/ovs/db/idl.py +++ b/python/ovs/db/idl.py @@ -779,8 +779,7 @@ class Idl(object): vlog.info('%s: replication server is disconnected from the ' 'replication source; trying another server' % session_name) - elif (database.model == CLUSTERED and - self._session.get_num_of_remotes() > 1): + elif database.model == CLUSTERED: if not database.schema: vlog.info('%s: clustered database server has not yet joined ' 'cluster; trying another server' % session_name) diff --git a/tests/ovsdb-cluster.at b/tests/ovsdb-cluster.at index abf57a6b9..385415afe 100644 --- a/tests/ovsdb-cluster.at +++ b/tests/ovsdb-cluster.at @@ -137,17 +137,26 @@ ovsdb_test_cluster_disconnect () { cleanup="$cleanup ${target}_backup" fi - test-ovsdb '-vPATTERN:console:test-ovsdb|%c|%m' -v -t10 idl unix:s${client_target}.ovsdb '[["idltest", + txn='[["idltest", {"op": "wait", "table": "simple", "where": [["i", "==", 1]], "columns": ["i"], "until": "==", - "rows": [{"i": 1}]}]]' > test-ovsdb.log 2>&1 & + "rows": [{"i": 1}]}]]' + + test-ovsdb '-vPATTERN:console:test-ovsdb|%c|%m' -v -t10 idl \ + unix:s${client_target}.ovsdb "$txn" > test-ovsdb.log 2>&1 & echo $! > test-ovsdb.pid OVS_WAIT_UNTIL([grep "000: table simple: i=1" test-ovsdb.log]) + $PYTHON3 $srcdir/test-ovsdb.py -t10 idl $abs_srcdir/idltest.ovsschema \ + unix:s${client_target}.ovsdb "$txn" > test-ovsdb-py.log 2>&1 & + echo $! > test-ovsdb-py.pid + + OVS_WAIT_UNTIL([grep "000: table simple: i=1" test-ovsdb-py.log]) + # Start collecting raft_is_connected logs for $target before shutting down # any servers. tail -f s$target.log > raft_is_connected.log & @@ -160,6 +169,7 @@ ovsdb_test_cluster_disconnect () { # The test-ovsdb should detect the disconnect and retry. OVS_WAIT_UNTIL([grep disconnect test-ovsdb.log]) + OVS_WAIT_UNTIL([grep disconnect test-ovsdb-py.log]) # The $target debug log should show raft_is_connected: false. OVS_WAIT_UNTIL([grep "raft_is_connected: false" raft_is_connected.log]) diff --git a/tests/test-ovsdb.py b/tests/test-ovsdb.py index 72a319123..5bc0bf681 100644 --- a/tests/test-ovsdb.py +++ b/tests/test-ovsdb.py @@ -651,7 +651,7 @@ def do_idl(schema_file, remote, *commands): commands = commands[1:] else: schema_helper.register_all() - idl = ovs.db.idl.Idl(remote, schema_helper) + idl = ovs.db.idl.Idl(remote, schema_helper, leader_only=False) if "simple3" in idl.tables: idl.index_create("simple3", "simple3_by_name")