From patchwork Thu Sep 1 02:32:41 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Zhou X-Patchwork-Id: 664741 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from archives.nicira.com (archives.nicira.com [96.126.127.54]) by ozlabs.org (Postfix) with ESMTP id 3sPmWg2w3dz9s8x for ; Thu, 1 Sep 2016 12:33:03 +1000 (AEST) Received: from archives.nicira.com (localhost [127.0.0.1]) by archives.nicira.com (Postfix) with ESMTP id 9F154109F2; Wed, 31 Aug 2016 19:33:02 -0700 (PDT) X-Original-To: dev@openvswitch.com Delivered-To: dev@openvswitch.com Received: from mail-pa0-f47.google.com (mail-pa0-f47.google.com [209.85.220.47]) by archives.nicira.com (Postfix) with ESMTPS id 8BA57109EC for ; Wed, 31 Aug 2016 19:33:01 -0700 (PDT) Received: by mail-pa0-f47.google.com with SMTP id hb8so24468118pac.2 for ; Wed, 31 Aug 2016 19:33:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=b84qx94lDGIcFEfVGAfCBDc3dHsxT+OgT8buG+487nw=; b=I86vRtOhplz7fGzcf7wbzju7yuCFWti/j3bW1OgGECjdK/nxtiOjhp9V4zWCw8Yr36 VkUjOU4KhfJaDGE1cUAX0L5nGl02JVoAKrDh3rH90vAx/pQkvs37erk7LsTUU06Pys58 zq+2lkHfnWGpbj6KfzfaaTiMuskZ4UC2pIjnh7gpLI9Sa7GzYNpZEI3F6aV4bZN5DeUK e8SvUk+G0hX88W/KXF0anPtYb9fvlFOtUOP+x8V3GX4ghzkY6YPNGuLDavkWC+McPspa SAFCNJCllFMdCIMhHFKGAenMY2L5dYRXDjDZuvS8Lj014C0DwZRg2nz/TlWiehYxKPmi JNvA== X-Gm-Message-State: AE9vXwPhCFZy14u6zK1loxENioSpKHJT0RLDJEwf320cab+jKKy6ZW8X6ZfQoY6uRwnftw== X-Received: by 10.66.132.38 with SMTP id or6mr22306530pab.84.1472697180431; Wed, 31 Aug 2016 19:33:00 -0700 (PDT) Received: from ubuntu.localdomain ([208.91.1.34]) by smtp.gmail.com with ESMTPSA id uk2sm2604842pab.39.2016.08.31.19.32.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 31 Aug 2016 19:32:59 -0700 (PDT) From: Andy Zhou To: dev@openvswitch.com Date: Wed, 31 Aug 2016 19:32:41 -0700 Message-Id: <1472697161-45377-1-git-send-email-azhou@ovn.org> X-Mailer: git-send-email 1.9.1 Subject: [ovs-dev] [v3] ovsdb: Replication usability improvements X-BeenThere: dev@openvswitch.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: dev-bounces@openvswitch.org Sender: "dev" Added the '--no-sync' option base on feedbacks of current implementation. Added appctl command "ovsdb-server/sync-status" based on feedbacks of current implementation. Added the RPL_S_INIT state in the state machine. This state is not strictly necessary for the replication state machine, but is introduced to make sure the state is update immediately when the state machine is reset, via replication_init(). Without it ovsdb/sync-status may display "replicating" or crash, if the command is issued between after replication_init() is called, but before the state variable is updated from replication_run(). Added a test to simulate the integration of HA manager with OVSDB server using replication. Other documentation and API improvements. Tested-by: Numan Siddique Signed-off-by: Andy Zhou --- Documentation/OVSDB-replication.md | 2 +- ovsdb/jsonrpc-server.c | 6 + ovsdb/jsonrpc-server.h | 1 + ovsdb/ovsdb-server.1.in | 24 ++-- ovsdb/ovsdb-server.c | 251 ++++++++++++++++++++++++++----------- ovsdb/replication.c | 89 ++++++++++--- ovsdb/replication.h | 28 ++++- ovsdb/replication.man | 15 ++- tests/ovsdb-server.at | 105 ++++++++++++++-- 9 files changed, 408 insertions(+), 113 deletions(-) diff --git a/Documentation/OVSDB-replication.md b/Documentation/OVSDB-replication.md index 74c9500..a9ab5cc 100644 --- a/Documentation/OVSDB-replication.md +++ b/Documentation/OVSDB-replication.md @@ -144,7 +144,7 @@ commands are the following. between the active server and frees the memory used for the replication configuration. -- ovsdb-server/set-sync-excluded-tables {db:table,...}: sets the tables list +- ovsdb-server/set-sync-exclude-tables {db:table,...}: sets the tables list that will be excluded from being replicated. - ovsdb-server/get-sync-excluded-tables: gets the tables list that is diff --git a/ovsdb/jsonrpc-server.c b/ovsdb/jsonrpc-server.c index 45975a3..427026d 100644 --- a/ovsdb/jsonrpc-server.c +++ b/ovsdb/jsonrpc-server.c @@ -343,6 +343,12 @@ ovsdb_jsonrpc_server_reconnect(struct ovsdb_jsonrpc_server *svr, bool read_only) } } +bool +ovsdb_jsonrpc_server_is_read_only(struct ovsdb_jsonrpc_server *svr) +{ + return svr->read_only; +} + void ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *svr) { diff --git a/ovsdb/jsonrpc-server.h b/ovsdb/jsonrpc-server.h index 955bbe4..f04b1e9 100644 --- a/ovsdb/jsonrpc-server.h +++ b/ovsdb/jsonrpc-server.h @@ -65,6 +65,7 @@ void ovsdb_jsonrpc_server_reconnect(struct ovsdb_jsonrpc_server *, bool read_onl void ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *); void ovsdb_jsonrpc_server_wait(struct ovsdb_jsonrpc_server *); +bool ovsdb_jsonrpc_server_is_read_only(struct ovsdb_jsonrpc_server *); void ovsdb_jsonrpc_server_get_memory_usage(const struct ovsdb_jsonrpc_server *, struct simap *usage); diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in index d1ba83b..e2e96ae 100644 --- a/ovsdb/ovsdb-server.1.in +++ b/ovsdb/ovsdb-server.1.in @@ -36,7 +36,7 @@ If none is specified, the default is \fB@DBDIR@/conf.db\fR. The database files must already have been created and initialized using, for example, \fBovsdb\-tool create\fR. . -.SH "ACTIVE and BACKUP " +.SH "ACTIVE and BACKUP" \fBovsdb\-server\fR runs either as a backup server, or as an active server. When \fBovsdb\-server\fR is running as a backup server, all transactions that can modify the database content, including the lock commands are rejected. @@ -45,12 +45,12 @@ When \fBovsdb\-server\fR role changes, all existing client connection are reset, requiring clients to reconnect to the server. .PP By default, \fBovsdb\-server\fR runs as an active server, except when the -\fB\-\-sync\-from=\fIserver\fR command line option is specified. During -runtime, \fBovsdb\-server\fR role can be switch by using appctl commands. +\fB\-\-sync\-from=\fIserver\fR command line option is specified and without +the \fB\-\-no\-sync option\fR. During runtime, \fBovsdb\-server\fR role can be switch by using appctl commands. .PP \fBovsdb-server/connect\-active\-ovsdb\-server\fR switches -\fBovsdb\-server\fR role into a backup server, Conversely, -\fBovsdb-server/disconnect\-active\-ovsdb\-server\fR changes server into +\fBovsdb\-server\fR into a backup server, Conversely, +\fBovsdb-server/disconnect\-active\-ovsdb\-server\fR switches server into an active one. . .SH OPTIONS @@ -220,12 +220,22 @@ specified by \fBovsdb\-server/set\-active\-ovsdb\-server\fR. .IP "\fBovsdb\-server/disconnect\-active\-ovsdb\-server" Causes \fBovsdb\-server\fR to stop synchronizing its databases with a active server. . -.IP "\fBovsdb\-server/set\-sync\-excluded\-tables \fIdb\fB:\fItable\fR[\fB,\fIdb\fB:\fItable\fR]..." +.IP "\fBovsdb\-server/set\-sync\-exclude\-tables \fIdb\fB:\fItable\fR[\fB,\fIdb\fB:\fItable\fR]..." Sets the \fItable\fR whitin \fIdb\fR that will be excluded from synchronization. . -.IP "\fBovsdb\-server/get\-sync\-excluded\-tables" +.IP "\fBovsdb\-server/get\-sync\-exclude\-tables" Gets the tables that are currently excluded from synchronization. . +.IP "\fBovsdb\-server/sync\-status" +Prints a summary of replication run time information. The \fBstate\fR +information is always provided, indicating whether the server is running +in the \fIactive\fR or the \fIbackup\fR mode. +When running in backup mode, replication connection status, which +can be either \fIconnecting\fR, \fIreplicating\fR or \fIerror\fR, are shown. +When the connection is in \fIreplicating\fR state, further output shows +the list of databases currently replicating, and the tables that are +excluded. +. .so lib/vlog-unixctl.man .so lib/memory-unixctl.man .so lib/coverage-unixctl.man diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index 4de370c..29092e0 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -76,9 +76,6 @@ static char *certificate_file; static char *ca_cert_file; static bool bootstrap_ca_cert; -/* Replication current state. */ -static bool is_backup_server; - static unixctl_cb_func ovsdb_server_exit; static unixctl_cb_func ovsdb_server_compact; static unixctl_cb_func ovsdb_server_reconnect; @@ -89,13 +86,17 @@ static unixctl_cb_func ovsdb_server_set_active_ovsdb_server; static unixctl_cb_func ovsdb_server_get_active_ovsdb_server; static unixctl_cb_func ovsdb_server_connect_active_ovsdb_server; static unixctl_cb_func ovsdb_server_disconnect_active_ovsdb_server; -static unixctl_cb_func ovsdb_server_set_sync_excluded_tables; -static unixctl_cb_func ovsdb_server_get_sync_excluded_tables; +static unixctl_cb_func ovsdb_server_set_sync_exclude_tables; +static unixctl_cb_func ovsdb_server_get_sync_exclude_tables; +static unixctl_cb_func ovsdb_server_get_sync_status; struct server_config { struct sset *remotes; struct shash *all_dbs; FILE *config_tmpfile; + char **sync_from; + char **sync_exclude; + bool *is_backup; struct ovsdb_jsonrpc_server *jsonrpc; }; static unixctl_cb_func ovsdb_server_add_remote; @@ -111,7 +112,8 @@ static void close_db(struct db *db); static void parse_options(int *argc, char **argvp[], struct sset *remotes, char **unixctl_pathp, - char **run_command); + char **run_command, char **sync_from, + char **sync_exclude, bool *is_backup); OVS_NO_RETURN static void usage(void); static char *reconfigure_remotes(struct ovsdb_jsonrpc_server *, @@ -125,15 +127,19 @@ static void update_remote_status(const struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs); static void save_config__(FILE *config_file, const struct sset *remotes, - const struct sset *db_filenames); + const struct sset *db_filenames, + const char *sync_from, const char *sync_exclude, + bool is_backup); static void save_config(struct server_config *); static void load_config(FILE *config_file, struct sset *remotes, - struct sset *db_filenames); + struct sset *db_filenames, char **sync_from, + char **sync_exclude, bool *is_backup); static void -ovsdb_replication_init(struct shash *all_dbs) +ovsdb_replication_init(const char *sync_from, const char *exclude, + struct shash *all_dbs) { - replication_init(); + replication_init(sync_from, exclude); struct shash_node *node; SHASH_FOR_EACH (node, all_dbs) { struct db *db = node->data; @@ -144,11 +150,12 @@ ovsdb_replication_init(struct shash *all_dbs) static void main_loop(struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs, struct unixctl_server *unixctl, struct sset *remotes, - struct process *run_process, bool *exiting) + struct process *run_process, bool *exiting, bool *is_backup) { char *remotes_error, *ssl_error; struct shash_node *node; long long int status_timer = LLONG_MIN; + bool last_role = *is_backup; *exiting = false; ssl_error = NULL; @@ -172,13 +179,13 @@ main_loop(struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs, /* Run unixctl_server_run() before reconfigure_remotes() because * ovsdb-server/add-remote and ovsdb-server/remove-remote can change * the set of remotes that reconfigure_remotes() uses. */ - bool last_role = is_backup_server; unixctl_server_run(unixctl); - /* In case unixctl commands change the role of ovsdb-server, - * from active to backup or vise versa, recoonect jsonrpc server. */ - if (last_role != is_backup_server) { - ovsdb_jsonrpc_server_reconnect(jsonrpc, is_backup_server); + /* In ovsdb-server's role (active or backup) has changed, restart + * the ovsdb jsonrpc server. */ + if (last_role != *is_backup) { + bool read_only = last_role = *is_backup; + ovsdb_jsonrpc_server_reconnect(jsonrpc, read_only); } report_error_if_changed( @@ -187,7 +194,7 @@ main_loop(struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs, report_error_if_changed(reconfigure_ssl(all_dbs), &ssl_error); ovsdb_jsonrpc_server_run(jsonrpc); - if (is_backup_server) { + if (*is_backup) { replication_run(); if (!replication_is_alive()) { int retval = replication_get_last_error(); @@ -213,7 +220,7 @@ main_loop(struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs, } memory_wait(); - if (is_backup_server) { + if (*is_backup) { replication_wait(); } @@ -247,6 +254,8 @@ main(int argc, char *argv[]) struct unixctl_server *unixctl; struct ovsdb_jsonrpc_server *jsonrpc; struct sset remotes, db_filenames; + char *sync_from, *sync_exclude; + bool is_backup; const char *db_filename; struct process *run_process; bool exiting; @@ -264,7 +273,11 @@ main(int argc, char *argv[]) fatal_ignore_sigpipe(); process_init(); - parse_options(&argc, &argv, &remotes, &unixctl_path, &run_command); + bool no_sync = false; + parse_options(&argc, &argv, &remotes, &unixctl_path, &run_command, + &sync_from, &sync_exclude, &no_sync); + is_backup = sync_from && !no_sync; + daemon_become_new_user(false); /* Create and initialize 'config_tmpfile' as a temporary file to hold @@ -291,17 +304,26 @@ main(int argc, char *argv[]) server_config.remotes = &remotes; server_config.config_tmpfile = config_tmpfile; - save_config__(config_tmpfile, &remotes, &db_filenames); + save_config__(config_tmpfile, &remotes, &db_filenames, sync_from, + sync_exclude, is_backup); daemonize_start(false); /* Load the saved config. */ - load_config(config_tmpfile, &remotes, &db_filenames); - jsonrpc = ovsdb_jsonrpc_server_create(is_backup_server); + load_config(config_tmpfile, &remotes, &db_filenames, &sync_from, + &sync_exclude, &is_backup); + + /* Start ovsdb jsonrpc server. When running as a backup server, + * jsonrpc connections are read only. Otherwise, both read + * and write transactions are allowed. */ + jsonrpc = ovsdb_jsonrpc_server_create(is_backup); shash_init(&all_dbs); server_config.all_dbs = &all_dbs; server_config.jsonrpc = jsonrpc; + server_config.sync_from = &sync_from; + server_config.sync_exclude = &sync_exclude; + server_config.is_backup = &is_backup; perf_counters_init(); @@ -373,34 +395,39 @@ main(int argc, char *argv[]) ovsdb_server_perf_counters_show, NULL); unixctl_command_register("ovsdb-server/perf-counters-clear", "", 0, 0, ovsdb_server_perf_counters_clear, NULL); - - unixctl_command_register("ovsdb-server/set-active-ovsdb-server", "", 0, 1, - ovsdb_server_set_active_ovsdb_server, NULL); + unixctl_command_register("ovsdb-server/set-active-ovsdb-server", "", 1, 1, + ovsdb_server_set_active_ovsdb_server, + &server_config); unixctl_command_register("ovsdb-server/get-active-ovsdb-server", "", 0, 0, - ovsdb_server_get_active_ovsdb_server, NULL); + ovsdb_server_get_active_ovsdb_server, + &server_config); unixctl_command_register("ovsdb-server/connect-active-ovsdb-server", "", 0, 0, ovsdb_server_connect_active_ovsdb_server, - &all_dbs); + &server_config); unixctl_command_register("ovsdb-server/disconnect-active-ovsdb-server", "", 0, 0, ovsdb_server_disconnect_active_ovsdb_server, + &server_config); + unixctl_command_register("ovsdb-server/set-sync-exclude-tables", "", + 0, 1, ovsdb_server_set_sync_exclude_tables, + &server_config); + unixctl_command_register("ovsdb-server/get-sync-exclude-tables", "", + 0, 0, ovsdb_server_get_sync_exclude_tables, NULL); - unixctl_command_register("ovsdb-server/set-sync-excluded-tables", "", - 0, 1, ovsdb_server_set_sync_excluded_tables, - &all_dbs); - unixctl_command_register("ovsdb-server/get-sync-excluded-tables", "", - 0, 0, ovsdb_server_get_sync_excluded_tables, - NULL); + unixctl_command_register("ovsdb-server/sync-status", "", + 0, 0, ovsdb_server_get_sync_status, + &server_config); /* Simulate the behavior of OVS release prior to version 2.5 that * does not support the monitor_cond method. */ unixctl_command_register("ovsdb-server/disable-monitor-cond", "", 0, 0, ovsdb_server_disable_monitor_cond, jsonrpc); - if (is_backup_server) { - ovsdb_replication_init(&all_dbs); + if (is_backup) { + ovsdb_replication_init(sync_from, sync_exclude, &all_dbs); } - main_loop(jsonrpc, &all_dbs, unixctl, &remotes, run_process, &exiting); + main_loop(jsonrpc, &all_dbs, unixctl, &remotes, run_process, &exiting, + &is_backup); ovsdb_jsonrpc_server_destroy(jsonrpc); SHASH_FOR_EACH_SAFE(node, next, &all_dbs) { @@ -411,6 +438,8 @@ main(int argc, char *argv[]) shash_destroy(&all_dbs); sset_destroy(&remotes); sset_destroy(&db_filenames); + free(sync_from); + free(sync_exclude); unixctl_server_destroy(unixctl); replication_destroy(); @@ -1097,9 +1126,16 @@ report_error_if_changed(char *error, char **last_errorp) static void ovsdb_server_set_active_ovsdb_server(struct unixctl_conn *conn, int argc OVS_UNUSED, const char *argv[], - void *arg_ OVS_UNUSED) + void *config_) { - set_active_ovsdb_server(argv[1]); + struct server_config *config = config_; + + if (*config->sync_from) { + free(*config->sync_from); + } + *config->sync_from = xstrdup(argv[1]); + save_config(config); + unixctl_command_reply(conn, NULL); } @@ -1107,49 +1143,65 @@ static void ovsdb_server_get_active_ovsdb_server(struct unixctl_conn *conn, int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, - void *arg_ OVS_UNUSED) + void *config_ ) { - unixctl_command_reply(conn, get_active_ovsdb_server()); + struct server_config *config = config_; + + unixctl_command_reply(conn, *config->sync_from); } static void ovsdb_server_connect_active_ovsdb_server(struct unixctl_conn *conn, int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, - void *all_dbs_) + void *config_) { - struct shash *all_dbs = all_dbs_; + struct server_config *config = config_; + char *msg = NULL; - if (!is_backup_server) { - ovsdb_replication_init(all_dbs); + if ( !*config->sync_from) { + msg = "Unable to connect: active server is not specified.\n"; + } else { + ovsdb_replication_init(*config->sync_from, *config->sync_exclude, + config->all_dbs); + if (!*config->is_backup) { + *config->is_backup = true; + save_config(config); + } } - is_backup_server = true; - unixctl_command_reply(conn, NULL); + unixctl_command_reply(conn, msg); } static void ovsdb_server_disconnect_active_ovsdb_server(struct unixctl_conn *conn, int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, - void *arg_ OVS_UNUSED) + void *config_) { + struct server_config *config = config_; + disconnect_active_server(); - is_backup_server = false; + *config->is_backup = false; + save_config(config); unixctl_command_reply(conn, NULL); } static void -ovsdb_server_set_sync_excluded_tables(struct unixctl_conn *conn, - int argc OVS_UNUSED, - const char *argv[], - void *all_dbs_) +ovsdb_server_set_sync_exclude_tables(struct unixctl_conn *conn, + int argc OVS_UNUSED, + const char *argv[], + void *config_) { - struct shash *all_dbs = all_dbs_; - char *err = set_blacklist_tables(argv[1], true); + struct server_config *config = config_; + char *err = set_blacklist_tables(argv[1], true); if (!err) { - if (is_backup_server) { - ovsdb_replication_init(all_dbs); + free(*config->sync_exclude); + *config->sync_exclude = xstrdup(argv[1]); + save_config(config); + if (*config->is_backup) { + ovsdb_replication_init(*config->sync_from, *config->sync_exclude, + config->all_dbs); } err = set_blacklist_tables(argv[1], false); } @@ -1158,10 +1210,10 @@ ovsdb_server_set_sync_excluded_tables(struct unixctl_conn *conn, } static void -ovsdb_server_get_sync_excluded_tables(struct unixctl_conn *conn, - int argc OVS_UNUSED, - const char *argv[] OVS_UNUSED, - void *arg_ OVS_UNUSED) +ovsdb_server_get_sync_exclude_tables(struct unixctl_conn *conn, + int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, + void *arg_ OVS_UNUSED) { unixctl_command_reply(conn, get_blacklist_tables()); } @@ -1206,9 +1258,10 @@ ovsdb_server_disable_monitor_cond(struct unixctl_conn *conn, void *jsonrpc_) { struct ovsdb_jsonrpc_server *jsonrpc = jsonrpc_; + bool read_only = ovsdb_jsonrpc_server_is_read_only(jsonrpc); ovsdb_jsonrpc_disable_monitor_cond(); - ovsdb_jsonrpc_server_reconnect(jsonrpc, is_backup_server); + ovsdb_jsonrpc_server_reconnect(jsonrpc, read_only); unixctl_command_reply(conn, NULL); } @@ -1263,8 +1316,9 @@ ovsdb_server_reconnect(struct unixctl_conn *conn, int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, void *jsonrpc_) { struct ovsdb_jsonrpc_server *jsonrpc = jsonrpc_; + bool read_only = ovsdb_jsonrpc_server_is_read_only(jsonrpc); - ovsdb_jsonrpc_server_reconnect(jsonrpc, is_backup_server); + ovsdb_jsonrpc_server_reconnect(jsonrpc, read_only); unixctl_command_reply(conn, NULL); } @@ -1350,8 +1404,9 @@ ovsdb_server_add_database(struct unixctl_conn *conn, int argc OVS_UNUSED, error = open_db(config, filename); if (!error) { save_config(config); - if (is_backup_server) { - ovsdb_replication_init(config->all_dbs); + if (*config->is_backup) { + ovsdb_replication_init(*config->sync_from, *config->sync_exclude, + config->all_dbs); } unixctl_command_reply(conn, NULL); } else { @@ -1383,8 +1438,9 @@ ovsdb_server_remove_database(struct unixctl_conn *conn, int argc OVS_UNUSED, shash_delete(config->all_dbs, node); save_config(config); - if (is_backup_server) { - ovsdb_replication_init(config->all_dbs); + if (*config->is_backup) { + ovsdb_replication_init(*config->sync_from, *config->sync_exclude, + config->all_dbs); } unixctl_command_reply(conn, NULL); } @@ -1412,8 +1468,27 @@ ovsdb_server_list_databases(struct unixctl_conn *conn, int argc OVS_UNUSED, } static void +ovsdb_server_get_sync_status(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, void *config_) +{ + struct server_config *config = config_; + bool is_backup = *config->is_backup; + struct ds ds = DS_EMPTY_INITIALIZER; + + ds_put_format(&ds, "state: %s\n", is_backup ? "backup" : "active"); + + if (is_backup) { + ds_put_and_free_cstr(&ds, replication_status()); + } + + unixctl_command_reply(conn, ds_cstr(&ds)); + ds_destroy(&ds); +} + +static void parse_options(int *argcp, char **argvp[], - struct sset *remotes, char **unixctl_pathp, char **run_command) + struct sset *remotes, char **unixctl_pathp, char **run_command, + char **sync_from, char **sync_exclude, bool *no_sync) { enum { OPT_REMOTE = UCHAR_MAX + 1, @@ -1423,6 +1498,7 @@ parse_options(int *argcp, char **argvp[], OPT_PEER_CA_CERT, OPT_SYNC_FROM, OPT_SYNC_EXCLUDE, + OPT_NO_SYNC, VLOG_OPTION_ENUMS, DAEMON_OPTION_ENUMS }; @@ -1443,12 +1519,15 @@ parse_options(int *argcp, char **argvp[], {"ca-cert", required_argument, NULL, 'C'}, {"sync-from", required_argument, NULL, OPT_SYNC_FROM}, {"sync-exclude-tables", required_argument, NULL, OPT_SYNC_EXCLUDE}, + {"no-sync", no_argument, NULL, OPT_NO_SYNC}, {NULL, 0, NULL, 0}, }; char *short_options = ovs_cmdl_long_options_to_short_options(long_options); int argc = *argcp; char **argv = *argvp; + *sync_from = NULL; + *sync_exclude = NULL; sset_init(remotes); for (;;) { int c; @@ -1504,8 +1583,7 @@ parse_options(int *argcp, char **argvp[], break; case OPT_SYNC_FROM: - set_active_ovsdb_server(optarg); - is_backup_server = true; + *sync_from = xstrdup(optarg); break; case OPT_SYNC_EXCLUDE: { @@ -1513,8 +1591,12 @@ parse_options(int *argcp, char **argvp[], if (err) { ovs_fatal(0, "%s", err); } + *sync_exclude = xstrdup(optarg); break; } + case OPT_NO_SYNC: + *no_sync = true; + break; case '?': exit(EXIT_FAILURE); @@ -1568,7 +1650,8 @@ sset_to_json(const struct sset *sset) * 'remotes' and 'db_filenames'. */ static void save_config__(FILE *config_file, const struct sset *remotes, - const struct sset *db_filenames) + const struct sset *db_filenames, const char *sync_from, + const char *sync_exclude, bool is_backup) { struct json *obj; char *s; @@ -1581,6 +1664,15 @@ save_config__(FILE *config_file, const struct sset *remotes, obj = json_object_create(); json_object_put(obj, "remotes", sset_to_json(remotes)); json_object_put(obj, "db_filenames", sset_to_json(db_filenames)); + if (sync_from) { + json_object_put(obj, "sync_from", json_string_create(sync_from)); + } + if (sync_exclude) { + json_object_put(obj, "sync_exclude", + json_string_create(sync_exclude)); + } + json_object_put(obj, "is_backup", json_boolean_create(is_backup)); + s = json_to_string(obj, 0); json_destroy(obj); @@ -1606,7 +1698,9 @@ save_config(struct server_config *config) sset_add(&db_filenames, db->filename); } - save_config__(config->config_tmpfile, config->remotes, &db_filenames); + save_config__(config->config_tmpfile, config->remotes, &db_filenames, + *config->sync_from, *config->sync_exclude, + *config->is_backup); sset_destroy(&db_filenames); } @@ -1628,7 +1722,8 @@ sset_from_json(struct sset *sset, const struct json *array) /* Clears and replaces 'remotes' and 'dbnames' by a configuration read from * 'config_file', which must have been previously written by save_config(). */ static void -load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames) +load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames, + char **sync_from, char **sync_exclude, bool *is_backup) { struct json *json; @@ -1644,5 +1739,17 @@ load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames) sset_from_json(remotes, shash_find_data(json_object(json), "remotes")); sset_from_json(db_filenames, shash_find_data(json_object(json), "db_filenames")); + + struct json *string; + string = shash_find_data(json_object(json), "sync_from"); + free(*sync_from); + *sync_from = string ? xstrdup(json_string(string)) : NULL; + + string = shash_find_data(json_object(json), "sync_exclude"); + free(*sync_exclude); + *sync_exclude = string ? xstrdup(json_string(string)) : NULL; + + *is_backup = json_boolean(shash_find_data(json_object(json), "is_backup")); + json_destroy(json); } diff --git a/ovsdb/replication.c b/ovsdb/replication.c index 2245566..40a9fd0 100644 --- a/ovsdb/replication.c +++ b/ovsdb/replication.c @@ -37,7 +37,7 @@ VLOG_DEFINE_THIS_MODULE(replication); -static char *active_ovsdb_server; +static char *sync_from; static struct jsonrpc_session *session = NULL; static unsigned int session_seqno = UINT_MAX; @@ -87,6 +87,7 @@ static void request_ids_destroy(void); void request_ids_clear(void); enum ovsdb_replication_state { + RPL_S_INIT, RPL_S_DB_REQUESTED, RPL_S_SCHEMA_REQUESTED, RPL_S_MONITOR_REQUESTED, @@ -108,8 +109,15 @@ static struct ovsdb* find_db(const char *db_name); void -replication_init(void) +replication_init(const char *sync_from_, const char *exclude_tables) { + free(sync_from); + sync_from = xstrdup(sync_from_); + char *err = set_blacklist_tables(exclude_tables, false); + /* Caller should have verified that the 'exclude_tables' is + * parseable. An error here is unexpected. */ + ovs_assert(!err); + shash_destroy(replication_dbs); replication_dbs = NULL; @@ -118,8 +126,9 @@ replication_init(void) jsonrpc_session_close(session); } - session = jsonrpc_session_open(active_ovsdb_server, true); + session = jsonrpc_session_open(sync_from, true); session_seqno = UINT_MAX; + state = RPL_S_INIT; } void @@ -143,7 +152,7 @@ replication_run(void) unsigned int seqno; seqno = jsonrpc_session_get_seqno(session); - if (seqno != session_seqno) { + if (seqno != session_seqno || state == RPL_S_INIT) { session_seqno = seqno; request_ids_clear(); struct jsonrpc_msg *request; @@ -156,6 +165,7 @@ replication_run(void) replication_dbs = replication_db_clone(&local_dbs); state = RPL_S_DB_REQUESTED; + VLOG_DBG("Send list_dbs request"); } msg = jsonrpc_session_recv(session); @@ -216,6 +226,7 @@ replication_run(void) } } } + VLOG_DBG("Send schema requests"); state = RPL_S_SCHEMA_REQUESTED; } break; @@ -271,6 +282,7 @@ replication_run(void) request_ids_add(request->id, db); jsonrpc_session_send(session, request); + VLOG_DBG("Send monitor requests"); state = RPL_S_MONITOR_REQUESTED; } } @@ -289,6 +301,7 @@ replication_run(void) /* Transition to replicating state after receiving * all replies of "monitor" requests. */ if (hmap_is_empty(&request_ids)) { + VLOG_DBG("Listening to monitor updates"); state = RPL_S_REPLICATING; } } @@ -299,6 +312,7 @@ replication_run(void) /* Ignore all messages */ break; + case RPL_S_INIT: case RPL_S_REPLICATING: default: OVS_NOT_REACHED(); @@ -318,18 +332,6 @@ replication_wait(void) } } -void -set_active_ovsdb_server(const char *active_server) -{ - active_ovsdb_server = nullable_xstrdup(active_server); -} - -const char * -get_active_ovsdb_server(void) -{ - return active_ovsdb_server; -} - /* Parse 'blacklist' to rebuild 'blacklist_tables'. If 'dryrun' is false, the * current black list tables will be wiped out, regardless of whether * 'blacklist' can be parsed. If 'dryrun' is true, only parses 'blacklist' and @@ -457,9 +459,9 @@ replication_destroy(void) blacklist_tables_clear(); shash_destroy(&blacklist_tables); - if (active_ovsdb_server) { - free(active_ovsdb_server); - active_ovsdb_server = NULL; + if (sync_from) { + free(sync_from); + sync_from = NULL; } request_ids_destroy(); @@ -855,12 +857,59 @@ replication_get_last_error(void) return err; } +char * +replication_status(void) +{ + bool alive = session && jsonrpc_session_is_alive(session); + struct ds ds = DS_EMPTY_INITIALIZER; + + if (alive) { + switch(state) { + case RPL_S_INIT: + case RPL_S_DB_REQUESTED: + case RPL_S_SCHEMA_REQUESTED: + case RPL_S_MONITOR_REQUESTED: + ds_put_format(&ds, "connecting: %s", sync_from); + break; + case RPL_S_REPLICATING: { + struct shash_node *node; + + ds_put_format(&ds, "replicating: %s\n", sync_from); + ds_put_cstr(&ds, "database:"); + SHASH_FOR_EACH (node, replication_dbs) { + ds_put_format(&ds, " %s,", node->name); + } + ds_chomp(&ds, ','); + + if (!shash_is_empty(&blacklist_tables)) { + ds_put_char(&ds, '\n'); + ds_put_cstr(&ds, "exclude: "); + ds_put_and_free_cstr(&ds, get_blacklist_tables()); + } + break; + } + case RPL_S_ERR: + ds_put_format(&ds, "Replication to (%s) failed\n", sync_from); + break; + default: + OVS_NOT_REACHED(); + break; + } + } else { + ds_put_format(&ds, "not connected to %s", sync_from); + } + return ds_steal_cstr(&ds); +} + void replication_usage(void) { printf("\n\ Syncing options:\n\ --sync-from=SERVER sync DATABASE from active SERVER\n\ + start in backup mode, except when\n\ + --no-sync is also specified\n\ --sync-exclude-tables=DB:TABLE,...\n\ - exclude the TABLE in DB from syncing\n"); + exclude the TABLE in DB from syncing\n\ + --no-sync start in active mode\n"); } diff --git a/ovsdb/replication.h b/ovsdb/replication.h index c873648..622d355 100644 --- a/ovsdb/replication.h +++ b/ovsdb/replication.h @@ -21,7 +21,30 @@ #include struct ovsdb; -void replication_init(void); +/* Replication module runs when OVSDB server runs in the backup mode. + * + * API Usage + *=========== + * + * - replication_init() needs to be called whenever OVSDB server switches into + * the backup mode. + * + * - replication_add_local_db() should be called immediately after to add all + * known database that OVSDB server owns, one at a time. + * + * - replication_destroy() should be called when OVSDB server shutdown to + * reclaim resources. + * + * - replication_run(), replication_wait(), replication_is_alive() and + * replication_get_last_error() should be call within the main loop + * whenever OVSDB server runs in the backup mode. + * + * - set_blacklist_tables(), get_blacklist_tables(), + * disconnect_active_server() and replication_usage() are support functions + * used mainly by uinxctl commands. + */ + +void replication_init(const char *sync_from, const char *exclude_tables); void replication_run(void); void replication_wait(void); void replication_destroy(void); @@ -29,9 +52,8 @@ void replication_usage(void); void replication_add_local_db(const char *databse, struct ovsdb *db); bool replication_is_alive(void); int replication_get_last_error(void); +char *replication_status(void); -void set_active_ovsdb_server(const char *remote_server); -const char *get_active_ovsdb_server(void); char *set_blacklist_tables(const char *blacklist, bool dryrun) OVS_WARN_UNUSED_RESULT; char *get_blacklist_tables(void) OVS_WARN_UNUSED_RESULT; diff --git a/ovsdb/replication.man b/ovsdb/replication.man index 26420bc..be827b9 100644 --- a/ovsdb/replication.man +++ b/ovsdb/replication.man @@ -3,6 +3,17 @@ with another running \fBovsdb\-server\fR. .TP \fB\-\-sync\-from=\fIserver\fR Causes \fBovsdb\-server\fR to synchronize its databases with the databases in -\fIserver\fR. Every transaction committed by \fIserver\fR will be replicated +\fIserver\fR. Every transaction committed by \fIserver\fR will be replicated to \fBovsdb\-server\fR. \fIserver\fR is an active connection method in one of -the forms documented in \fBovsdb\-client(1). +the forms documented in \fBovsdb\-client(1)\fR. +However if the \fB\-\-no\-sync\fR is also specified, replication will be +postponed until \fBovs-appctl(1)\fR command +\fBovsdb\-server/connect\-active\-ovsdb\-server\fR is issued. +.TP +\fB\-\-sync\-exclude-tables=\fIdb:table[,db:table]...\fR +Causes the specified tables to be excluded from replication. +.TP +\fB\-\-no\-sync\fR +Always start the server as an active server. Use this option to allow +the syncing options to be specified using command line options, yet start +the server, as the default, active server. diff --git a/tests/ovsdb-server.at b/tests/ovsdb-server.at index d2f3089..e7daefd 100644 --- a/tests/ovsdb-server.at +++ b/tests/ovsdb-server.at @@ -1100,22 +1100,22 @@ AT_CHECK([ovs-appctl -t ovsdb-server ovsdb-server/get-active-ovsdb-server], ]) AT_CLEANUP -#ovsdb-server/get-sync-excluded-tables command -AT_SETUP([ovsdb-server/get-sync-excluded-tables]) -AT_KEYWORDS([ovsdb server replication get-excluded-tables]) +#ovsdb-server/get-sync-exclude-tables command +AT_SETUP([ovsdb-server/get-sync-exclude-tables]) +AT_KEYWORDS([ovsdb server replication get-exclude-tables]) ordinal_schema > schema AT_CHECK([ovsdb-tool create db schema], [0], [ignore], [ignore]) on_exit 'kill `cat *.pid`' AT_CHECK([ovsdb-server --detach --no-chdir --pidfile --sync-exclude-tables=mydb:db1,mydb:db2 db]) -AT_CHECK([ovs-appctl -t ovsdb-server ovsdb-server/get-sync-excluded-tables], +AT_CHECK([ovs-appctl -t ovsdb-server ovsdb-server/get-sync-exclude-tables], [0], [mydb:db1,mydb:db2 ]) AT_CLEANUP -#ovsdb-server/set-sync-excluded-tables command -AT_SETUP([ovsdb-server/set-sync-excluded-tables]) -AT_KEYWORDS([ovsdb server replication set-excluded-tables]) +#ovsdb-server/set-sync-exclude-tables command +AT_SETUP([ovsdb-server/set-sync-exclude-tables]) +AT_KEYWORDS([ovsdb server replication set-exclude-tables]) replication_schema > schema AT_CHECK([ovsdb-tool create db1 schema], [0], [stdout], [ignore]) AT_CHECK([ovsdb-tool create db2 schema], [0], [stdout], [ignore]) @@ -1126,7 +1126,7 @@ on_exit 'test ! -e pid || kill `cat pid`' AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server2.log --pidfile="`pwd`"/pid2 --remote=punix:db2.sock --unixctl="`pwd`"/unixctl2 --sync-from=unix:db.sock db2], [0], [ignore], [ignore]) on_exit 'test ! -e pid2 || kill `cat pid2`' -AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/set-sync-excluded-tables mydb:b], [0], [ignore], [ignore], [test ! -e pid || kill `cat pid`; test ! -e pid2 || kill `cat pid2`]) +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/set-sync-exclude-tables mydb:b], [0], [ignore], [ignore], [test ! -e pid || kill `cat pid`; test ! -e pid2 || kill `cat pid2`]) AT_CHECK([ovsdb-client transact unix:db.sock \ '[["mydb", @@ -1174,6 +1174,11 @@ on_exit 'test ! -e pid || kill `cat pid`' AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server2.log --pidfile="`pwd`"/pid2 --remote=punix:db2.sock --unixctl="`pwd`"/unixctl2 db2], [0], [ignore], [ignore]) on_exit 'test ! -e pid2 || kill `cat pid2`' +dnl Try to connect without specifying the active server. +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/connect-active-ovsdb-server], [0], +[Unable to connect: active server is not specified. +], [ignore]) + AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/set-active-ovsdb-server unix:db.sock], [0], [stdout], [ignore]) AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/connect-active-ovsdb-server], [0], [stdout], [ignore]) @@ -1278,3 +1283,87 @@ _uuid name number OVSDB_SERVER_SHUTDOWN OVSDB_SERVER_SHUTDOWN2 AT_CLEANUP + +#ovsdb-server/active-backup-role-switching +AT_SETUP([ovsdb-server/active-backup-role-switching]) +AT_KEYWORDS([ovsdb server replication active-backup-switching]) +replication_schema > schema +AT_CHECK([ovsdb-tool create db1 schema], [0], [stdout], [ignore]) +AT_CHECK([ovsdb-tool create db2 schema], [0], [stdout], [ignore]) + +dnl Add some data to both DBs +AT_CHECK([ovsdb-tool transact db1 \ +'[["mydb", + {"op": "insert", + "table": "a", + "row": {"number": 9, "name": "nine"}}]]'], [0], [ignore], [ignore]) + +AT_CHECK([ovsdb-tool transact db2 \ +'[["mydb", + {"op": "insert", + "table": "a", + "row": {"number": 9, "name": "nine"}}]]'], [0], [ignore], [ignore]) + +dnl Start both 'db1' and 'db2' in backup mode. Let them backup from each +dnl other. This is not an supported operation state, but to simulate a start +dnl up condition where an HA manger can select which one to be an active +dnl server soon after. +AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server1.log --pidfile="`pwd`"/pid --remote=punix:db.sock --unixctl="`pwd`"/unixctl db1 --sync-from=unix:db2.sock --no-sync ], [0], [ignore], [ignore]) +on_exit 'test ! -e pid || kill `cat pid`' + +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/connect-active-ovsdb-server]) + +AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server2.log --pidfile="`pwd`"/pid2 --remote=punix:db2.sock --unixctl="`pwd`"/unixctl2 --sync-from=unix:db.sock db2], [0], [ignore], [ignore]) +on_exit 'test ! -e pid2 || kill `cat pid2`' + +dnl +dnl make sure both servers reached the replication state +OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status |grep replicating]) +OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/sync-status |grep replicating]) + +dnl Switch the 'db1' to active +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/disconnect-active-ovsdb-server]) +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status], [0], [state: active +]) + +dnl Issue a transaction to 'db1' +AT_CHECK([ovsdb-client transact unix:db.sock \ +'[["mydb", + {"op": "insert", + "table": "a", + "row": {"number": 0, "name": "zero"}}]]'], [0], [ignore]) + +dnl It should be replicated to 'db2' +OVS_WAIT_UNTIL([ovsdb-client dump unix:db2.sock | grep zero]) + +dnl Flip the role of 'db1' and 'db2'. 'db1' becomes backup, and db2 becomes active +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/disconnect-active-ovsdb-server]) +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/connect-active-ovsdb-server]) + +dnl Verify the change happend +OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status |grep replicating]) +AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/sync-status], [0], [state: active +]) + +dnl Issue an transaction to 'db2' which is now active. +AT_CHECK([ovsdb-client transact unix:db2.sock \ +'[["mydb", + {"op": "insert", + "table": "b", + "row": {"number": 1, "name": "one"}}]]'], [0], [ignore]) + +dnl The transaction should be replicated to 'db1' +OVS_WAIT_UNTIL([ovsdb-client dump unix:db.sock | grep one]) + +dnl Both servers should have the same content. +AT_CHECK([ovsdb-client dump unix:db.sock], [0], [stdout]) +cat stdout > dump1 + +AT_CHECK([ovsdb-client dump unix:db2.sock], [0], [stdout]) +cat stdout > dump2 + +AT_CHECK([diff dump1 dump2]) + +dnl OVSDB_SERVER_SHUTDOWN +dnl OVSDB_SERVER_SHUTDOWN2 +AT_CLEANUP