From patchwork Fri Aug 30 01:18:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: aginwala aginwala X-Patchwork-Id: 1155643 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QuqnixfC"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46KM8y1rw5z9sBp for ; Fri, 30 Aug 2019 11:20:01 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id C2F2E5464; Fri, 30 Aug 2019 01:19:57 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 4D79046CF for ; Fri, 30 Aug 2019 01:18:32 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 483B1F1 for ; Fri, 30 Aug 2019 01:18:31 +0000 (UTC) Received: by mail-wr1-f44.google.com with SMTP id s18so5287400wrn.1 for ; Thu, 29 Aug 2019 18:18:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=xuTe28i2XrayPfy3scjzes6izSpOfw/b1sMV0QCnVUY=; b=QuqnixfCOV/9e6jfho5KFppT8hoaODt6iFKjPiDTRfO6Cc9AYXLlkP9BT7KFuqOCvK +qmshM+65zjAOGKnSaGDL+kWxvKMSnfq3kLqZFAdQqKGZCkkTj+LQJx70bbgUYih5jzZ d5HSiKjIzh3A5lhxKfk2XVfVGf8O0jpqxNuFYpjxo6CTgvBDqXm0nb7C8QSJCB7lb71I bOXXbyLHd2JTheA4XdzOxNZhUQTfihi6Zkd9Esby2yDlWoECwut8I3tPUqXheaGFVZ+B B74nDZyDB7pdI7VJ3Rp0UWH8Vn0/YboxRkJS2KclIgMOmAHCqVCcmaz9SZklFJe8E0zh NLCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=xuTe28i2XrayPfy3scjzes6izSpOfw/b1sMV0QCnVUY=; b=jq4TqTi4I369nLyt96FOmqG4ULO8LYFD70iaxvbzzr6Tq6nMCHSO7t8sFAYXLMBLE9 q5QWeuCyyMHmwLp9T746qBm3Ojp/1BCKLEwNU6EgHqmILz8wAc+yZjuwLFH53IZSParh rX2sDdbqQvvTZvauap6AYb9k8RM7mwnb4JClBrjk/ER0uqJc2z2WandvvrTE/xaZ9y1I AQWVXQKncj1sHlMSxzfyZ8yPGtRnRvbzfLQsxIDJuUAqduPisv+JKJC45xghwTKzZBOm YqUnuqzRSWOHDZ9AFtaRUnR0GEi4gVi90PS6JhtVmkTHZZoaIgz5sR0YFQ+QaZrWuE1m nJzg== X-Gm-Message-State: APjAAAUBvD2+7owXSVNIJFX24RArWiBgCAtxur0LnO3mfeS3Q+Sieq1L vVXYUfw86kaM/WpTw9C6tvxfEFFe X-Google-Smtp-Source: APXvYqzITYGgMGT9SUsegIaNPlA//As4oeZkqpXNNfnTLOGElAAwpqKaiJxVlo8XxdJu17CD9XfuHg== X-Received: by 2002:a5d:4b41:: with SMTP id w1mr13980841wrs.23.1567127909428; Thu, 29 Aug 2019 18:18:29 -0700 (PDT) Received: from LM-SJC-11015761.corp.ebay.com ([216.113.160.77]) by smtp.gmail.com with ESMTPSA id f10sm2875804wrw.16.2019.08.29.18.18.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 29 Aug 2019 18:18:28 -0700 (PDT) From: amginwal@gmail.com To: dev@openvswitch.org Date: Thu, 29 Aug 2019 18:18:24 -0700 Message-Id: <20190830011824.71334-1-amginwal@gmail.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) MIME-Version: 1.0 X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: Aliasgar Ginwala Subject: [ovs-dev] [PATCH v9] ovsdb-tool: Convert clustered db to standalone db. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Aliasgar Ginwala Add support in ovsdb-tool for migrating clustered dbs to standalone dbs. E.g. usage to migrate nb/sb db to standalone db from raft: ovsdb-tool cluster-to-standalone ovnnb_db.db ovnnb_db_cluster.db Signed-off-by: Aliasgar Ginwala Acked-by: Han Zhou --- Documentation/ref/ovsdb.7.rst | 3 + NEWS | 3 + ovsdb/ovsdb-tool.1.in | 8 +++ ovsdb/ovsdb-tool.c | 101 +++++++++++++++++++++++++++++++++- tests/ovsdb-tool.at | 43 +++++++++++++++ 5 files changed, 157 insertions(+), 1 deletion(-) diff --git a/Documentation/ref/ovsdb.7.rst b/Documentation/ref/ovsdb.7.rst index cd1c63d64..b12d8066c 100644 --- a/Documentation/ref/ovsdb.7.rst +++ b/Documentation/ref/ovsdb.7.rst @@ -514,6 +514,9 @@ standalone database from the contents of a running clustered database. When the cluster is down and cannot be revived, ``ovsdb-client backup`` will not work. +Use ``ovsdb-tool cluster-to-standalone`` to convert clustered database to +standalone database when the cluster is down and cannot be revived. + Upgrading or Downgrading a Database ----------------------------------- diff --git a/NEWS b/NEWS index c5caa13d6..a02f9f1a6 100644 --- a/NEWS +++ b/NEWS @@ -49,6 +49,9 @@ v2.12.0 - xx xxx xxxx quickly after a brief disconnection, saving bandwidth and CPU time. See section 4.1.15 of ovsdb-server(7) for details of related OVSDB protocol extension. + * Support to convert from cluster database to standalone database is now + available when clustered is down and cannot be revived using ovsdb-tool + . Check "Database Migration Commands" in ovsdb-tool man section. - OVN: * IPAM/MACAM: - select IPAM mac_prefix in a random manner if not provided by the user diff --git a/ovsdb/ovsdb-tool.1.in b/ovsdb/ovsdb-tool.1.in index ec85e14c4..31a918d90 100644 --- a/ovsdb/ovsdb-tool.1.in +++ b/ovsdb/ovsdb-tool.1.in @@ -147,6 +147,14 @@ avoid this possibility, specify \fB\-\-cid=\fIuuid\fR, where \fIuuid\fR is the cluster ID of the cluster to join, as printed by \fBovsdb\-tool get\-cid\fR. . +.SS "Database Migration Commands" +This commands will convert cluster database to standalone database. +. +.IP "\fBcluster\-to\-standalone\fI db clusterdb" +Use this command to convert to standalone database from clustered database +when the cluster is down and cannot be revived. It creates new standalone +\fIdb\fR file from the given cluster \fIdb\fR file. +. .SS "Version Management Commands" .so ovsdb/ovsdb-schemas.man .PP diff --git a/ovsdb/ovsdb-tool.c b/ovsdb/ovsdb-tool.c index 438f97590..3bbf4c8bc 100644 --- a/ovsdb/ovsdb-tool.c +++ b/ovsdb/ovsdb-tool.c @@ -173,6 +173,9 @@ usage(void) " compare-versions A OP B compare OVSDB schema version numbers\n" " query [DB] TRNS execute read-only transaction on DB\n" " transact [DB] TRNS execute read/write transaction on DB\n" + " cluster-to-standalone DB DB Convert clustered DB to\n" + " standalone DB when cluster is down and cannot be\n" + " revived\n" " [-m]... show-log [DB] print DB's log entries\n" "The default DB is %s.\n" "The default SCHEMA is %s.\n", @@ -942,6 +945,55 @@ print_raft_record(const struct raft_record *r, } } +static void +raft_header_to_standalone_log(const struct raft_header *h, + struct ovsdb_log *db_log_data) +{ + if (h->snap_index) { + if (!h->snap.data || json_array(h->snap.data)->n != 2) { + ovs_fatal(0, "Incorrect raft header data array length"); + } + + struct json *schema_json = json_array(h->snap.data)->elems[0]; + if (schema_json->type != JSON_NULL) { + struct ovsdb_schema *schema; + check_ovsdb_error(ovsdb_schema_from_json(schema_json, &schema)); + ovsdb_schema_destroy(schema); + check_ovsdb_error(ovsdb_log_write_and_free(db_log_data, + schema_json)); + } + + struct json *data_json = json_array(h->snap.data)->elems[1]; + if (!data_json || data_json->type != JSON_OBJECT) { + ovs_fatal(0, "Invalid raft header data"); + } + if (data_json->type != JSON_NULL) { + check_ovsdb_error(ovsdb_log_write_and_free(db_log_data, + data_json)); + } + } +} + +static void +raft_record_to_standalone_log(const struct raft_record *r, + struct ovsdb_log *db_log_data) +{ + if (r->type == RAFT_REC_ENTRY) { + if (!r->entry.data) { + return; + } + if (json_array(r->entry.data)->n != 2) { + ovs_fatal(0, "Incorrect raft record array length"); + } + + struct json *data_json = json_array(r->entry.data)->elems[1]; + if (data_json->type != JSON_NULL) { + check_ovsdb_error(ovsdb_log_write_and_free(db_log_data, + data_json)); + } + } +} + static void do_show_log_cluster(struct ovsdb_log *log) { @@ -1511,6 +1563,51 @@ do_compare_versions(struct ovs_cmdl_context *ctx) exit(result ? 0 : 2); } +static void +do_convert_to_standalone(struct ovsdb_log *log, struct ovsdb_log *db_log_data) +{ + for (unsigned int i = 0; ; i++) { + struct json *json; + check_ovsdb_error(ovsdb_log_read(log, &json)); + if (!json) { + break; + } + + if (i == 0) { + struct raft_header h; + check_ovsdb_error(raft_header_from_json(&h, json)); + raft_header_to_standalone_log(&h, db_log_data); + raft_header_uninit(&h); + } else { + struct raft_record r; + check_ovsdb_error(raft_record_from_json(&r, json)); + raft_record_to_standalone_log(&r, db_log_data); + raft_record_uninit(&r); + } + } +} + +static void +do_cluster_standalone(struct ovs_cmdl_context *ctx) +{ + const char *db_file_name = ctx->argv[1]; + const char *cluster_db_file_name = ctx->argv[2]; + struct ovsdb_log *log; + struct ovsdb_log *db_log_data; + + check_ovsdb_error(ovsdb_log_open(cluster_db_file_name, + OVSDB_MAGIC"|"RAFT_MAGIC, + OVSDB_LOG_READ_ONLY, -1, &log)); + check_ovsdb_error(ovsdb_log_open(db_file_name, OVSDB_MAGIC, + OVSDB_LOG_CREATE_EXCL, -1, &db_log_data)); + if (strcmp(ovsdb_log_get_magic(log), RAFT_MAGIC) != 0) { + ovs_fatal(0, "Database is not clustered db.\n"); + } + do_convert_to_standalone(log, db_log_data); + check_ovsdb_error(ovsdb_log_commit_block(db_log_data)); + ovsdb_log_close(db_log_data); + ovsdb_log_close(log); +} static void do_help(struct ovs_cmdl_context *ctx OVS_UNUSED) @@ -1550,7 +1647,9 @@ static const struct ovs_cmdl_command all_commands[] = { { "compare-versions", "a op b", 3, 3, do_compare_versions, OVS_RO }, { "help", NULL, 0, INT_MAX, do_help, OVS_RO }, { "list-commands", NULL, 0, INT_MAX, do_list_commands, OVS_RO }, - { NULL, NULL, 0, 0, NULL, OVS_RO }, + { "cluster-to-standalone", "db clusterdb", 2, 2, + do_cluster_standalone, OVS_RW }, + { NULL, NULL, 2, 2, NULL, OVS_RO }, }; static const struct ovs_cmdl_command *get_all_commands(void) diff --git a/tests/ovsdb-tool.at b/tests/ovsdb-tool.at index 69c5d6afa..953b1ed8f 100644 --- a/tests/ovsdb-tool.at +++ b/tests/ovsdb-tool.at @@ -459,3 +459,46 @@ OVS_APP_EXIT_AND_WAIT([ovsdb-server]) # Make sure that the clustered data matched the standalone data. AT_CHECK([cat dump2], [0], [expout]) AT_CLEANUP + +AT_SETUP([ovsdb-tool convert-to-standalone]) +AT_KEYWORDS([ovsdb file positive]) +ordinal_schema > schema +AT_CHECK([ovsdb-tool create-cluster db schema unix:s1.raft], [0], [stdout], [ignore]) +AT_CHECK([ovsdb-server --detach --no-chdir --pidfile --remote=punix:socket --log-file db >/dev/null 2>&1]) +for txn in m4_foreach([txn], [[[["ordinals", + {"op": "insert", + "table": "ordinals", + "row": {"number": 0, "name": "zero"}}, + {"op": "insert", + "table": "ordinals", + "row": {"number": 1, "name": "one"}}, + {"op": "insert", + "table": "ordinals", + "row": {"number": 2, "name": "two"}}]]]], ['txn' ]); do +AT_CHECK([ovsdb-client transact unix:socket "$txn"], [0], [ignore], [ignore]) +done +AT_CHECK([ovsdb-client transact unix:socket '[["ordinals"]]'], [0], + [ignore], [ignore]) +AT_CHECK([ovsdb-client dump unix:socket > clusterdump]) +AT_CHECK([ovs-appctl -t ovsdb-server -e exit], [0], [ignore], [ignore]) + +# Convert to standalone database from clustered database. +AT_CHECK(ovsdb-tool cluster-to-standalone db1 db) + +# Check its standalone db +AT_CHECK([ovsdb-tool db-is-standalone db1]) + +# Dump the standalone db data. +AT_CHECK([ovsdb-server -vconsole:off -vfile -vvlog:off --detach --no-chdir --pidfile --log-file --remote=punix:db.sock db1]) +AT_CHECK([ovsdb_client_wait ordinals connected]) +AT_CHECK([ovsdb-client dump > standalonedump]) +OVS_APP_EXIT_AND_WAIT([ovsdb-server]) + +# Make sure both standalone and cluster db data matches. +AT_CHECK([grep "zero" clusterdump], [0], [ignore]) +AT_CHECK([grep "zero" standalonedump], [0], [ignore]) +AT_CHECK([grep "one" clusterdump], [0], [ignore]) +AT_CHECK([grep "one" standalonedump], [0], [ignore]) +AT_CHECK([grep "two" clusterdump], [0], [ignore]) +AT_CHECK([grep "two" standalonedump], [0], [ignore]) +AT_CLEANUP