From patchwork Fri Feb 15 20:26:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1043171 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="R/GXTKBq"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 441PvX152dz9sDr for ; Sat, 16 Feb 2019 07:27:35 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 0A165C9F; Fri, 15 Feb 2019 20:26:18 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 7504ABAC for ; Fri, 15 Feb 2019 20:26:15 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id BD2667FD for ; Fri, 15 Feb 2019 20:26:14 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id s1so5467525plp.9 for ; Fri, 15 Feb 2019 12:26:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uCX/tA+uaYQC25gJQ9/kD/BYQrjXh5P5eId6P2Jx/wc=; b=R/GXTKBqs+I08RtsPQDOsIylPW74IrwJHzicGFz/wb4/S5mTRI1tPcU1RhCloCRik1 IhUzF98Yd/OGAMqhtMYcJeH8p/WCjlZNAKcV5y0AvMlW5FMvT8Pmc0V1y3auZcaXdpOb xRdchdOF0lxPT70il139NrSnSk6RK1Uny5JdRZhgZOzZMA6Z/hOtSeOQxpS4fNmXOR3D 0wH2Xh9KkIO3d94fBsRwzjvWwPW+DyrnASISVOJYiij+61TfnIBLz3gcPPytghrigyop Wu08R1RSuqttfIMTI3tjyOxRySgGDxQ5f9iACI4kJd/HH2WL+ooypBmNHT23bfdcIqbr lIyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uCX/tA+uaYQC25gJQ9/kD/BYQrjXh5P5eId6P2Jx/wc=; b=SizXMcHuOBLc0lPSKtkXgv1Ha5Q5t8kBT9udmbUaRmusbhLlzWMj4UGgm+jgNA83uM pzuuZaMRaRlFLi52vhco5fMY9QJ3PPYI+4FQM26ajpGvS8xn7cthmCuq+Fp7qXij0ax6 ctsSeK8TuuKhnXtpQTwLDZ/AC9FHja9i0hppn5zkAu0kFctZZYxEvQeRqvHQtnoy1jIF jVzetZTeSK8fCWh4r/eH9ClpBdpEELbVJBg3mqMvo6+KjLMaSLfwzp3QmAAsgoHTq5gY lhlGtZq9nSrP4nOxgyT/Jp6Eu81p8pUoPmkuCdV4ByISakLHJdyEZhJo79QIAvqsS5a1 zMqQ== X-Gm-Message-State: AHQUAubLcGh8e7L0CoIegi+5LBBAJidyLc+XMcgt1daVQRKzyM4pkV1F 4ZLRTPMKkWd0WKwqunGVpcav1IEI X-Google-Smtp-Source: AHgI3IZeRM4E5i8BFQrqOBL0aiRNiaGBJQ+nPH75vFnziKSYoYOVrE+scPwPbjjJ+AV+LbNFlBagpw== X-Received: by 2002:a17:902:b60a:: with SMTP id b10mr11409371pls.303.1550262374090; Fri, 15 Feb 2019 12:26:14 -0800 (PST) Received: from localhost.localdomain.localdomain ([216.113.160.77]) by smtp.gmail.com with ESMTPSA id b12sm10359164pfb.30.2019.02.15.12.26.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Feb 2019 12:26:13 -0800 (PST) From: Han Zhou X-Google-Original-From: Han Zhou To: dev@openvswitch.org Date: Fri, 15 Feb 2019 12:26:00 -0800 Message-Id: <1550262363-86117-5-git-send-email-hzhou8@ebay.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1550262363-86117-1-git-send-email-hzhou8@ebay.com> References: <1550262363-86117-1-git-send-email-hzhou8@ebay.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2 4/7] ovsdb-server: Transaction history tracking. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Han Zhou Maintaining last N (n = 100) transactions in memory, which will be used for future patches for generating monitor data from any point in this N transactions. Signed-off-by: Han Zhou --- ovsdb/ovsdb-server.c | 11 +++++ ovsdb/ovsdb.c | 3 ++ ovsdb/ovsdb.h | 10 +++++ ovsdb/transaction.c | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++- ovsdb/transaction.h | 3 ++ 5 files changed, 137 insertions(+), 1 deletion(-) diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c index 65a47a4..321381e 100644 --- a/ovsdb/ovsdb-server.c +++ b/ovsdb/ovsdb-server.c @@ -219,6 +219,7 @@ main_loop(struct server_config *config, struct shash_node *next; SHASH_FOR_EACH_SAFE (node, next, all_dbs) { struct db *db = node->data; + ovsdb_txn_history_run(db->db); if (ovsdb_trigger_run(db->db, time_msec())) { /* The message below is currently the only reason to disconnect * all clients. */ @@ -568,6 +569,7 @@ parse_txn(struct server_config *config, struct db *db, error = ovsdb_file_txn_from_json(db->db, txn_json, false, &txn); if (!error) { + ovsdb_txn_set_txnid(txnid, txn); log_and_free_error(ovsdb_txn_replay_commit(txn)); } if (!error && !uuid_is_zero(txnid)) { @@ -658,6 +660,13 @@ open_db(struct server_config *config, const char *filename) db->db = ovsdb_create(schema, storage); ovsdb_jsonrpc_server_add_db(config->jsonrpc, db->db); + /* Enable txn history for clustered mode. It is not enabled for other mode + * for now, since txn id is available for clustered mode only. */ + if (ovsdb_storage_is_clustered(storage)) { + db->db->need_txn_history = true; + db->db->n_txn_history = 0; + ovs_list_init(&db->db->txn_history); + } read_db(config, db); error = (db->db->name[0] == '_' @@ -695,6 +704,8 @@ add_server_db(struct server_config *config) json_destroy(schema_json); struct db *db = xzalloc(sizeof *db); + /* We don't need txn_history for server_db. */ + db->filename = xstrdup(""); db->db = ovsdb_create(schema, ovsdb_storage_create_unbacked()); bool ok OVS_UNUSED = ovsdb_jsonrpc_server_add_db(config->jsonrpc, db->db); diff --git a/ovsdb/ovsdb.c b/ovsdb/ovsdb.c index da8db82..ea7dd23 100644 --- a/ovsdb/ovsdb.c +++ b/ovsdb/ovsdb.c @@ -455,6 +455,9 @@ ovsdb_destroy(struct ovsdb *db) /* Remove all the monitors. */ ovsdb_monitors_remove(db); + /* Destroy txn history. */ + ovsdb_txn_history_destroy(db); + /* The caller must ensure that no triggers remain. */ ovs_assert(ovs_list_is_empty(&db->triggers)); diff --git a/ovsdb/ovsdb.h b/ovsdb/ovsdb.h index d96b1c2..32e5333 100644 --- a/ovsdb/ovsdb.h +++ b/ovsdb/ovsdb.h @@ -67,6 +67,11 @@ bool ovsdb_parse_version(const char *, struct ovsdb_version *); bool ovsdb_is_valid_version(const char *); /* Database. */ +struct ovsdb_txn_history_node { + struct ovs_list node; /* Element in struct ovsdb's txn_history list */ + struct ovsdb_txn *txn; +}; + struct ovsdb { char *name; struct ovsdb_schema *schema; @@ -80,6 +85,11 @@ struct ovsdb { bool run_triggers; struct ovsdb_table *rbac_role; + + /* History trasanctions for incremental monitor transfer. */ + bool need_txn_history; /* Need to maintain history of transactions. */ + unsigned int n_txn_history; /* Current number of history transactions. */ + struct ovs_list txn_history; /* Contains "struct ovsdb_txn_history_node. */ }; struct ovsdb *ovsdb_create(struct ovsdb_schema *, struct ovsdb_storage *); diff --git a/ovsdb/transaction.c b/ovsdb/transaction.c index 5a43132..3485383 100644 --- a/ovsdb/transaction.c +++ b/ovsdb/transaction.c @@ -40,6 +40,7 @@ struct ovsdb_txn { struct ovsdb *db; struct ovs_list txn_tables; /* Contains "struct ovsdb_txn_table"s. */ struct ds comment; + struct uuid txnid; /* For clustered mode only. It is the eid. */ }; /* A table modified by a transaction. */ @@ -106,13 +107,19 @@ static unsigned int serial; struct ovsdb_txn * ovsdb_txn_create(struct ovsdb *db) { - struct ovsdb_txn *txn = xmalloc(sizeof *txn); + struct ovsdb_txn *txn = xzalloc(sizeof *txn); txn->db = db; ovs_list_init(&txn->txn_tables); ds_init(&txn->comment); return txn; } +void +ovsdb_txn_set_txnid(const struct uuid *txnid, struct ovsdb_txn *txn) +{ + txn->txnid = *txnid; +} + static void ovsdb_txn_free(struct ovsdb_txn *txn) { @@ -881,11 +888,79 @@ ovsdb_txn_precommit(struct ovsdb_txn *txn) return error; } +static struct ovsdb_txn* +ovsdb_txn_clone(const struct ovsdb_txn *txn) +{ + struct ovsdb_txn *txn_cloned = xzalloc(sizeof *txn_cloned); + ovs_list_init(&txn_cloned->txn_tables); + txn_cloned->txnid = txn->txnid; + + struct ovsdb_txn_table *t; + LIST_FOR_EACH (t, node, &txn->txn_tables) { + struct ovsdb_txn_table *t_cloned = xmalloc(sizeof *t_cloned); + ovs_list_push_back(&txn_cloned->txn_tables, &t_cloned->node); + hmap_init(&t_cloned->txn_rows); + + struct ovsdb_txn_row *r; + HMAP_FOR_EACH (r, hmap_node, &t->txn_rows) { + size_t n_columns = shash_count(&t->table->schema->columns); + struct ovsdb_txn_row *r_cloned = + xzalloc(offsetof(struct ovsdb_txn_row, changed) + + bitmap_n_bytes(n_columns)); + + r_cloned->uuid = r->uuid; + r_cloned->table = r->table; + r_cloned->old = r->old ? ovsdb_row_clone(r->old) : NULL; + r_cloned->new = r->new ? ovsdb_row_clone(r->new) : NULL; + memcpy(&r_cloned->changed, &r->changed, bitmap_n_bytes(n_columns)); + hmap_insert(&t_cloned->txn_rows, &r_cloned->hmap_node, + uuid_hash(&r_cloned->uuid)); + } + } + return txn_cloned; +} + +static void +ovsdb_txn_destroy_cloned(struct ovsdb_txn *txn) +{ + ovs_assert(!txn->db); + struct ovsdb_txn_table *t, *next_txn_table; + LIST_FOR_EACH_SAFE (t, next_txn_table, node, &txn->txn_tables) { + struct ovsdb_txn_row *r, *next_txn_row; + HMAP_FOR_EACH_SAFE (r, next_txn_row, hmap_node, &t->txn_rows) { + if (r->old) { + ovsdb_row_destroy(r->old); + } + if (r->new) { + ovsdb_row_destroy(r->new); + } + hmap_remove(&t->txn_rows, &r->hmap_node); + free(r); + } + hmap_destroy(&t->txn_rows); + ovs_list_remove(&t->node); + free(t); + } + free(txn); +} + +static void +ovsdb_txn_add_to_history(struct ovsdb_txn *txn) +{ + if (txn->db->need_txn_history) { + struct ovsdb_txn_history_node *node = xzalloc(sizeof *node); + node->txn = ovsdb_txn_clone(txn); + ovs_list_push_back(&txn->db->txn_history, &node->node); + txn->db->n_txn_history++; + } +} + /* Finalize commit. */ void ovsdb_txn_complete(struct ovsdb_txn *txn) { if (!ovsdb_txn_is_empty(txn)) { + txn->db->run_triggers = true; ovsdb_monitors_commit(txn->db, txn); ovsdb_error_assert(for_each_txn_row(txn, ovsdb_txn_update_weak_refs)); @@ -906,6 +981,7 @@ ovsdb_txn_replay_commit(struct ovsdb_txn *txn) if (error) { ovsdb_txn_abort(txn); } else { + ovsdb_txn_add_to_history(txn); ovsdb_txn_complete(txn); } return error; @@ -1304,3 +1380,36 @@ for_each_txn_row(struct ovsdb_txn *txn, return NULL; } + +void +ovsdb_txn_history_run(struct ovsdb *db) +{ + if (!db->need_txn_history) { + return; + } + /* Remove old histories to limit the size of the history */ + while (db->n_txn_history > 100) { + struct ovsdb_txn_history_node *txn_h_node = CONTAINER_OF( + ovs_list_pop_front(&db->txn_history), + struct ovsdb_txn_history_node, node); + + ovsdb_txn_destroy_cloned(txn_h_node->txn); + free(txn_h_node); + db->n_txn_history--; + } +} + +void +ovsdb_txn_history_destroy(struct ovsdb *db) +{ + + if (!db->need_txn_history) { + return; + } + + struct ovsdb_txn_history_node *txn_h_node, *next; + LIST_FOR_EACH_SAFE (txn_h_node, next, node, &db->txn_history) { + ovsdb_txn_destroy_cloned(txn_h_node->txn); + free(txn_h_node); + } +} diff --git a/ovsdb/transaction.h b/ovsdb/transaction.h index 32384fc..c601d47 100644 --- a/ovsdb/transaction.h +++ b/ovsdb/transaction.h @@ -25,6 +25,7 @@ struct ovsdb_table; struct uuid; struct ovsdb_txn *ovsdb_txn_create(struct ovsdb *); +void ovsdb_txn_set_txnid(const struct uuid *, struct ovsdb_txn *); void ovsdb_txn_abort(struct ovsdb_txn *); struct ovsdb_error *ovsdb_txn_replay_commit(struct ovsdb_txn *) @@ -59,5 +60,7 @@ void ovsdb_txn_for_each_change(const struct ovsdb_txn *, void ovsdb_txn_add_comment(struct ovsdb_txn *, const char *); const char *ovsdb_txn_get_comment(const struct ovsdb_txn *); +void ovsdb_txn_history_run(struct ovsdb *); +void ovsdb_txn_history_destroy(struct ovsdb *); #endif /* ovsdb/transaction.h */