From patchwork Fri Feb 15 20:26:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1043174 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="F9lM/fds"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 441PxQ249Jz9sDr for ; Sat, 16 Feb 2019 07:29:14 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 38AD2CDD; Fri, 15 Feb 2019 20:26:21 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id F0332BA2 for ; Fri, 15 Feb 2019 20:26:17 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 86BE37E9 for ; Fri, 15 Feb 2019 20:26:17 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id y1so5299496pgk.11 for ; Fri, 15 Feb 2019 12:26:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zYd+KyQrDMg1fq7WlA7SFG6MWEJMWtY9zXDB7DeE6sM=; b=F9lM/fdsb8/GS+FMcGoRzswGwnTcLPxPwpGKPxYmZCqKvzq1KBsNaRBw8n0+WCID26 HG7kuYkrfdYqEqxAQzZNMR4LaSvn3moY6lG9yiRAaFqlTo/7yiH6ImLsn8M5EF20Y3vC yyIiC9m38dUBRJwrjglwjhqnHGcSNt1s84nfzgo+M2rmX6/z4Ic950E5SkR6PHwkP1p6 Sn6CnQCdpvBhvxO6+Nq403TRtfHQQ1HU0TEEvnWPxHFXKrwhPoACmahrBXl+SxKnWITs 74YTjJWChKcaQcwjY8uVyl9YERoP9RE+q9eTxNeMZM8x9kbfY6cBV3ER9qcOFuuoSdTQ Iy5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zYd+KyQrDMg1fq7WlA7SFG6MWEJMWtY9zXDB7DeE6sM=; b=hTeN0R0EEve/9nPqkSiS89hryDk6NVRJJZE4r+CV7+m2teF47rWZjclhJsQVpU5tFB 3d+/lsmjyhmQe3G39ZcH6sid45FIsE/go3Mu9QJb0ABXCP3dyS/0IBLXS/qUxtc0qeoC FcIbJHMqnhhthgKHtnY4HvM2qNPqLjCQTZemWSrUBN/LVgCdi27P6UruKqqhg+5Ot1Wl ADJjsrEqUorcCOt+R94fyCe/2WvFWs96or4LatEVnAGlEMzxW74PbZ7WjEImQJKaBPXU Fnq0Qkd1h0nMAUfmEMvRyPaXFU0HQ3d/UYzlIxFvXhvR2tJUJNVilQ5g1gE0HxQKa5PC bZbA== X-Gm-Message-State: AHQUAuaOHsdOwE4z6StJkXTLulqqSw4iUjNUcGHCpg85Jy/X622v+y25 cbyupNuOhzQPbmMHO2NBmaKl7BlT X-Google-Smtp-Source: AHgI3IYu2maTYvMsQp8/OvqhuOSrydn4KsSlf5VK3+B2niw0mCv7Xo+iuzTAfwLEwK1a8e/pQsPKvA== X-Received: by 2002:a63:9402:: with SMTP id m2mr6839215pge.93.1550262376899; Fri, 15 Feb 2019 12:26:16 -0800 (PST) Received: from localhost.localdomain.localdomain ([216.113.160.77]) by smtp.gmail.com with ESMTPSA id b12sm10359164pfb.30.2019.02.15.12.26.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Feb 2019 12:26:15 -0800 (PST) From: Han Zhou X-Google-Original-From: Han Zhou To: dev@openvswitch.org Date: Fri, 15 Feb 2019 12:26:03 -0800 Message-Id: <1550262363-86117-8-git-send-email-hzhou8@ebay.com> X-Mailer: git-send-email 2.1.0 In-Reply-To: <1550262363-86117-1-git-send-email-hzhou8@ebay.com> References: <1550262363-86117-1-git-send-email-hzhou8@ebay.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH v2 7/7] ovsdb-idl.c: Fast resync from server when connection reset. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Han Zhou Use monitor_cond_since to request changes after last version of local data when connection to server is reset, without clearing the local data. It falls back to clearing and repopulating all the data when the requested id cannot be fulfilled by the server. Test result at ovn-scale-test environment using clustered mode: - 1K HVs (ovsdb clients) - 10K lports Without the patch it took 30+ min for the SB ovsdb-server to calm down and HVs to stablize the connectin and finish syncing data. With the patch there were no noticible CPU spike of SB ovsdb-server, and all HVs were in sync with SB within 1 min, which is the probe interval set in this test (so it took at most 1 min for HVs to notice the TCP connection reset and reconnect and resync finished immediately after that). Reported-at: https://mail.openvswitch.org/pipermail/ovs-discuss/2018-September/047457.html Signed-off-by: Han Zhou --- lib/ovsdb-idl.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/lib/ovsdb-idl.c b/lib/ovsdb-idl.c index 5d86046..8cfb201 100644 --- a/lib/ovsdb-idl.c +++ b/lib/ovsdb-idl.c @@ -216,6 +216,9 @@ struct ovsdb_idl_db { bool has_lock; /* Has db server told us we have the lock? */ bool is_lock_contended; /* Has db server told us we can't get lock? */ struct json *lock_request_id; /* JSON-RPC ID of in-flight lock request. */ + + /* Last db txn id, used for fast resync through monitor_cond_since */ + struct uuid last_id; }; static void ovsdb_idl_db_track_clear(struct ovsdb_idl_db *); @@ -2070,9 +2073,8 @@ ovsdb_idl_send_monitor_request(struct ovsdb_idl *idl, struct ovsdb_idl_db *db, break; case OVSDB_IDL_MM_MONITOR_COND_SINCE: method = "monitor_cond_since"; - struct uuid last_id = UUID_ZERO; struct json *json_last_id = json_string_create_nocopy( - xasprintf(UUID_FMT, UUID_ARGS(&last_id))); + xasprintf(UUID_FMT, UUID_ARGS(&db->last_id))); json_array_add(params, json_last_id); break; default: @@ -2100,6 +2102,7 @@ ovsdb_idl_db_parse_monitor_reply(struct ovsdb_idl_db *db, { db->change_seqno++; const struct json *table_updates = result; + bool clear_db = true; if (method == OVSDB_IDL_MM_MONITOR_COND_SINCE) { if (result->type != JSON_ARRAY || result->array.n != 3) { struct ovsdb_error *error = ovsdb_syntax_error(result, NULL, @@ -2109,10 +2112,25 @@ ovsdb_idl_db_parse_monitor_reply(struct ovsdb_idl_db *db, return; } - table_updates = result->array.elems[2]; + bool found = json_boolean(result->array.elems[0]); + if (found) { + clear_db = false; + } + const char *last_id = json_string(result->array.elems[1]); + if (!uuid_from_string(&db->last_id, last_id)) { + struct ovsdb_error *error = ovsdb_syntax_error(result, NULL, + "Last-id %s is not in UUID format.", + last_id); + log_parse_update_error(error); + return; + } + + table_updates = result->array.elems[2]; + } + if (clear_db) { + ovsdb_idl_db_clear(db); } - ovsdb_idl_db_clear(db); ovsdb_idl_db_parse_update(db, table_updates, method); } @@ -2155,6 +2173,14 @@ ovsdb_idl_db_parse_update_rpc(struct ovsdb_idl_db *db, struct json *table_updates = params->array.elems[1]; if (!strcmp(msg->method, "update3")) { table_updates = params->array.elems[2]; + const char *last_id = json_string(params->array.elems[1]); + if (!uuid_from_string(&db->last_id, last_id)) { + struct ovsdb_error *error = ovsdb_syntax_error(params, NULL, + "Last-id %s is not in UUID format.", + last_id); + log_parse_update_error(error); + return false; + } } ovsdb_idl_db_parse_update(db, table_updates, mm); return true;