From patchwork Tue Sep 8 20:10:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Han Zhou X-Patchwork-Id: 1360026 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=ovn.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4BmGVk6Z5tz9sTK for ; Wed, 9 Sep 2020 06:10:54 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 77AE2873A1; Tue, 8 Sep 2020 20:10:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hiid1N-R9qhe; Tue, 8 Sep 2020 20:10:49 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by hemlock.osuosl.org (Postfix) with ESMTP id CA085872FF; Tue, 8 Sep 2020 20:10:49 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id AD8ACC0859; Tue, 8 Sep 2020 20:10:49 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 22DA5C0051 for ; Tue, 8 Sep 2020 20:10:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0ACC086D03 for ; Tue, 8 Sep 2020 20:10:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WV1ItV-MrU4A for ; Tue, 8 Sep 2020 20:10:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 7C8988562A for ; Tue, 8 Sep 2020 20:10:46 +0000 (UTC) X-Originating-IP: 73.241.94.255 Received: from localhost.localdomain.localdomain (unknown [73.241.94.255]) (Authenticated sender: hzhou@ovn.org) by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 22D0660003; Tue, 8 Sep 2020 20:10:41 +0000 (UTC) From: Han Zhou To: dev@openvswitch.org Date: Tue, 8 Sep 2020 13:10:35 -0700 Message-Id: <1599595835-124391-1-git-send-email-hzhou@ovn.org> X-Mailer: git-send-email 2.1.0 Cc: Han Zhou Subject: [ovs-dev] [PATCH ovn] ovn-controller: Unix command debug/delay-nb-cfg-report. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This command is added to avoid the flooding of nb_cfg updates from a large nubmer of hypervisors interfering with the original SB DB data distribution and handling, so that the e2e control plane latency can be measured more accurately during scale testing with the command ovn-nbctl --print-wait-time --wait=hv ... Without this, there can be hypervisors updating back to SB before the SB server finishes sending notifications to the rest of hypervisors. In my test with 3K HVs, delaying the report for 2 seconds is enough to get accurate latency report. Signed-off-by: Han Zhou Acked-by: Numan Siddique --- This patch is a revision of the last patch in a previous patch series to address comments from Numan. The other patches in that series have been merged, so this patch is sent as a new patch. controller/ovn-controller.8.xml | 9 +++++++++ controller/ovn-controller.c | 30 ++++++++++++++++++++++++++++++ tests/ovn-controller.at | 30 ++++++++++++++++++++++++++++++ 3 files changed, 69 insertions(+) diff --git a/controller/ovn-controller.8.xml b/controller/ovn-controller.8.xml index 6687731..16bc47b 100644 --- a/controller/ovn-controller.8.xml +++ b/controller/ovn-controller.8.xml @@ -507,6 +507,15 @@ local index so that it can interact with the southbound database again.

+ +
debug/delay-nb-cfg-report seconds
+
+ This command is used to delay ovn-controller updating the + nb_cfg back to OVN_Southbound database. This + is useful when ovn-nbctl --wait=hv is used to measure + end-to-end latency in a large scale environment. See + ovn-nbctl(8) for more details. +

diff --git a/controller/ovn-controller.c b/controller/ovn-controller.c index 674a094..e2e9095 100644 --- a/controller/ovn-controller.c +++ b/controller/ovn-controller.c @@ -76,6 +76,7 @@ static unixctl_cb_func cluster_state_reset_cmd; static unixctl_cb_func debug_pause_execution; static unixctl_cb_func debug_resume_execution; static unixctl_cb_func debug_status_execution; +static unixctl_cb_func debug_delay_nb_cfg_report; #define DEFAULT_BRIDGE_NAME "br-int" #define DEFAULT_PROBE_INTERVAL_MSEC 5000 @@ -2360,6 +2361,10 @@ main(int argc, char *argv[]) unixctl_command_register("debug/status", "", 0, 0, debug_status_execution, &paused); + unsigned int delay_nb_cfg_report = 0; + unixctl_command_register("debug/delay-nb-cfg-report", "SECONDS", 1, 1, + debug_delay_nb_cfg_report, &delay_nb_cfg_report); + unsigned int ovs_cond_seqno = UINT_MAX; unsigned int ovnsb_cond_seqno = UINT_MAX; @@ -2571,6 +2576,10 @@ main(int argc, char *argv[]) sbrec_chassis_private_set_nb_cfg(chassis_private, cur_cfg); sbrec_chassis_private_set_nb_cfg_timestamp( chassis_private, time_wall_msec()); + if (delay_nb_cfg_report) { + VLOG_INFO("Sleep for %u sec", delay_nb_cfg_report); + xsleep(delay_nb_cfg_report); + } } } @@ -2927,3 +2936,24 @@ debug_status_execution(struct unixctl_conn *conn, int argc OVS_UNUSED, unixctl_command_reply(conn, "running"); } } + +static void +debug_delay_nb_cfg_report(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[], void *delay_) +{ + unsigned int *delay = delay_; + + if (!str_to_uint(argv[1], 10, delay)) { + unixctl_command_reply_error(conn, "unsigned integer required"); + return; + } + + char *msg; + if (*delay) { + msg = xasprintf("delay nb_cfg report for %u seconds.", *delay); + unixctl_command_reply(conn, msg); + free(msg); + } else { + unixctl_command_reply(conn, "no delay for nb_cfg report."); + } +} diff --git a/tests/ovn-controller.at b/tests/ovn-controller.at index 812946b..7ed5216 100644 --- a/tests/ovn-controller.at +++ b/tests/ovn-controller.at @@ -382,3 +382,33 @@ OVS_WAIT_UNTIL([test x0 = x`ovn-sbctl --columns nb_cfg --bare find chassis`]) OVN_CLEANUP([hv]) AT_CLEANUP + +# Test unix command: debug/delay-nb-cfg-report +AT_SETUP([ovn-controller - debug/delay-nb-cfg-report]) +AT_KEYWORDS([ovn]) +ovn_start + +net_add n1 +sim_add hv +as hv +ovs-vsctl add-br br-phys +ovn_attach n1 br-phys 192.168.0.1 + +OVS_WAIT_UNTIL([test xhv = x`ovn-sbctl --columns name --bare find chassis`]) + +AT_CHECK([ovn-appctl -t ovn-controller debug/delay-nb-cfg-report 2], [0], + [delay nb_cfg report for 2 seconds. +]) + +AT_FAIL_IF([ovn-nbctl --timeout=1 --wait=hv sync]) + +AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync]) + +AT_CHECK([ovn-appctl -t ovn-controller debug/delay-nb-cfg-report 0], [0], + [no delay for nb_cfg report. +]) + +AT_CHECK([ovn-nbctl --timeout=1 --wait=hv sync]) + +OVN_CLEANUP([hv]) +AT_CLEANUP