From patchwork Tue Sep 8 20:10:35 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Han Zhou
X-Patchwork-Id: 1360026
Return-Path:
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org
(client-ip=140.211.166.133; helo=hemlock.osuosl.org;
envelope-from=ovs-dev-bounces@openvswitch.org; receiver=)
Authentication-Results: ozlabs.org;
dmarc=none (p=none dis=none) header.from=ovn.org
Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by ozlabs.org (Postfix) with ESMTPS id 4BmGVk6Z5tz9sTK
for ; Wed, 9 Sep 2020 06:10:54 +1000 (AEST)
Received: from localhost (localhost [127.0.0.1])
by hemlock.osuosl.org (Postfix) with ESMTP id 77AE2873A1;
Tue, 8 Sep 2020 20:10:52 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
Received: from hemlock.osuosl.org ([127.0.0.1])
by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id Hiid1N-R9qhe; Tue, 8 Sep 2020 20:10:49 +0000 (UTC)
Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56])
by hemlock.osuosl.org (Postfix) with ESMTP id CA085872FF;
Tue, 8 Sep 2020 20:10:49 +0000 (UTC)
Received: from lf-lists.osuosl.org (localhost [127.0.0.1])
by lists.linuxfoundation.org (Postfix) with ESMTP id AD8ACC0859;
Tue, 8 Sep 2020 20:10:49 +0000 (UTC)
X-Original-To: dev@openvswitch.org
Delivered-To: ovs-dev@lists.linuxfoundation.org
Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137])
by lists.linuxfoundation.org (Postfix) with ESMTP id 22DA5C0051
for ; Tue, 8 Sep 2020 20:10:48 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
by fraxinus.osuosl.org (Postfix) with ESMTP id 0ACC086D03
for ; Tue, 8 Sep 2020 20:10:48 +0000 (UTC)
X-Virus-Scanned: amavisd-new at osuosl.org
Received: from fraxinus.osuosl.org ([127.0.0.1])
by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id WV1ItV-MrU4A for ;
Tue, 8 Sep 2020 20:10:46 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net
[217.70.183.195])
by fraxinus.osuosl.org (Postfix) with ESMTPS id 7C8988562A
for ; Tue, 8 Sep 2020 20:10:46 +0000 (UTC)
X-Originating-IP: 73.241.94.255
Received: from localhost.localdomain.localdomain (unknown [73.241.94.255])
(Authenticated sender: hzhou@ovn.org)
by relay3-d.mail.gandi.net (Postfix) with ESMTPSA id 22D0660003;
Tue, 8 Sep 2020 20:10:41 +0000 (UTC)
From: Han Zhou
To: dev@openvswitch.org
Date: Tue, 8 Sep 2020 13:10:35 -0700
Message-Id: <1599595835-124391-1-git-send-email-hzhou@ovn.org>
X-Mailer: git-send-email 2.1.0
Cc: Han Zhou
Subject: [ovs-dev] [PATCH ovn] ovn-controller: Unix command
debug/delay-nb-cfg-report.
X-BeenThere: ovs-dev@openvswitch.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id:
List-Unsubscribe: ,
List-Archive:
List-Post:
List-Help:
List-Subscribe: ,
MIME-Version: 1.0
Errors-To: ovs-dev-bounces@openvswitch.org
Sender: "dev"
This command is added to avoid the flooding of nb_cfg updates from
a large nubmer of hypervisors interfering with the original SB DB
data distribution and handling, so that the e2e control plane latency
can be measured more accurately during scale testing with the command
ovn-nbctl --print-wait-time --wait=hv ...
Without this, there can be hypervisors updating back to SB before the
SB server finishes sending notifications to the rest of hypervisors.
In my test with 3K HVs, delaying the report for 2 seconds is enough to
get accurate latency report.
Signed-off-by: Han Zhou
Acked-by: Numan Siddique
---
This patch is a revision of the last patch in a previous patch series to
address comments from Numan. The other patches in that series have been
merged, so this patch is sent as a new patch.
controller/ovn-controller.8.xml | 9 +++++++++
controller/ovn-controller.c | 30 ++++++++++++++++++++++++++++++
tests/ovn-controller.at | 30 ++++++++++++++++++++++++++++++
3 files changed, 69 insertions(+)
diff --git a/controller/ovn-controller.8.xml b/controller/ovn-controller.8.xml
index 6687731..16bc47b 100644
--- a/controller/ovn-controller.8.xml
+++ b/controller/ovn-controller.8.xml
@@ -507,6 +507,15 @@
local index so that it can interact with the southbound database again.
+
+ debug/delay-nb-cfg-report
seconds
+
+ This command is used to delay ovn-controller updating the
+ nb_cfg
back to OVN_Southbound
database. This
+ is useful when ovn-nbctl --wait=hv
is used to measure
+ end-to-end latency in a large scale environment. See
+ ovn-nbctl
(8) for more details.
+
diff --git a/controller/ovn-controller.c b/controller/ovn-controller.c
index 674a094..e2e9095 100644
--- a/controller/ovn-controller.c
+++ b/controller/ovn-controller.c
@@ -76,6 +76,7 @@ static unixctl_cb_func cluster_state_reset_cmd;
static unixctl_cb_func debug_pause_execution;
static unixctl_cb_func debug_resume_execution;
static unixctl_cb_func debug_status_execution;
+static unixctl_cb_func debug_delay_nb_cfg_report;
#define DEFAULT_BRIDGE_NAME "br-int"
#define DEFAULT_PROBE_INTERVAL_MSEC 5000
@@ -2360,6 +2361,10 @@ main(int argc, char *argv[])
unixctl_command_register("debug/status", "", 0, 0, debug_status_execution,
&paused);
+ unsigned int delay_nb_cfg_report = 0;
+ unixctl_command_register("debug/delay-nb-cfg-report", "SECONDS", 1, 1,
+ debug_delay_nb_cfg_report, &delay_nb_cfg_report);
+
unsigned int ovs_cond_seqno = UINT_MAX;
unsigned int ovnsb_cond_seqno = UINT_MAX;
@@ -2571,6 +2576,10 @@ main(int argc, char *argv[])
sbrec_chassis_private_set_nb_cfg(chassis_private, cur_cfg);
sbrec_chassis_private_set_nb_cfg_timestamp(
chassis_private, time_wall_msec());
+ if (delay_nb_cfg_report) {
+ VLOG_INFO("Sleep for %u sec", delay_nb_cfg_report);
+ xsleep(delay_nb_cfg_report);
+ }
}
}
@@ -2927,3 +2936,24 @@ debug_status_execution(struct unixctl_conn *conn, int argc OVS_UNUSED,
unixctl_command_reply(conn, "running");
}
}
+
+static void
+debug_delay_nb_cfg_report(struct unixctl_conn *conn, int argc OVS_UNUSED,
+ const char *argv[], void *delay_)
+{
+ unsigned int *delay = delay_;
+
+ if (!str_to_uint(argv[1], 10, delay)) {
+ unixctl_command_reply_error(conn, "unsigned integer required");
+ return;
+ }
+
+ char *msg;
+ if (*delay) {
+ msg = xasprintf("delay nb_cfg report for %u seconds.", *delay);
+ unixctl_command_reply(conn, msg);
+ free(msg);
+ } else {
+ unixctl_command_reply(conn, "no delay for nb_cfg report.");
+ }
+}
diff --git a/tests/ovn-controller.at b/tests/ovn-controller.at
index 812946b..7ed5216 100644
--- a/tests/ovn-controller.at
+++ b/tests/ovn-controller.at
@@ -382,3 +382,33 @@ OVS_WAIT_UNTIL([test x0 = x`ovn-sbctl --columns nb_cfg --bare find chassis`])
OVN_CLEANUP([hv])
AT_CLEANUP
+
+# Test unix command: debug/delay-nb-cfg-report
+AT_SETUP([ovn-controller - debug/delay-nb-cfg-report])
+AT_KEYWORDS([ovn])
+ovn_start
+
+net_add n1
+sim_add hv
+as hv
+ovs-vsctl add-br br-phys
+ovn_attach n1 br-phys 192.168.0.1
+
+OVS_WAIT_UNTIL([test xhv = x`ovn-sbctl --columns name --bare find chassis`])
+
+AT_CHECK([ovn-appctl -t ovn-controller debug/delay-nb-cfg-report 2], [0],
+ [delay nb_cfg report for 2 seconds.
+])
+
+AT_FAIL_IF([ovn-nbctl --timeout=1 --wait=hv sync])
+
+AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync])
+
+AT_CHECK([ovn-appctl -t ovn-controller debug/delay-nb-cfg-report 0], [0],
+ [no delay for nb_cfg report.
+])
+
+AT_CHECK([ovn-nbctl --timeout=1 --wait=hv sync])
+
+OVN_CLEANUP([hv])
+AT_CLEANUP