From patchwork Wed Nov 1 17:46:43 2017
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: "Chandran, Sugesh"
X-Patchwork-Id: 833062
Return-Path:
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
spf=pass (mailfrom) smtp.mailfrom=openvswitch.org
(client-ip=140.211.169.12; helo=mail.linuxfoundation.org;
envelope-from=ovs-dev-bounces@openvswitch.org;
receiver=)
Received: from mail.linuxfoundation.org (mail.linuxfoundation.org
[140.211.169.12])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256
bits)) (No client certificate requested)
by ozlabs.org (Postfix) with ESMTPS id 3yRwdw5lTLz9ryT
for ;
Thu, 2 Nov 2017 04:47:16 +1100 (AEDT)
Received: from mail.linux-foundation.org (localhost [127.0.0.1])
by mail.linuxfoundation.org (Postfix) with ESMTP id AA09CCC5;
Wed, 1 Nov 2017 17:46:48 +0000 (UTC)
X-Original-To: dev@openvswitch.org
Delivered-To: ovs-dev@mail.linuxfoundation.org
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id C47FECBB
for ; Wed, 1 Nov 2017 17:46:45 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 2C1A179
for ; Wed, 1 Nov 2017 17:46:45 +0000 (UTC)
Received: from orsmga005.jf.intel.com ([10.7.209.41])
by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;
01 Nov 2017 10:46:44 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.44,329,1505804400"; d="scan'208";a="167946579"
Received: from silpixa00389820.ir.intel.com ([10.237.222.155])
by orsmga005.jf.intel.com with ESMTP; 01 Nov 2017 10:46:44 -0700
From: Sugesh Chandran
To: dev@openvswitch.org
Date: Wed, 1 Nov 2017 17:46:43 +0000
Message-Id: <1509558404-130908-2-git-send-email-sugesh.chandran@intel.com>
X-Mailer: git-send-email 2.7.4
In-Reply-To: <1509558404-130908-1-git-send-email-sugesh.chandran@intel.com>
References: <1509558404-130908-1-git-send-email-sugesh.chandran@intel.com>
X-Spam-Status: No, score=-5.0 required=5.0 tests=RCVD_IN_DNSWL_HI,
RP_MATCHES_RCVD autolearn=disabled version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Subject: [ovs-dev] [RFC PATCH 1/2] Adding DPDK configuration option to
isolate rte-mempool allocation.
X-BeenThere: ovs-dev@openvswitch.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id:
List-Unsubscribe: ,
List-Archive:
List-Post:
List-Help:
List-Subscribe: ,
MIME-Version: 1.0
Sender: ovs-dev-bounces@openvswitch.org
Errors-To: ovs-dev-bounces@openvswitch.org
DPDK allocate memory regions at the time of vswitchd init. To run multiple
primary instance of DPDK including OVS on a single platform, the memory map
regions in the filesystem should be distinct.
The new configuration option let user to enable the memory isolation in need.
By default, OVS uses default dpdk memory regions.
To isolate the memory regions, DPDK prefix the memory map files with user
input string. This implementation uses the pid of vswitchd process as a memory
map prefix, because its unique in the platform.
For eg: a vswitchd process with pid '12345' create memory map regions with
prefix 'ovs-12345' in the filesystem.
The following configuration option is used to enable the feature and changing
this value requires restarting the daemon.
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-isolate-mem=true
Signed-off-by: Sugesh Chandran
---
lib/daemon.c | 9 +++++++++
lib/daemon.h | 1 +
lib/dpdk.c | 34 ++++++++++++++++++++++++++++++++++
vswitchd/vswitch.xml | 23 +++++++++++++++++++++++
4 files changed, 67 insertions(+)
diff --git a/lib/daemon.c b/lib/daemon.c
index 3249c5a..3aa61e1 100644
--- a/lib/daemon.c
+++ b/lib/daemon.c
@@ -66,6 +66,15 @@ set_pidfile(const char *name)
pidfile = make_pidfile_name(name);
}
+/*
+ * Return the name of pidfile, NULL when its none.
+ */
+const char *
+get_pidfile(void)
+{
+ return pidfile;
+}
+
/* Disables self confinement. */
void
daemon_disable_self_confinement(void)
diff --git a/lib/daemon.h b/lib/daemon.h
index bfa0640..1df156e 100644
--- a/lib/daemon.h
+++ b/lib/daemon.h
@@ -157,5 +157,6 @@ void service_stop(void);
bool should_service_stop(void);
void set_pidfile(const char *name);
void close_standard_fds(void);
+const char *get_pidfile(void);
#endif /* daemon.h */
diff --git a/lib/dpdk.c b/lib/dpdk.c
index 8da6c32..cbbf28d 100644
--- a/lib/dpdk.c
+++ b/lib/dpdk.c
@@ -35,6 +35,7 @@
#include "openvswitch/dynamic-string.h"
#include "openvswitch/vlog.h"
#include "smap.h"
+#include "daemon.h"
VLOG_DEFINE_THIS_MODULE(dpdk);
@@ -302,6 +303,38 @@ static cookie_io_functions_t dpdk_log_func = {
.write = dpdk_log_write,
};
+/*
+ * Isolate the dpdk rte_memory pool for the vswitch process.
+ * The isolation is achieved by using a specific prefix for rte memory maps.
+ * the prefix is created from the pid of the vswitchd. This allows to run
+ * multiple ovs-dpdk instance on same platform in need.
+ */
+static void
+dpdk_isolate_rte_mem_config(const struct smap *ovs_other_config,
+ char ***argv, int *argc)
+{
+ if (smap_get_bool(ovs_other_config, "dpdk-isolate-mem", false)) {
+ /*
+ * pid file is present only on Linux distribution, so DPDK?
+ */
+ pid_t pid;
+ const char *pidfile;
+ pidfile = get_pidfile();
+ if (!pidfile) {
+ VLOG_INFO("Error : Cannot find ovs-vswitchd pidfile");
+ return;
+ }
+ pid = read_pidfile(pidfile);
+ if (pid < 0) {
+ VLOG_INFO("Error : Invalid pid in the ovs-vswitchd pidfile");
+ return;
+ }
+ *argv = grow_argv(argv, *argc, 2);
+ (*argv)[(*argc)++] = xstrdup("--file-prefix");
+ (*argv)[(*argc)++] = xasprintf("%s-%X", "ovs", (int)pid);
+ }
+}
+
static void
dpdk_init__(const struct smap *ovs_other_config)
{
@@ -388,6 +421,7 @@ dpdk_init__(const struct smap *ovs_other_config)
}
}
+ dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc);
argv = grow_argv(&argv, argc, 1);
argv[argc] = NULL;
diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml
index d7f6839..3a507ea 100644
--- a/vswitchd/vswitch.xml
+++ b/vswitchd/vswitch.xml
@@ -419,6 +419,29 @@
VLAN.
+
+
+
+ Set this value to true
to isolate DPDK rte mempool at
+ the time of allocation. The option is valid only when DPDK is enabled
+ in OVS.
+
+
+ The default value is false
. Changing this value requires
+ restarting the daemon.
+
+
+ The rte_mempool map is created at the time of vswitchd init and its
+ been used by all DPDK processes including vswitchd in the platform.
+ dpdk-isolate-mem
option let vswitchd to use a separate
+ memory region than other processes for isolation. The separate
+ memory region map are prefixed by ovs-pid
+ in the filesystem. pid is the process id of vswitchd
+ process.
+
+
+
From patchwork Wed Nov 1 17:46:44 2017
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: "Chandran, Sugesh"
X-Patchwork-Id: 833063
Return-Path:
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@bilbo.ozlabs.org
Authentication-Results: ozlabs.org;
spf=pass (mailfrom) smtp.mailfrom=openvswitch.org
(client-ip=140.211.169.12; helo=mail.linuxfoundation.org;
envelope-from=ovs-dev-bounces@openvswitch.org;
receiver=)
Received: from mail.linuxfoundation.org (mail.linuxfoundation.org
[140.211.169.12])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256
bits)) (No client certificate requested)
by ozlabs.org (Postfix) with ESMTPS id 3yRwfY1Bbsz9sRW
for ;
Thu, 2 Nov 2017 04:47:49 +1100 (AEDT)
Received: from mail.linux-foundation.org (localhost [127.0.0.1])
by mail.linuxfoundation.org (Postfix) with ESMTP id 91914CC7;
Wed, 1 Nov 2017 17:46:49 +0000 (UTC)
X-Original-To: dev@openvswitch.org
Delivered-To: ovs-dev@mail.linuxfoundation.org
Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org
[172.17.192.35])
by mail.linuxfoundation.org (Postfix) with ESMTPS id 99D98CB9
for ; Wed, 1 Nov 2017 17:46:46 +0000 (UTC)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6
Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 46BEE79
for ; Wed, 1 Nov 2017 17:46:46 +0000 (UTC)
Received: from orsmga005.jf.intel.com ([10.7.209.41])
by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;
01 Nov 2017 10:46:45 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.44,329,1505804400"; d="scan'208";a="167946585"
Received: from silpixa00389820.ir.intel.com ([10.237.222.155])
by orsmga005.jf.intel.com with ESMTP; 01 Nov 2017 10:46:45 -0700
From: Sugesh Chandran
To: dev@openvswitch.org
Date: Wed, 1 Nov 2017 17:46:44 +0000
Message-Id: <1509558404-130908-3-git-send-email-sugesh.chandran@intel.com>
X-Mailer: git-send-email 2.7.4
In-Reply-To: <1509558404-130908-1-git-send-email-sugesh.chandran@intel.com>
References: <1509558404-130908-1-git-send-email-sugesh.chandran@intel.com>
X-Spam-Status: No, score=-5.0 required=5.0 tests=RCVD_IN_DNSWL_HI,
RP_MATCHES_RCVD autolearn=disabled version=3.3.1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
smtp1.linux-foundation.org
Subject: [ovs-dev] [RFC PATCH 2/2] Adding configuration option to whitelist
DPDK physical ports.
X-BeenThere: ovs-dev@openvswitch.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id:
List-Unsubscribe: ,
List-Archive:
List-Post:
List-Help:
List-Subscribe: ,
MIME-Version: 1.0
Sender: ovs-dev-bounces@openvswitch.org
Errors-To: ovs-dev-bounces@openvswitch.org
Adding a OVS configuration option to whitelist DPDK physical ports. By default
running multiple instances of DPDK on a single platform cannot use physical
ports at the same time even though they are distinct.
The eal init scans all the ports that are bound to DPDK and initialize the
drivers accordingly. This happens for every DPDK process init.
On a multi instance deployment usecase, it causes issues for using physical
NIC ports.
Consider a two DPDK process that are running on a single platform,
the second DPDK primary process will try to initialize the drivers for all the
physical ports even though it may be used in first DPDK process.
To avoid this situation user can whitelist the ports for each DPDK application.
Whitelisting of ports/PCI-ID in a DPDK process will limit the eal-init only on
those ports.
To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1", the
configuration option in OVS would be
ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist-pci-ids="0000:06:00.0,0000:06:00.1"
To update the whitelist ports, OVS daemon has to be restarted.
Signed-off-by: Sugesh Chandran
---
lib/dpdk.c | 29 +++++++++++++++++++++++++++++
vswitchd/vswitch.xml | 21 +++++++++++++++++++++
2 files changed, 50 insertions(+)
diff --git a/lib/dpdk.c b/lib/dpdk.c
index cbbf28d..ea7400f 100644
--- a/lib/dpdk.c
+++ b/lib/dpdk.c
@@ -336,6 +336,34 @@ dpdk_isolate_rte_mem_config(const struct smap *ovs_other_config,
}
static void
+dpdk_whitelist_pci_ids(const struct smap *ovs_other_config, char ***argv,
+ int *argc)
+{
+ const char *pci_ids;
+ char *pci_dev;
+ int len;
+ int i;
+ pci_ids = smap_get(ovs_other_config, "dpdk-whitelist-pci-ids");
+ if (!pci_ids) {
+ return;
+ }
+ len = strlen(pci_ids);
+ do {
+ i = strcspn(pci_ids, ",");
+ pci_dev = xmemdup0(pci_ids, i);
+ if (!strlen(pci_dev)) {
+ break;
+ }
+ *argv = grow_argv(argv, *argc, 2);
+ (*argv)[(*argc)++] = xstrdup("-w");
+ (*argv)[(*argc)++] = pci_dev;
+ i++;
+ pci_ids += i;
+ len -= i;
+ } while (pci_ids && len > 0);
+}
+
+static void
dpdk_init__(const struct smap *ovs_other_config)
{
char **argv = NULL, **argv_to_release = NULL;
@@ -422,6 +450,7 @@ dpdk_init__(const struct smap *ovs_other_config)
}
dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc);
+ dpdk_whitelist_pci_ids(ovs_other_config, &argv, &argc);
argv = grow_argv(&argv, argc, 1);
argv[argc] = NULL;
diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml
index 3a507ea..04baf97 100644
--- a/vswitchd/vswitch.xml
+++ b/vswitchd/vswitch.xml
@@ -442,6 +442,27 @@
+
+
+ Specifies list of pci-ids separated by , for whitelisting available
+ physical NIC ports in OVS. The option valid only when DPDK is enabled
+ in OVS and the ports are already bound to DPDK userspace driver.
+
+
+ By default, all the DPDK bound ports are initialized at the time of
+ vswitchd start. Whitelisting a list of pci-ids is used to limit the
+ initialization only to specific ports. Changing this value requires
+ restarting the daemon.
+
+
+ This option allow user to run multiple instance of DPDK including
+ vswitchd simultaneously by exclusively allocating the physical NIC
+ ports between them. Its impossible to use the NIC ports in
+ multiple primary DPDK processes without whitelisting them
+ even the application is using distinct ports.
+
+
+