[ovs-dev,RFC,1/2] Adding DPDK configuration option to isolate rte-mempool allocation.

Message ID 1509558404-130908-2-git-send-email-sugesh.chandran@intel.com
State New
Headers show
Series
  • Adding ovs configuration options to run multiple DPDK instances on a single platform.
Related show

Commit Message

Sugesh Chandran Nov. 1, 2017, 5:46 p.m.
DPDK allocate memory regions at the time of vswitchd init. To run multiple
primary instance of DPDK including OVS on a single platform, the memory map
regions in the filesystem should be distinct.

The new configuration option let user to enable the memory isolation in need.
By default, OVS uses default dpdk memory regions.

To isolate the memory regions, DPDK prefix the memory map files with user
input string. This implementation uses the pid of vswitchd process as a memory
map prefix, because its unique in the platform.

For eg: a vswitchd process with pid '12345' create memory map regions with
prefix 'ovs-12345' in the filesystem.

The following configuration option is used to enable the feature and changing
this value requires restarting the daemon.

 ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-isolate-mem=true

Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com>
---
 lib/daemon.c         |  9 +++++++++
 lib/daemon.h         |  1 +
 lib/dpdk.c           | 34 ++++++++++++++++++++++++++++++++++
 vswitchd/vswitch.xml | 23 +++++++++++++++++++++++
 4 files changed, 67 insertions(+)

Comments

Ben Pfaff Nov. 3, 2017, 5:54 p.m. | #1
On Wed, Nov 01, 2017 at 05:46:43PM +0000, Sugesh Chandran wrote:
> DPDK allocate memory regions at the time of vswitchd init. To run multiple
> primary instance of DPDK including OVS on a single platform, the memory map
> regions in the filesystem should be distinct.
> 
> The new configuration option let user to enable the memory isolation in need.
> By default, OVS uses default dpdk memory regions.
> 
> To isolate the memory regions, DPDK prefix the memory map files with user
> input string. This implementation uses the pid of vswitchd process as a memory
> map prefix, because its unique in the platform.
> 
> For eg: a vswitchd process with pid '12345' create memory map regions with
> prefix 'ovs-12345' in the filesystem.
> 
> The following configuration option is used to enable the feature and changing
> this value requires restarting the daemon.
> 
>  ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-isolate-mem=true
> 
> Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com>

I believe that this patch adds code to ovs-vswitchd to read
ovs-vswitchd's own pidfile.  I don't understand this.  Wouldn't it make
more sense to just call getpid()?

Thanks,

Ben.
Sugesh Chandran Nov. 5, 2017, 10:14 p.m. | #2
Hi Ben,
Thank you for looking into the patch.

Regards
_Sugesh

> -----Original Message-----
> From: Ben Pfaff [mailto:blp@ovn.org]
> Sent: Friday, November 3, 2017 5:55 PM
> To: Chandran, Sugesh <sugesh.chandran@intel.com>
> Cc: dev@openvswitch.org
> Subject: Re: [ovs-dev] [RFC PATCH 1/2] Adding DPDK configuration option to
> isolate rte-mempool allocation.
> 
.....
> > The following configuration option is used to enable the feature and
> > changing this value requires restarting the daemon.
> >
> >  ovs-vsctl --no-wait set Open_vSwitch .
> > other_config:dpdk-isolate-mem=true
> >
> > Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com>
> 
> I believe that this patch adds code to ovs-vswitchd to read ovs-vswitchd's own
> pidfile.  I don't understand this.  Wouldn't it make more sense to just call
> getpid()?
[Sugesh]  Yes, It make more sense than reading from the file, will change to use the 'getpid()' when sending the proper patch.
> 
> Thanks,
> 
> Ben.

Patch

diff --git a/lib/daemon.c b/lib/daemon.c
index 3249c5a..3aa61e1 100644
--- a/lib/daemon.c
+++ b/lib/daemon.c
@@ -66,6 +66,15 @@  set_pidfile(const char *name)
     pidfile = make_pidfile_name(name);
 }
 
+/*
+ * Return the name of pidfile, NULL when its none.
+ */
+const char *
+get_pidfile(void)
+{
+    return pidfile;
+}
+
 /* Disables self confinement. */
 void
 daemon_disable_self_confinement(void)
diff --git a/lib/daemon.h b/lib/daemon.h
index bfa0640..1df156e 100644
--- a/lib/daemon.h
+++ b/lib/daemon.h
@@ -157,5 +157,6 @@  void service_stop(void);
 bool should_service_stop(void);
 void set_pidfile(const char *name);
 void close_standard_fds(void);
+const char *get_pidfile(void);
 
 #endif /* daemon.h */
diff --git a/lib/dpdk.c b/lib/dpdk.c
index 8da6c32..cbbf28d 100644
--- a/lib/dpdk.c
+++ b/lib/dpdk.c
@@ -35,6 +35,7 @@ 
 #include "openvswitch/dynamic-string.h"
 #include "openvswitch/vlog.h"
 #include "smap.h"
+#include "daemon.h"
 
 VLOG_DEFINE_THIS_MODULE(dpdk);
 
@@ -302,6 +303,38 @@  static cookie_io_functions_t dpdk_log_func = {
     .write = dpdk_log_write,
 };
 
+/*
+ * Isolate the dpdk rte_memory pool for the vswitch process.
+ * The isolation is achieved by using a specific prefix for rte memory maps.
+ * the prefix is created from the pid of the vswitchd. This allows to run
+ * multiple ovs-dpdk instance on same platform in need.
+ */
+static void
+dpdk_isolate_rte_mem_config(const struct smap *ovs_other_config,
+                            char ***argv, int *argc)
+{
+    if (smap_get_bool(ovs_other_config, "dpdk-isolate-mem", false)) {
+        /*
+         * pid file is present only on Linux distribution, so DPDK?
+         */
+        pid_t pid;
+        const char *pidfile;
+        pidfile = get_pidfile();
+        if (!pidfile) {
+            VLOG_INFO("Error : Cannot find ovs-vswitchd pidfile");
+            return;
+        }
+        pid = read_pidfile(pidfile);
+        if (pid < 0) {
+            VLOG_INFO("Error : Invalid pid in the ovs-vswitchd pidfile");
+            return;
+        }
+        *argv = grow_argv(argv, *argc, 2);
+        (*argv)[(*argc)++] = xstrdup("--file-prefix");
+        (*argv)[(*argc)++] = xasprintf("%s-%X", "ovs", (int)pid);
+    }
+}
+
 static void
 dpdk_init__(const struct smap *ovs_other_config)
 {
@@ -388,6 +421,7 @@  dpdk_init__(const struct smap *ovs_other_config)
         }
     }
 
+    dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc);
     argv = grow_argv(&argv, argc, 1);
     argv[argc] = NULL;
 
diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml
index d7f6839..3a507ea 100644
--- a/vswitchd/vswitch.xml
+++ b/vswitchd/vswitch.xml
@@ -419,6 +419,29 @@ 
           VLAN.
         </p>
       </column>
+
+      <column name="other_config" key="dpdk-isolate-mem"
+              type='{"type": "boolean"}'>
+        <p>
+          Set this value to <code>true</code> to isolate DPDK rte mempool at
+          the time of allocation. The option is valid only when DPDK is enabled
+          in OVS.
+        </p>
+        <p>
+          The default value is <code>false</code>. Changing this value requires
+          restarting the daemon.
+        </p>
+        <p>
+          The rte_mempool map is created at the time of vswitchd init and its
+          been used by all DPDK processes including vswitchd in the platform.
+          <code>dpdk-isolate-mem</code> option let vswitchd to use a separate
+          memory region than other processes for isolation. The separate
+          memory region map are prefixed by <code>ovs-<var>pid</var></code>
+          in the filesystem. <var>pid</var> is the process id of vswitchd
+          process.
+        </p>
+      </column>
+
     </group>
 
     <group title="Status">