| Message ID | 1510277313-44817-3-git-send-email-sugesh.chandran@intel.com |
|---|---|
| State | Deferred |
| Delegated to: | Ian Stokes |
| Headers | show |
| Series | Adding ovs configuration options to run multiple DPDK instances on a single platform. | expand |
Hi Sugesh,
This is definitely a very useful feature. I'm looking forward to running trex on the same DUT as my ovs-dpdk.
However I'd suggest adding an sscanf or some such to verify that the domain is also specified for each whitelist member. And either add the default of '0000' or complain loudly if the domain is absent.
Currently (without this patch) you must specify the domain when adding ports:
Vsctl add-port ... options:dpdk-devargs=0000:05:00.0
Or else an error such as 'Cannot find unplugged device (05:00.0)' is reported.
And with the patch if you include the domain in the other_config (e.g. other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything works just as before.
However with the patch if you add the whitelist *without* a domain e.g.
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-whitelist-pci-ids="05:00.0"
There is no immediate error. However later when doing add-port if you include the domain (current required practice) you will get an error. If you omit the domain all is well.
It's a little bit strange as regardless of domain or no domain in the other_config the PCI probe always reports the NIC as expected:
2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device 0000:05:00.0 on NUMA socket -1
2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL: probe driver: 8086:1572 net_i40e
I'll be using the other patch in this series "isolate rte-mempool allocation" over the next few days so I'll review that in due course.
Thanks,
Billy.
> -----Original Message-----
> From: ovs-dev-bounces@openvswitch.org [mailto:ovs-dev-
> bounces@openvswitch.org] On Behalf Of Sugesh Chandran
> Sent: Friday, November 10, 2017 1:29 AM
> To: dev@openvswitch.org; blp@ovn.org
> Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist DPDK
> physical ports.
>
> Adding a OVS configuration option to whitelist DPDK physical ports. By default
> running multiple instances of DPDK on a single platform cannot use physical
> ports at the same time even though they are distinct.
>
> The eal init scans all the ports that are bound to DPDK and initialize the drivers
> accordingly. This happens for every DPDK process init.
> On a multi instance deployment usecase, it causes issues for using physical NIC
> ports.
> Consider a two DPDK process that are running on a single platform, the second
> DPDK primary process will try to initialize the drivers for all the physical ports
> even though it may be used in first DPDK process.
>
> To avoid this situation user can whitelist the ports for each DPDK application.
> Whitelisting of ports/PCI-ID in a DPDK process will limit the eal-init only on those
> ports.
>
> To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1", the
> configuration option in OVS would be
> ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist-pci-
> ids="0000:06:00.0,0000:06:00.1"
>
> To update the whitelist ports, OVS daemon has to be restarted.
>
> Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com>
> ---
> lib/dpdk.c | 29 +++++++++++++++++++++++++++++
> vswitchd/vswitch.xml | 21 +++++++++++++++++++++
> 2 files changed, 50 insertions(+)
>
> diff --git a/lib/dpdk.c b/lib/dpdk.c
> index 9d187c7..0f11977 100644
> --- a/lib/dpdk.c
> +++ b/lib/dpdk.c
> @@ -323,6 +323,34 @@ dpdk_isolate_rte_mem_config(const struct smap
> *ovs_other_config, }
>
> static void
> +dpdk_whitelist_pci_ids(const struct smap *ovs_other_config, char ***argv,
> + int *argc)
> +{
> + const char *pci_ids;
> + char *pci_dev;
> + int len;
> + int i;
> + pci_ids = smap_get(ovs_other_config, "dpdk-whitelist-pci-ids");
> + if (!pci_ids) {
> + return;
> + }
> + len = strlen(pci_ids);
> + do {
> + i = strcspn(pci_ids, ",");
> + pci_dev = xmemdup0(pci_ids, i);
> + if (!strlen(pci_dev)) {
> + break;
> + }
> + *argv = grow_argv(argv, *argc, 2);
> + (*argv)[(*argc)++] = xstrdup("-w");
> + (*argv)[(*argc)++] = pci_dev;
> + i++;
> + pci_ids += i;
> + len -= i;
> + } while (pci_ids && len > 0);
> +}
> +
> +static void
> dpdk_init__(const struct smap *ovs_other_config) {
> char **argv = NULL, **argv_to_release = NULL; @@ -409,6 +437,7 @@
> dpdk_init__(const struct smap *ovs_other_config)
> }
>
> dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc);
> + dpdk_whitelist_pci_ids(ovs_other_config, &argv, &argc);
> argv = grow_argv(&argv, argc, 1);
> argv[argc] = NULL;
>
> diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index
> 7462b30..0b64b25 100644
> --- a/vswitchd/vswitch.xml
> +++ b/vswitchd/vswitch.xml
> @@ -442,6 +442,27 @@
> </p>
> </column>
>
> + <column name="other_config" key="dpdk-whitelist-pci-ids">
> + <p>
> + Specifies list of pci-ids separated by , for whitelisting available
> + physical NIC ports in OVS. The option valid only when DPDK is enabled
> + in OVS and the ports are already bound to DPDK userspace driver.
> + </p>
> + <p>
> + By default, all the DPDK bound ports are initialized at the time of
> + vswitchd start. Whitelisting a list of pci-ids is used to limit the
> + initialization only to specific ports. Changing this value requires
> + restarting the daemon.
> + </p>
> + <p>
> + This option allow user to run multiple instance of DPDK including
> + vswitchd simultaneously by exclusively allocating the physical NIC
> + ports between them. Its impossible to use the NIC ports in
> + multiple primary DPDK processes without whitelisting them
> + even the application is using distinct ports.
> + </p>
> + </column>
> +
> </group>
>
> <group title="Status">
> --
> 2.7.4
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Thank you Billy for the review. Please find below my reply. Regards _Sugesh > -----Original Message----- > From: O Mahony, Billy > Sent: Wednesday, December 6, 2017 5:31 PM > To: Chandran, Sugesh <sugesh.chandran@intel.com>; dev@openvswitch.org; > blp@ovn.org > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist > DPDK physical ports. > > Hi Sugesh, > > This is definitely a very useful feature. I'm looking forward to running trex on the > same DUT as my ovs-dpdk. > > However I'd suggest adding an sscanf or some such to verify that the domain is > also specified for each whitelist member. And either add the default of '0000' or > complain loudly if the domain is absent. [Sugesh] Will throw an error in that case then . > > Currently (without this patch) you must specify the domain when adding ports: > Vsctl add-port ... options:dpdk-devargs=0000:05:00.0 Or else an error such as > 'Cannot find unplugged device (05:00.0)' is reported. > > And with the patch if you include the domain in the other_config (e.g. > other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything works just as > before. > > However with the patch if you add the whitelist *without* a domain e.g. > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-whitelist-pci- > ids="05:00.0" > > There is no immediate error. However later when doing add-port if you include > the domain (current required practice) you will get an error. If you omit the > domain all is well. [Sugesh] It looks to me, the dpdk-devargs need the PCI id with the '0000'. But to bind and PCI scan its not necessary. So to keep it consistent, I would add check for PCI-ID in whitelist config too, and throw error incase pci-id are mentioned wrong(means without '0000'. Does it looks OK to you? > > It's a little bit strange as regardless of domain or no domain in the other_config > the PCI probe always reports the NIC as expected: > 2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device 0000:05:00.0 on > NUMA socket -1 > 2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL: Invalid NUMA socket, > default to 0 > 2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL: probe driver: 8086:1572 > net_i40e > > I'll be using the other patch in this series "isolate rte-mempool allocation" over > the next few days so I'll review that in due course. > > Thanks, > Billy. > > > -----Original Message----- > > From: ovs-dev-bounces@openvswitch.org [mailto:ovs-dev- > > bounces@openvswitch.org] On Behalf Of Sugesh Chandran > > Sent: Friday, November 10, 2017 1:29 AM > > To: dev@openvswitch.org; blp@ovn.org > > Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to > > whitelist DPDK physical ports. > > > > Adding a OVS configuration option to whitelist DPDK physical ports. By > > default running multiple instances of DPDK on a single platform cannot > > use physical ports at the same time even though they are distinct. > > > > The eal init scans all the ports that are bound to DPDK and initialize > > the drivers accordingly. This happens for every DPDK process init. > > On a multi instance deployment usecase, it causes issues for using > > physical NIC ports. > > Consider a two DPDK process that are running on a single platform, the > > second DPDK primary process will try to initialize the drivers for all > > the physical ports even though it may be used in first DPDK process. > > > > To avoid this situation user can whitelist the ports for each DPDK application. > > Whitelisting of ports/PCI-ID in a DPDK process will limit the eal-init > > only on those ports. > > > > To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1", the > > configuration option in OVS would be > > ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist-pci- > > ids="0000:06:00.0,0000:06:00.1" > > > > To update the whitelist ports, OVS daemon has to be restarted. > > > > Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com> > > --- > > lib/dpdk.c | 29 +++++++++++++++++++++++++++++ > > vswitchd/vswitch.xml | 21 +++++++++++++++++++++ > > 2 files changed, 50 insertions(+) > > > > diff --git a/lib/dpdk.c b/lib/dpdk.c > > index 9d187c7..0f11977 100644 > > --- a/lib/dpdk.c > > +++ b/lib/dpdk.c > > @@ -323,6 +323,34 @@ dpdk_isolate_rte_mem_config(const struct smap > > *ovs_other_config, } > > > > static void > > +dpdk_whitelist_pci_ids(const struct smap *ovs_other_config, char ***argv, > > + int *argc) > > +{ > > + const char *pci_ids; > > + char *pci_dev; > > + int len; > > + int i; > > + pci_ids = smap_get(ovs_other_config, "dpdk-whitelist-pci-ids"); > > + if (!pci_ids) { > > + return; > > + } > > + len = strlen(pci_ids); > > + do { > > + i = strcspn(pci_ids, ","); > > + pci_dev = xmemdup0(pci_ids, i); > > + if (!strlen(pci_dev)) { > > + break; > > + } > > + *argv = grow_argv(argv, *argc, 2); > > + (*argv)[(*argc)++] = xstrdup("-w"); > > + (*argv)[(*argc)++] = pci_dev; > > + i++; > > + pci_ids += i; > > + len -= i; > > + } while (pci_ids && len > 0); > > +} > > + > > +static void > > dpdk_init__(const struct smap *ovs_other_config) { > > char **argv = NULL, **argv_to_release = NULL; @@ -409,6 +437,7 @@ > > dpdk_init__(const struct smap *ovs_other_config) > > } > > > > dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc); > > + dpdk_whitelist_pci_ids(ovs_other_config, &argv, &argc); > > argv = grow_argv(&argv, argc, 1); > > argv[argc] = NULL; > > > > diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index > > 7462b30..0b64b25 100644 > > --- a/vswitchd/vswitch.xml > > +++ b/vswitchd/vswitch.xml > > @@ -442,6 +442,27 @@ > > </p> > > </column> > > > > + <column name="other_config" key="dpdk-whitelist-pci-ids"> > > + <p> > > + Specifies list of pci-ids separated by , for whitelisting available > > + physical NIC ports in OVS. The option valid only when DPDK is enabled > > + in OVS and the ports are already bound to DPDK userspace driver. > > + </p> > > + <p> > > + By default, all the DPDK bound ports are initialized at the time of > > + vswitchd start. Whitelisting a list of pci-ids is used to limit the > > + initialization only to specific ports. Changing this value requires > > + restarting the daemon. > > + </p> > > + <p> > > + This option allow user to run multiple instance of DPDK including > > + vswitchd simultaneously by exclusively allocating the physical NIC > > + ports between them. Its impossible to use the NIC ports in > > + multiple primary DPDK processes without whitelisting them > > + even the application is using distinct ports. > > + </p> > > + </column> > > + > > </group> > > > > <group title="Status"> > > -- > > 2.7.4 > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Hi Sugesh, > -----Original Message----- > From: Chandran, Sugesh > Sent: Wednesday, December 6, 2017 6:23 PM > To: O Mahony, Billy <billy.o.mahony@intel.com>; dev@openvswitch.org; > blp@ovn.org > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist > DPDK physical ports. > > Thank you Billy for the review. > Please find below my reply. > > Regards > _Sugesh > > > > -----Original Message----- > > From: O Mahony, Billy > > Sent: Wednesday, December 6, 2017 5:31 PM > > To: Chandran, Sugesh <sugesh.chandran@intel.com>; dev@openvswitch.org; > > blp@ovn.org > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to > > whitelist DPDK physical ports. > > > > Hi Sugesh, > > > > This is definitely a very useful feature. I'm looking forward to > > running trex on the same DUT as my ovs-dpdk. > > > > However I'd suggest adding an sscanf or some such to verify that the > > domain is also specified for each whitelist member. And either add the > > default of '0000' or complain loudly if the domain is absent. > [Sugesh] Will throw an error in that case then . > > > > > Currently (without this patch) you must specify the domain when adding ports: > > Vsctl add-port ... options:dpdk-devargs=0000:05:00.0 Or else an > > error such as 'Cannot find unplugged device (05:00.0)' is reported. > > > > And with the patch if you include the domain in the other_config (e.g. > > other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything works > > just as before. > > > > However with the patch if you add the whitelist *without* a domain e.g. > > ovs-vsctl --no-wait set Open_vSwitch . > > other_config:dpdk-whitelist-pci- ids="05:00.0" > > > > There is no immediate error. However later when doing add-port if you > > include the domain (current required practice) you will get an error. > > If you omit the domain all is well. > [Sugesh] It looks to me, the dpdk-devargs need the PCI id with the '0000'. > But to bind and PCI scan its not necessary. > So to keep it consistent, I would add check for PCI-ID in whitelist config too, and > throw error incase pci-id are mentioned wrong(means without '0000'. > Does it looks OK to you? [[BO'M]] I think the error is the right thing to do. It would be tempting to insert the default '0000' if the domain is omitted but then you would have a confusing inconsistency in that it would be ok to omit the domain in one place (whitelist) but not in the other (add-port). > > > > It's a little bit strange as regardless of domain or no domain in the > > other_config the PCI probe always reports the NIC as expected: > > 2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device 0000:05:00.0 > > on NUMA socket -1 > > 2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL: Invalid NUMA socket, > > default to 0 > > 2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL: probe driver: 8086:1572 > > net_i40e > > > > I'll be using the other patch in this series "isolate rte-mempool > > allocation" over the next few days so I'll review that in due course. > > > > Thanks, > > Billy. > > > > > -----Original Message----- > > > From: ovs-dev-bounces@openvswitch.org [mailto:ovs-dev- > > > bounces@openvswitch.org] On Behalf Of Sugesh Chandran > > > Sent: Friday, November 10, 2017 1:29 AM > > > To: dev@openvswitch.org; blp@ovn.org > > > Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > whitelist DPDK physical ports. > > > > > > Adding a OVS configuration option to whitelist DPDK physical ports. > > > By default running multiple instances of DPDK on a single platform > > > cannot use physical ports at the same time even though they are distinct. > > > > > > The eal init scans all the ports that are bound to DPDK and > > > initialize the drivers accordingly. This happens for every DPDK process init. > > > On a multi instance deployment usecase, it causes issues for using > > > physical NIC ports. > > > Consider a two DPDK process that are running on a single platform, > > > the second DPDK primary process will try to initialize the drivers > > > for all the physical ports even though it may be used in first DPDK process. > > > > > > To avoid this situation user can whitelist the ports for each DPDK application. > > > Whitelisting of ports/PCI-ID in a DPDK process will limit the > > > eal-init only on those ports. > > > > > > To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1", > > > the configuration option in OVS would be > > > ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist-pci- > > > ids="0000:06:00.0,0000:06:00.1" > > > > > > To update the whitelist ports, OVS daemon has to be restarted. > > > > > > Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com> > > > --- > > > lib/dpdk.c | 29 +++++++++++++++++++++++++++++ > > > vswitchd/vswitch.xml | 21 +++++++++++++++++++++ > > > 2 files changed, 50 insertions(+) > > > > > > diff --git a/lib/dpdk.c b/lib/dpdk.c index 9d187c7..0f11977 100644 > > > --- a/lib/dpdk.c > > > +++ b/lib/dpdk.c > > > @@ -323,6 +323,34 @@ dpdk_isolate_rte_mem_config(const struct smap > > > *ovs_other_config, } > > > > > > static void > > > +dpdk_whitelist_pci_ids(const struct smap *ovs_other_config, char ***argv, > > > + int *argc) > > > +{ > > > + const char *pci_ids; > > > + char *pci_dev; > > > + int len; > > > + int i; > > > + pci_ids = smap_get(ovs_other_config, "dpdk-whitelist-pci-ids"); > > > + if (!pci_ids) { > > > + return; > > > + } > > > + len = strlen(pci_ids); > > > + do { > > > + i = strcspn(pci_ids, ","); > > > + pci_dev = xmemdup0(pci_ids, i); > > > + if (!strlen(pci_dev)) { > > > + break; > > > + } > > > + *argv = grow_argv(argv, *argc, 2); > > > + (*argv)[(*argc)++] = xstrdup("-w"); > > > + (*argv)[(*argc)++] = pci_dev; > > > + i++; > > > + pci_ids += i; > > > + len -= i; > > > + } while (pci_ids && len > 0); > > > +} > > > + > > > +static void > > > dpdk_init__(const struct smap *ovs_other_config) { > > > char **argv = NULL, **argv_to_release = NULL; @@ -409,6 +437,7 > > > @@ dpdk_init__(const struct smap *ovs_other_config) > > > } > > > > > > dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc); > > > + dpdk_whitelist_pci_ids(ovs_other_config, &argv, &argc); > > > argv = grow_argv(&argv, argc, 1); > > > argv[argc] = NULL; > > > > > > diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index > > > 7462b30..0b64b25 100644 > > > --- a/vswitchd/vswitch.xml > > > +++ b/vswitchd/vswitch.xml > > > @@ -442,6 +442,27 @@ > > > </p> > > > </column> > > > > > > + <column name="other_config" key="dpdk-whitelist-pci-ids"> > > > + <p> > > > + Specifies list of pci-ids separated by , for whitelisting available > > > + physical NIC ports in OVS. The option valid only when DPDK is enabled > > > + in OVS and the ports are already bound to DPDK userspace driver. > > > + </p> > > > + <p> > > > + By default, all the DPDK bound ports are initialized at the time of > > > + vswitchd start. Whitelisting a list of pci-ids is used to limit the > > > + initialization only to specific ports. Changing this value requires > > > + restarting the daemon. > > > + </p> > > > + <p> > > > + This option allow user to run multiple instance of DPDK including > > > + vswitchd simultaneously by exclusively allocating the physical NIC > > > + ports between them. Its impossible to use the NIC ports in > > > + multiple primary DPDK processes without whitelisting them > > > + even the application is using distinct ports. > > > + </p> > > > + </column> > > > + > > > </group> > > > > > > <group title="Status"> > > > -- > > > 2.7.4 > > > > > > _______________________________________________ > > > dev mailing list > > > dev@openvswitch.org > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Regards _Sugesh > -----Original Message----- > From: O Mahony, Billy > Sent: Thursday, December 7, 2017 11:47 AM > To: Chandran, Sugesh <sugesh.chandran@intel.com>; dev@openvswitch.org; > blp@ovn.org > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist > DPDK physical ports. > > Hi Sugesh, > > > -----Original Message----- > > From: Chandran, Sugesh > > Sent: Wednesday, December 6, 2017 6:23 PM > > To: O Mahony, Billy <billy.o.mahony@intel.com>; dev@openvswitch.org; > > blp@ovn.org > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to > > whitelist DPDK physical ports. > > > > Thank you Billy for the review. > > Please find below my reply. > > > > Regards > > _Sugesh > > > > > > > -----Original Message----- > > > From: O Mahony, Billy > > > Sent: Wednesday, December 6, 2017 5:31 PM > > > To: Chandran, Sugesh <sugesh.chandran@intel.com>; > > > dev@openvswitch.org; blp@ovn.org > > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > whitelist DPDK physical ports. > > > > > > Hi Sugesh, > > > > > > This is definitely a very useful feature. I'm looking forward to > > > running trex on the same DUT as my ovs-dpdk. > > > > > > However I'd suggest adding an sscanf or some such to verify that the > > > domain is also specified for each whitelist member. And either add > > > the default of '0000' or complain loudly if the domain is absent. > > [Sugesh] Will throw an error in that case then . > > > > > > > > Currently (without this patch) you must specify the domain when adding > ports: > > > Vsctl add-port ... options:dpdk-devargs=0000:05:00.0 Or else an > > > error such as 'Cannot find unplugged device (05:00.0)' is reported. > > > > > > And with the patch if you include the domain in the other_config (e.g. > > > other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything works > > > just as before. > > > > > > However with the patch if you add the whitelist *without* a domain e.g. > > > ovs-vsctl --no-wait set Open_vSwitch . > > > other_config:dpdk-whitelist-pci- ids="05:00.0" > > > > > > There is no immediate error. However later when doing add-port if > > > you include the domain (current required practice) you will get an error. > > > If you omit the domain all is well. > > [Sugesh] It looks to me, the dpdk-devargs need the PCI id with the '0000'. > > But to bind and PCI scan its not necessary. > > So to keep it consistent, I would add check for PCI-ID in whitelist > > config too, and throw error incase pci-id are mentioned wrong(means without > '0000'. > > Does it looks OK to you? > > [[BO'M]] I think the error is the right thing to do. It would be tempting to insert > the default '0000' if the domain is omitted but then you would have a confusing > inconsistency in that it would be ok to omit the domain in one place (whitelist) > but not in the other (add-port). > [Sugesh] Thank you Billy,Will add the check and error out accordingly. Will release the next patch version by incorporating the comments, once you have complete the review of second part of the patch too. > > > > > > It's a little bit strange as regardless of domain or no domain in > > > the other_config the PCI probe always reports the NIC as expected: > > > 2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device > > > 0000:05:00.0 on NUMA socket -1 > > > 2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL: Invalid NUMA socket, > > > default to 0 > > > 2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL: probe driver: 8086:1572 > > > net_i40e > > > > > > I'll be using the other patch in this series "isolate rte-mempool > > > allocation" over the next few days so I'll review that in due course. > > > > > > Thanks, > > > Billy. > > > > > > > -----Original Message----- > > > > From: ovs-dev-bounces@openvswitch.org [mailto:ovs-dev- > > > > bounces@openvswitch.org] On Behalf Of Sugesh Chandran > > > > Sent: Friday, November 10, 2017 1:29 AM > > > > To: dev@openvswitch.org; blp@ovn.org > > > > Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > > whitelist DPDK physical ports. > > > > > > > > Adding a OVS configuration option to whitelist DPDK physical ports. > > > > By default running multiple instances of DPDK on a single platform > > > > cannot use physical ports at the same time even though they are distinct. > > > > > > > > The eal init scans all the ports that are bound to DPDK and > > > > initialize the drivers accordingly. This happens for every DPDK process init. > > > > On a multi instance deployment usecase, it causes issues for using > > > > physical NIC ports. > > > > Consider a two DPDK process that are running on a single platform, > > > > the second DPDK primary process will try to initialize the drivers > > > > for all the physical ports even though it may be used in first DPDK process. > > > > > > > > To avoid this situation user can whitelist the ports for each DPDK > application. > > > > Whitelisting of ports/PCI-ID in a DPDK process will limit the > > > > eal-init only on those ports. > > > > > > > > To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1", > > > > the configuration option in OVS would be > > > > ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist-pci- > > > > ids="0000:06:00.0,0000:06:00.1" > > > > > > > > To update the whitelist ports, OVS daemon has to be restarted. > > > > > > > > Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com> > > > > --- > > > > lib/dpdk.c | 29 +++++++++++++++++++++++++++++ [snip] > > > > > > > > _______________________________________________ > > > > dev mailing list > > > > dev@openvswitch.org > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> -----Original Message----- > From: ovs-dev-bounces@openvswitch.org [mailto:ovs-dev- > bounces@openvswitch.org] On Behalf Of Chandran, Sugesh > Sent: Thursday, December 7, 2017 5:07 PM > To: O Mahony, Billy <billy.o.mahony@intel.com>; dev@openvswitch.org; > blp@ovn.org > Subject: Re: [ovs-dev] [PATCH 2/2] Adding configuration option to > whitelist DPDK physical ports. > > > > Regards > _Sugesh > > > -----Original Message----- > > From: O Mahony, Billy > > Sent: Thursday, December 7, 2017 11:47 AM > > To: Chandran, Sugesh <sugesh.chandran@intel.com>; > dev@openvswitch.org; > > blp@ovn.org > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to > > whitelist DPDK physical ports. > > > > Hi Sugesh, > > > > > -----Original Message----- > > > From: Chandran, Sugesh > > > Sent: Wednesday, December 6, 2017 6:23 PM > > > To: O Mahony, Billy <billy.o.mahony@intel.com>; > dev@openvswitch.org; > > > blp@ovn.org > > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > whitelist DPDK physical ports. > > > > > > Thank you Billy for the review. > > > Please find below my reply. > > > > > > Regards > > > _Sugesh > > > > > > > > > > -----Original Message----- > > > > From: O Mahony, Billy > > > > Sent: Wednesday, December 6, 2017 5:31 PM > > > > To: Chandran, Sugesh <sugesh.chandran@intel.com>; > > > > dev@openvswitch.org; blp@ovn.org > > > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > > whitelist DPDK physical ports. > > > > > > > > Hi Sugesh, > > > > > > > > This is definitely a very useful feature. I'm looking forward to > > > > running trex on the same DUT as my ovs-dpdk. [Mooney, Sean K] you can all ready to this you just need to set the whitelist In other_config:dpdk-extra just repeat "-w $address" for each device. To have two dpdk primary processes on the same system you will also need to change The hugepage prfix used be dpdk which you can also do via the dpdk-extra option. After this patch we will still be able to specify the whitelist using other_config:dpdk-extra correct? If not this may break ovs-dpdk support in openstack installers. I ported our whitelist code in networking-ovs-dpdk to use dpdk-extra when when we moved the dpdk params to the db and I also added it to kolla. im pretty sure tripple0 and fule also do the same. > > > > > > > > However I'd suggest adding an sscanf or some such to verify that > > > > the domain is also specified for each whitelist member. And > either > > > > add the default of '0000' or complain loudly if the domain is > absent. > > > [Sugesh] Will throw an error in that case then . > > > > > > > > > > > Currently (without this patch) you must specify the domain when > > > > adding > > ports: > > > > Vsctl add-port ... options:dpdk-devargs=0000:05:00.0 Or else > an > > > > error such as 'Cannot find unplugged device (05:00.0)' is > reported. > > > > > > > > And with the patch if you include the domain in the other_config > (e.g. > > > > other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything > > > > works just as before. > > > > > > > > However with the patch if you add the whitelist *without* a > domain e.g. > > > > ovs-vsctl --no-wait set Open_vSwitch . > > > > other_config:dpdk-whitelist-pci- ids="05:00.0" > > > > > > > > There is no immediate error. However later when doing add-port if > > > > you include the domain (current required practice) you will get > an error. > > > > If you omit the domain all is well. > > > [Sugesh] It looks to me, the dpdk-devargs need the PCI id with the > '0000'. > > > But to bind and PCI scan its not necessary. > > > So to keep it consistent, I would add check for PCI-ID in whitelist > > > config too, and throw error incase pci-id are mentioned wrong(means > > > without > > '0000'. [Mooney, Sean K] don't assume it is '0000' it is only '0000' if the pci device is A child of the first pci root usally connected to socket 0 unless you server is old Enough to still use a FSB instead of qpi/dmi. > > > Does it looks OK to you? > > > > [[BO'M]] I think the error is the right thing to do. It would be > > tempting to insert the default '0000' if the domain is omitted but > > then you would have a confusing inconsistency in that it would be ok > > to omit the domain in one place (whitelist) but not in the other > (add-port). > > > [Sugesh] Thank you Billy,Will add the check and error out accordingly. > Will release the next patch version by incorporating the comments, once > you have complete the review of second part of the patch too. [Mooney, Sean K] it should be noted that the dpdk devbing script will only work without The domain if its unambiguious. On a multi socket system with the right or wrong depending on your perspective placement of physical pci devise it possible to get colltions in which case dpdk-devbind will not be able to correctly determing what device to manage so you should always use the full pci address. > > > > > > > > It's a little bit strange as regardless of domain or no domain in > > > > the other_config the PCI probe always reports the NIC as > expected: > > > > 2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device > > > > 0000:05:00.0 on NUMA socket -1 > > > > 2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL: Invalid NUMA > socket, > > > > default to 0 > > > > 2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL: probe driver: > 8086:1572 > > > > net_i40e > > > > > > > > I'll be using the other patch in this series "isolate rte-mempool > > > > allocation" over the next few days so I'll review that in due > course. > > > > > > > > Thanks, > > > > Billy. > > > > > > > > > -----Original Message----- > > > > > From: ovs-dev-bounces@openvswitch.org [mailto:ovs-dev- > > > > > bounces@openvswitch.org] On Behalf Of Sugesh Chandran > > > > > Sent: Friday, November 10, 2017 1:29 AM > > > > > To: dev@openvswitch.org; blp@ovn.org > > > > > Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > > > whitelist DPDK physical ports. > > > > > > > > > > Adding a OVS configuration option to whitelist DPDK physical > ports. > > > > > By default running multiple instances of DPDK on a single > > > > > platform cannot use physical ports at the same time even though > they are distinct. > > > > > > > > > > The eal init scans all the ports that are bound to DPDK and > > > > > initialize the drivers accordingly. This happens for every DPDK > process init. > > > > > On a multi instance deployment usecase, it causes issues for > > > > > using physical NIC ports. > > > > > Consider a two DPDK process that are running on a single > > > > > platform, the second DPDK primary process will try to > initialize > > > > > the drivers for all the physical ports even though it may be > used in first DPDK process. > > > > > > > > > > To avoid this situation user can whitelist the ports for each > > > > > DPDK > > application. > > > > > Whitelisting of ports/PCI-ID in a DPDK process will limit the > > > > > eal-init only on those ports. > > > > > > > > > > To whitelist two physical ports "0000:06:00.0" and > > > > > "0000:06:00.1", the configuration option in OVS would be > > > > > ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist- > pci- > > > > > ids="0000:06:00.0,0000:06:00.1" > > > > > > > > > > To update the whitelist ports, OVS daemon has to be restarted. > > > > > > > > > > Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com> > > > > > --- > > > > > lib/dpdk.c | 29 +++++++++++++++++++++++++++++ > [snip] > > > > > > > > > > _______________________________________________ > > > > > dev mailing list > > > > > dev@openvswitch.org > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
I can confirm that using other_config:dpdk-extra is indeed already effective to change the hugepage file prefix (admittedly without inserting a ref to the pid) and specify a pci whitelist. Regards, Billy. > -----Original Message----- > From: Mooney, Sean K > Sent: Thursday, December 7, 2017 5:53 PM > To: Chandran, Sugesh <sugesh.chandran@intel.com>; O Mahony, Billy > <billy.o.mahony@intel.com>; dev@openvswitch.org; blp@ovn.org > Cc: Mooney, Sean K <sean.k.mooney@intel.com> > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist > DPDK physical ports. > > > > > -----Original Message----- > > From: ovs-dev-bounces@openvswitch.org [mailto:ovs-dev- > > bounces@openvswitch.org] On Behalf Of Chandran, Sugesh > > Sent: Thursday, December 7, 2017 5:07 PM > > To: O Mahony, Billy <billy.o.mahony@intel.com>; dev@openvswitch.org; > > blp@ovn.org > > Subject: Re: [ovs-dev] [PATCH 2/2] Adding configuration option to > > whitelist DPDK physical ports. > > > > > > > > Regards > > _Sugesh > > > > > -----Original Message----- > > > From: O Mahony, Billy > > > Sent: Thursday, December 7, 2017 11:47 AM > > > To: Chandran, Sugesh <sugesh.chandran@intel.com>; > > dev@openvswitch.org; > > > blp@ovn.org > > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > whitelist DPDK physical ports. > > > > > > Hi Sugesh, > > > > > > > -----Original Message----- > > > > From: Chandran, Sugesh > > > > Sent: Wednesday, December 6, 2017 6:23 PM > > > > To: O Mahony, Billy <billy.o.mahony@intel.com>; > > dev@openvswitch.org; > > > > blp@ovn.org > > > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > > whitelist DPDK physical ports. > > > > > > > > Thank you Billy for the review. > > > > Please find below my reply. > > > > > > > > Regards > > > > _Sugesh > > > > > > > > > > > > > -----Original Message----- > > > > > From: O Mahony, Billy > > > > > Sent: Wednesday, December 6, 2017 5:31 PM > > > > > To: Chandran, Sugesh <sugesh.chandran@intel.com>; > > > > > dev@openvswitch.org; blp@ovn.org > > > > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option > > > > > to whitelist DPDK physical ports. > > > > > > > > > > Hi Sugesh, > > > > > > > > > > This is definitely a very useful feature. I'm looking forward to > > > > > running trex on the same DUT as my ovs-dpdk. > [Mooney, Sean K] you can all ready to this you just need to set the whitelist In > other_config:dpdk-extra just repeat "-w $address" for each device. > To have two dpdk primary processes on the same system you will also need to > change The hugepage prfix used be dpdk which you can also do via the dpdk- > extra option. > > After this patch we will still be able to specify the whitelist using > other_config:dpdk-extra correct? If not this may break ovs-dpdk support in > openstack installers. I ported our whitelist code in networking-ovs-dpdk to use > dpdk-extra when when we moved the dpdk params to the db and I also added it > to kolla. > im pretty sure tripple0 and fule also do the same. > > > > > > > > > > > However I'd suggest adding an sscanf or some such to verify that > > > > > the domain is also specified for each whitelist member. And > > either > > > > > add the default of '0000' or complain loudly if the domain is > > absent. > > > > [Sugesh] Will throw an error in that case then . > > > > > > > > > > > > > > Currently (without this patch) you must specify the domain when > > > > > adding > > > ports: > > > > > Vsctl add-port ... options:dpdk-devargs=0000:05:00.0 Or else > > an > > > > > error such as 'Cannot find unplugged device (05:00.0)' is > > reported. > > > > > > > > > > And with the patch if you include the domain in the other_config > > (e.g. > > > > > other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything > > > > > works just as before. > > > > > > > > > > However with the patch if you add the whitelist *without* a > > domain e.g. > > > > > ovs-vsctl --no-wait set Open_vSwitch . > > > > > other_config:dpdk-whitelist-pci- ids="05:00.0" > > > > > > > > > > There is no immediate error. However later when doing add-port > > > > > if you include the domain (current required practice) you will > > > > > get > > an error. > > > > > If you omit the domain all is well. > > > > [Sugesh] It looks to me, the dpdk-devargs need the PCI id with the > > '0000'. > > > > But to bind and PCI scan its not necessary. > > > > So to keep it consistent, I would add check for PCI-ID in > > > > whitelist config too, and throw error incase pci-id are mentioned > > > > wrong(means without > > > '0000'. > [Mooney, Sean K] don't assume it is '0000' it is only '0000' if the pci device is A > child of the first pci root usally connected to socket 0 unless you server is old > Enough to still use a FSB instead of qpi/dmi. > > > > > Does it looks OK to you? > > > > > > [[BO'M]] I think the error is the right thing to do. It would be > > > tempting to insert the default '0000' if the domain is omitted but > > > then you would have a confusing inconsistency in that it would be ok > > > to omit the domain in one place (whitelist) but not in the other > > (add-port). > > > > > [Sugesh] Thank you Billy,Will add the check and error out accordingly. > > Will release the next patch version by incorporating the comments, > > once you have complete the review of second part of the patch too. > [Mooney, Sean K] it should be noted that the dpdk devbing script will only work > without The domain if its unambiguious. On a multi socket system with the right > or wrong depending on your perspective placement of physical pci devise it > possible to get colltions in which case dpdk-devbind will not be able to correctly > determing what device to manage so you should always use the full pci address. > > > > > > > > > > > It's a little bit strange as regardless of domain or no domain > > > > > in the other_config the PCI probe always reports the NIC as > > expected: > > > > > 2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device > > > > > 0000:05:00.0 on NUMA socket -1 > > > > > 2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL: Invalid NUMA > > socket, > > > > > default to 0 > > > > > 2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL: probe driver: > > 8086:1572 > > > > > net_i40e > > > > > > > > > > I'll be using the other patch in this series "isolate > > > > > rte-mempool allocation" over the next few days so I'll review > > > > > that in due > > course. > > > > > > > > > > Thanks, > > > > > Billy. > > > > > > > > > > > -----Original Message----- > > > > > > From: ovs-dev-bounces@openvswitch.org [mailto:ovs-dev- > > > > > > bounces@openvswitch.org] On Behalf Of Sugesh Chandran > > > > > > Sent: Friday, November 10, 2017 1:29 AM > > > > > > To: dev@openvswitch.org; blp@ovn.org > > > > > > Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to > > > > > > whitelist DPDK physical ports. > > > > > > > > > > > > Adding a OVS configuration option to whitelist DPDK physical > > ports. > > > > > > By default running multiple instances of DPDK on a single > > > > > > platform cannot use physical ports at the same time even > > > > > > though > > they are distinct. > > > > > > > > > > > > The eal init scans all the ports that are bound to DPDK and > > > > > > initialize the drivers accordingly. This happens for every > > > > > > DPDK > > process init. > > > > > > On a multi instance deployment usecase, it causes issues for > > > > > > using physical NIC ports. > > > > > > Consider a two DPDK process that are running on a single > > > > > > platform, the second DPDK primary process will try to > > initialize > > > > > > the drivers for all the physical ports even though it may be > > used in first DPDK process. > > > > > > > > > > > > To avoid this situation user can whitelist the ports for each > > > > > > DPDK > > > application. > > > > > > Whitelisting of ports/PCI-ID in a DPDK process will limit the > > > > > > eal-init only on those ports. > > > > > > > > > > > > To whitelist two physical ports "0000:06:00.0" and > > > > > > "0000:06:00.1", the configuration option in OVS would be > > > > > > ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist- > > pci- > > > > > > ids="0000:06:00.0,0000:06:00.1" > > > > > > > > > > > > To update the whitelist ports, OVS daemon has to be restarted. > > > > > > > > > > > > Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com> > > > > > > --- > > > > > > lib/dpdk.c | 29 +++++++++++++++++++++++++++++ > > [snip] > > > > > > > > > > > > _______________________________________________ > > > > > > dev mailing list > > > > > > dev@openvswitch.org > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
diff --git a/lib/dpdk.c b/lib/dpdk.c index 9d187c7..0f11977 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -323,6 +323,34 @@ dpdk_isolate_rte_mem_config(const struct smap *ovs_other_config, } static void +dpdk_whitelist_pci_ids(const struct smap *ovs_other_config, char ***argv, + int *argc) +{ + const char *pci_ids; + char *pci_dev; + int len; + int i; + pci_ids = smap_get(ovs_other_config, "dpdk-whitelist-pci-ids"); + if (!pci_ids) { + return; + } + len = strlen(pci_ids); + do { + i = strcspn(pci_ids, ","); + pci_dev = xmemdup0(pci_ids, i); + if (!strlen(pci_dev)) { + break; + } + *argv = grow_argv(argv, *argc, 2); + (*argv)[(*argc)++] = xstrdup("-w"); + (*argv)[(*argc)++] = pci_dev; + i++; + pci_ids += i; + len -= i; + } while (pci_ids && len > 0); +} + +static void dpdk_init__(const struct smap *ovs_other_config) { char **argv = NULL, **argv_to_release = NULL; @@ -409,6 +437,7 @@ dpdk_init__(const struct smap *ovs_other_config) } dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc); + dpdk_whitelist_pci_ids(ovs_other_config, &argv, &argc); argv = grow_argv(&argv, argc, 1); argv[argc] = NULL; diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 7462b30..0b64b25 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -442,6 +442,27 @@ </p> </column> + <column name="other_config" key="dpdk-whitelist-pci-ids"> + <p> + Specifies list of pci-ids separated by , for whitelisting available + physical NIC ports in OVS. The option valid only when DPDK is enabled + in OVS and the ports are already bound to DPDK userspace driver. + </p> + <p> + By default, all the DPDK bound ports are initialized at the time of + vswitchd start. Whitelisting a list of pci-ids is used to limit the + initialization only to specific ports. Changing this value requires + restarting the daemon. + </p> + <p> + This option allow user to run multiple instance of DPDK including + vswitchd simultaneously by exclusively allocating the physical NIC + ports between them. Its impossible to use the NIC ports in + multiple primary DPDK processes without whitelisting them + even the application is using distinct ports. + </p> + </column> + </group> <group title="Status">
Adding a OVS configuration option to whitelist DPDK physical ports. By default running multiple instances of DPDK on a single platform cannot use physical ports at the same time even though they are distinct. The eal init scans all the ports that are bound to DPDK and initialize the drivers accordingly. This happens for every DPDK process init. On a multi instance deployment usecase, it causes issues for using physical NIC ports. Consider a two DPDK process that are running on a single platform, the second DPDK primary process will try to initialize the drivers for all the physical ports even though it may be used in first DPDK process. To avoid this situation user can whitelist the ports for each DPDK application. Whitelisting of ports/PCI-ID in a DPDK process will limit the eal-init only on those ports. To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1", the configuration option in OVS would be ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist-pci-ids="0000:06:00.0,0000:06:00.1" To update the whitelist ports, OVS daemon has to be restarted. Signed-off-by: Sugesh Chandran <sugesh.chandran@intel.com> --- lib/dpdk.c | 29 +++++++++++++++++++++++++++++ vswitchd/vswitch.xml | 21 +++++++++++++++++++++ 2 files changed, 50 insertions(+)