| Message ID | 1449116634-14671-3-git-send-email-aconole@redhat.com |
|---|---|
| State | RFC |
| Headers | show |
> -----Original Message----- > From: dev [mailto:dev-bounces@openvswitch.org] On Behalf Of Aaron > Conole > Sent: Thursday, December 3, 2015 4:24 AM > To: dev@openvswitch.org > Cc: Flavio Leitner > Subject: [ovs-dev] [RFC 2/2] dpdk: Update INSTALL.DPDK.md to reflect new > config > > A previous change modified the way DPDK configuration would be accepted > by the system. This updates the documentation to align with the new > method of configuring DPDK. > > Signed-off-by: Aaron Conole <aconole@redhat.com> > --- > INSTALL.DPDK.md | 52 ++++++++++++++++++++++++++++++++++++++---- > ---------- > 1 file changed, 38 insertions(+), 14 deletions(-) > > diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 96b686c..789ea68 > 100644 > --- a/INSTALL.DPDK.md > +++ b/INSTALL.DPDK.md > @@ -143,22 +143,46 @@ Using the DPDK with ovs-vswitchd: > [Gray, Mark D] On a general note, I guess for most of these database entries it doesn’t make sense to change them at runtime as they only affect the initialization stage. Off the top of my head, I can't think of any other entries that behave like this (perhaps there are though). Does this mean it would be possible to change them at runtime which would make the switch look like it was configured in a different way? Take the number of memory channels as an example. It may not be a problem but it is strange behavior. > 5. Start vswitchd: > > - DPDK configuration arguments can be passed to vswitchd via `--dpdk` > - argument. This needs to be first argument passed to vswitchd process. > - dpdk arg -c is ignored by ovs-dpdk, but it is a required parameter > - for dpdk initialization. > - > + DPDK configuration arguments can be passed to vswitchd via the > Open_vSwitch > + other_config database. The recognized configuration options are listed. > + > + * dpdk > + This is a boolean configuration option. A value of 'true' tells > + Open_vSwitch to initialize the DPDK EAL. A set of nominal defaults are > + provided so that simply enabling this option will be sufficient to configure > + DPDK enabled ports. > + * dpdk_core_mask > + This is DPDK's -c command line option, and specifies the core mask to > + the EAL. Why exactly do we need this again? We have "other_config:pmd-cpu-mask" which should set the affinity of the pmd threads. The affinities of the other standard pthreads should by a system administrator. Or, if the intention is that this would allow us to set affinities for the other threads, then the name should be changed to something more meaningful. > + * dpdk_mem_channels > + This is DPDK's -n option, to configure the number of memory spread > channels > + into the CPU. It is purely an optimization flag. > + * dpdk_alloc_mem > + Total memory to pre-allocate > + * dpdk_socket_mem > + Memory to allocate on specific sockets (use comma separated values) > + * dpdk_hugepage_dir > + Directory where hugetlbfs is mounted > + * cuse_dev_name > + Option to set the vhost_cuse character device name. > + * vhost_sock_dir > + Specify a custom directory for the vhost_user unix socket files. > + > + Note that changing any of these options requires restarting the ovs- > vswitchd > + application. > + > ``` > export DB_SOCK=/usr/local/var/run/openvswitch/db.sock > - ovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK --pidfile --detach > + ovs-vsctl set Open_vSwitch . other_config:dpdk=true > + ovs-vswitchd unix:$DB_SOCK --pidfile --detach > ``` > > If allocated more than one GB hugepage (as for IVSHMEM), set amount > and > use NUMA node 0 memory: > > ``` > - ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 1024,0 \ > - -- unix:$DB_SOCK --pidfile --detach > + ovs-vsctl set Open_vSwitch . other_config:dpdk_socket_mem="1024,0" > + ovs-vswitchd unix:$DB_SOCK --pidfile --detach > ``` > > 6. Add bridge & ports > @@ -523,9 +547,9 @@ have arbitrary names. > found in the next section "DPDK vhost-user VM configuration" > Note: If you wish for the vhost-user sockets to be created in a > directory other than `/usr/local/var/run/openvswitch`, you may specify > - another location on the ovs-vswitchd command line like so: > + another location in the ovsdb like: > > - `./vswitchd/ovs-vswitchd --dpdk -vhost_sock_dir /my-dir -c 0x1 ...` > + `./ovs-vsctl set Open_vSwitch . > + other_config:vhost_sock_dir=location` > > DPDK vhost-user VM configuration: > --------------------------------- > @@ -641,11 +665,11 @@ DPDK vhost-cuse VM configuration: > The new character device filename must be specified on the vswitchd > commandline: > > - `./vswitchd/ovs-vswitchd --dpdk --cuse_dev_name my-vhost-net -c 0x1 > ...` > + `./utilities/ovs-vsctl set Open_vSwitch . \ > + other_config:cuse_dev_name=my-vhost-net ` > > - Note that the `--cuse_dev_name` argument and associated string must be > the first > - arguments after `--dpdk` and come before the EAL arguments. In the > example > - above, the character device to be used will be `/dev/my-vhost-net`. > + In the example above, the character device to be used will > + be `/dev/my-vhost-net`. > > 2. This step is only needed if reusing the standard character device. It will > conflict with the kernel vhost character device so the user must first > -- > 2.6.1.133.gf5b6079 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev
"Gray, Mark D" <mark.d.gray@intel.com> writes: >> -----Original Message----- >> From: dev [mailto:dev-bounces@openvswitch.org] On Behalf Of Aaron >> Conole >> Sent: Thursday, December 3, 2015 4:24 AM >> To: dev@openvswitch.org >> Cc: Flavio Leitner >> Subject: [ovs-dev] [RFC 2/2] dpdk: Update INSTALL.DPDK.md to reflect new >> config >> >> A previous change modified the way DPDK configuration would be accepted >> by the system. This updates the documentation to align with the new >> method of configuring DPDK. >> >> Signed-off-by: Aaron Conole <aconole@redhat.com> >> --- >> INSTALL.DPDK.md | 52 ++++++++++++++++++++++++++++++++++++++---- >> ---------- >> 1 file changed, 38 insertions(+), 14 deletions(-) >> >> diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 96b686c..789ea68 >> 100644 >> --- a/INSTALL.DPDK.md >> +++ b/INSTALL.DPDK.md >> @@ -143,22 +143,46 @@ Using the DPDK with ovs-vswitchd: >> > [Gray, Mark D] > On a general note, I guess for most of these database entries it > doesn’t make sense to > change them at runtime as they only affect the initialization > stage. Off the top of my > head, I can't think of any other entries that behave like this > (perhaps there are > though). Does this mean it would be possible to change them at runtime > which would make the switch look like it was configured in a different way? > Take the number of memory channels as an example. It may not be a problem > but it is strange behavior. I wrote this approach for two reasons after reading the following: http://openvswitch.org/pipermail/dev/2015-December/062859.html 1. "OTOH the command line variant is perhaps less effort." - I wanted to see how much effort would really be required to do this. It wasn't much to put a strawman together, so here we are. 2. Ben's reply to another similar component: " >> It would seem to make sense to configure DPDK the same way as everything >> else in OVS: through the database. Sounds like what we want. " Why delay the inevitable? :) There are CONs to this approach, no doubt. As you've said there are possibilities for things to go out of sync; that's a good feature to add to this series - ensure that values haven't changed. Possibly we could have a graceful restart facility if someone does change. Not sure how it will look. Anyway, it's the kind of discussion that I want for this change, so I'm hoping others also chime in with thoughts. >> 5. Start vswitchd: >> >> - DPDK configuration arguments can be passed to vswitchd via `--dpdk` >> - argument. This needs to be first argument passed to vswitchd process. >> - dpdk arg -c is ignored by ovs-dpdk, but it is a required parameter >> - for dpdk initialization. >> - >> + DPDK configuration arguments can be passed to vswitchd via the >> Open_vSwitch >> + other_config database. The recognized configuration options are listed. >> + >> + * dpdk >> + This is a boolean configuration option. A value of 'true' tells >> + Open_vSwitch to initialize the DPDK EAL. A set of nominal defaults are >> + provided so that simply enabling this option will be sufficient to >> configure >> + DPDK enabled ports. >> + * dpdk_core_mask >> + This is DPDK's -c command line option, and specifies the core mask to >> + the EAL. > > Why exactly do we need this again? We have "other_config:pmd-cpu-mask" which > should set the affinity of the pmd threads. The affinities of the > other standard pthreads > should by a system administrator. Or, if the intention is that this > would allow us > to set affinities for the other threads, then the name should be > changed to something > more meaningful. I think we don't need a new config and pmd-cpu-mask is the correct value to use for this configuration. I'll make the change when I spin a formal patch. Thanks for the review, Mark!
> > Why delay the inevitable? :) > Absolutely, I agree with the approach. I was just pointing out that issue.
On 04/12/15 16:15, Gray, Mark D wrote: >> 5. Start vswitchd: >> > >> >- DPDK configuration arguments can be passed to vswitchd via `--dpdk` >> >- argument. This needs to be first argument passed to vswitchd process. >> >- dpdk arg -c is ignored by ovs-dpdk, but it is a required parameter >> >- for dpdk initialization. >> >- >> >+ DPDK configuration arguments can be passed to vswitchd via the >> >Open_vSwitch >> >+ other_config database. The recognized configuration options are listed. >> >+ >> >+ * dpdk >> >+ This is a boolean configuration option. A value of 'true' tells >> >+ Open_vSwitch to initialize the DPDK EAL. A set of nominal defaults are >> >+ provided so that simply enabling this option will be sufficient to configure >> >+ DPDK enabled ports. >> >+ * dpdk_core_mask >> >+ This is DPDK's -c command line option, and specifies the core mask to >> >+ the EAL. > Why exactly do we need this again? We have "other_config:pmd-cpu-mask" which > should set the affinity of the pmd threads. The affinities of the other standard pthreads > should by a system administrator. Or, if the intention is that this would allow us > to set affinities for the other threads, then the name should be changed to something > more meaningful. > My understanding is that OVS-DPDK doesn't really use any "lcore"'s from DPDK, but rather creates its own threads, and set the lcore_id in pmd_thread_setaffinity_cpu(). At least, I haven't seen any rte_eal_[mp_]remote_launch() call in the code, so I think the -c option should only have 1 core set for the master thread, everything else just waste memory.
> -----Original Message----- > From: dev [mailto:dev-bounces@openvswitch.org] On Behalf Of Zoltan Kiss > Sent: Monday, December 7, 2015 11:57 AM > To: Gray, Mark D; Aaron Conole; dev@openvswitch.org > Cc: Flavio Leitner > Subject: Re: [ovs-dev] [RFC 2/2] dpdk: Update INSTALL.DPDK.md to reflect new > config > > > > On 04/12/15 16:15, Gray, Mark D wrote: > >> 5. Start vswitchd: > >> > > >> >- DPDK configuration arguments can be passed to vswitchd via `--dpdk` > >> >- argument. This needs to be first argument passed to vswitchd process. > >> >- dpdk arg -c is ignored by ovs-dpdk, but it is a required parameter > >> >- for dpdk initialization. > >> >- > >> >+ DPDK configuration arguments can be passed to vswitchd via the > >> >Open_vSwitch > >> >+ other_config database. The recognized configuration options are > listed. > >> >+ > >> >+ * dpdk > >> >+ This is a boolean configuration option. A value of 'true' tells > >> >+ Open_vSwitch to initialize the DPDK EAL. A set of nominal defaults > are > >> >+ provided so that simply enabling this option will be sufficient to > configure > >> >+ DPDK enabled ports. > >> >+ * dpdk_core_mask > >> >+ This is DPDK's -c command line option, and specifies the core mask to > >> >+ the EAL. > > Why exactly do we need this again? We have "other_config:pmd-cpu-mask" > which > > should set the affinity of the pmd threads. The affinities of the other > standard pthreads > > should by a system administrator. Or, if the intention is that this would > allow us > > to set affinities for the other threads, then the name should be changed to > something > > more meaningful. > > > > My understanding is that OVS-DPDK doesn't really use any "lcore"'s from > DPDK, but rather creates its own threads, and set the lcore_id in > pmd_thread_setaffinity_cpu(). At least, I haven't seen any > rte_eal_[mp_]remote_launch() call in the code, so I think the -c option > should only have 1 core set for the master thread, everything else just > waste memory. I agree. At present I see it only uses the lowest core in the -c option as the master thread. Aside from removing the command line args, I think we need to up level the docs so that an OVS user knows what the -c does and the (lack of) relationship with pmd-cpu-mask. I've seen a few examples recently where there is confusion between them. I can put in a patch to give a brief description in the docs for the short term, and we can keep this in mind for a bigger change like Aaron's. > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev
diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 96b686c..789ea68 100644 --- a/INSTALL.DPDK.md +++ b/INSTALL.DPDK.md @@ -143,22 +143,46 @@ Using the DPDK with ovs-vswitchd: 5. Start vswitchd: - DPDK configuration arguments can be passed to vswitchd via `--dpdk` - argument. This needs to be first argument passed to vswitchd process. - dpdk arg -c is ignored by ovs-dpdk, but it is a required parameter - for dpdk initialization. - + DPDK configuration arguments can be passed to vswitchd via the Open_vSwitch + other_config database. The recognized configuration options are listed. + + * dpdk + This is a boolean configuration option. A value of 'true' tells + Open_vSwitch to initialize the DPDK EAL. A set of nominal defaults are + provided so that simply enabling this option will be sufficient to configure + DPDK enabled ports. + * dpdk_core_mask + This is DPDK's -c command line option, and specifies the core mask to + the EAL. + * dpdk_mem_channels + This is DPDK's -n option, to configure the number of memory spread channels + into the CPU. It is purely an optimization flag. + * dpdk_alloc_mem + Total memory to pre-allocate + * dpdk_socket_mem + Memory to allocate on specific sockets (use comma separated values) + * dpdk_hugepage_dir + Directory where hugetlbfs is mounted + * cuse_dev_name + Option to set the vhost_cuse character device name. + * vhost_sock_dir + Specify a custom directory for the vhost_user unix socket files. + + Note that changing any of these options requires restarting the ovs-vswitchd + application. + ``` export DB_SOCK=/usr/local/var/run/openvswitch/db.sock - ovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK --pidfile --detach + ovs-vsctl set Open_vSwitch . other_config:dpdk=true + ovs-vswitchd unix:$DB_SOCK --pidfile --detach ``` If allocated more than one GB hugepage (as for IVSHMEM), set amount and use NUMA node 0 memory: ``` - ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 1024,0 \ - -- unix:$DB_SOCK --pidfile --detach + ovs-vsctl set Open_vSwitch . other_config:dpdk_socket_mem="1024,0" + ovs-vswitchd unix:$DB_SOCK --pidfile --detach ``` 6. Add bridge & ports @@ -523,9 +547,9 @@ have arbitrary names. found in the next section "DPDK vhost-user VM configuration" Note: If you wish for the vhost-user sockets to be created in a directory other than `/usr/local/var/run/openvswitch`, you may specify - another location on the ovs-vswitchd command line like so: + another location in the ovsdb like: - `./vswitchd/ovs-vswitchd --dpdk -vhost_sock_dir /my-dir -c 0x1 ...` + `./ovs-vsctl set Open_vSwitch . other_config:vhost_sock_dir=location` DPDK vhost-user VM configuration: --------------------------------- @@ -641,11 +665,11 @@ DPDK vhost-cuse VM configuration: The new character device filename must be specified on the vswitchd commandline: - `./vswitchd/ovs-vswitchd --dpdk --cuse_dev_name my-vhost-net -c 0x1 ...` + `./utilities/ovs-vsctl set Open_vSwitch . \ + other_config:cuse_dev_name=my-vhost-net ` - Note that the `--cuse_dev_name` argument and associated string must be the first - arguments after `--dpdk` and come before the EAL arguments. In the example - above, the character device to be used will be `/dev/my-vhost-net`. + In the example above, the character device to be used will + be `/dev/my-vhost-net`. 2. This step is only needed if reusing the standard character device. It will conflict with the kernel vhost character device so the user must first
A previous change modified the way DPDK configuration would be accepted by the system. This updates the documentation to align with the new method of configuring DPDK. Signed-off-by: Aaron Conole <aconole@redhat.com> --- INSTALL.DPDK.md | 52 ++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 38 insertions(+), 14 deletions(-)