| Message ID | 1512730427-34577-2-git-send-email-mark.b.kavanagh@intel.com |
|---|---|
| State | Accepted |
| Delegated to: | Ian Stokes |
| Headers | show |
| Series | DPDK v17.11 Support | expand |
On 12/08/2017 10:53 AM, Mark Kavanagh wrote: > This commit adds support for DPDK v17.11: > - minor updates to accomodate DPDK API changes > - update references to DPDK version in Documentation > - update DPDK version in travis' linux-build script > - document DPDK v17.11 virtio driver bug > > Signed-off-by: Mark Kavanagh <mark.b.kavanagh@intel.com> > Acked-by: Maxime Coquelin <maxime.coquelin@redhat.com> > Acked-by: Ciara Loftus <ciara.loftus@intel.com> > Acked-by: Jan Scheurich <jan.scheurich@ericsson.com> > Tested-by: Jan Scheurich <jan.scheurich@ericsson.com> > Tested-by: Guoshuai Li <ligs@dtdream.com> > --- Acked-by: Kevin Traynor <ktraynor@redhat.com>
> This commit adds support for DPDK v17.11: > - minor updates to accomodate DPDK API changes > - update references to DPDK version in Documentation > - update DPDK version in travis' linux-build script > - document DPDK v17.11 virtio driver bug > > Signed-off-by: Mark Kavanagh <mark.b.kavanagh@intel.com> > Acked-by: Maxime Coquelin <maxime.coquelin@redhat.com> > Acked-by: Ciara Loftus <ciara.loftus@intel.com> > Acked-by: Jan Scheurich <jan.scheurich@ericsson.com> > Tested-by: Jan Scheurich <jan.scheurich@ericsson.com> > Tested-by: Guoshuai Li <ligs@dtdream.com> Thanks all for the work on this patchset, We seem to have reached a consensus so I'll queue this to the DPDK_Merge branch and it will be part of today's pull request. Ian > --- > .travis/linux-build.sh | 2 +- > Documentation/faq/releases.rst | 1 + > Documentation/intro/install/dpdk.rst | 10 +++++----- > Documentation/topics/dpdk/ring.rst | 2 +- > Documentation/topics/dpdk/vhost-user.rst | 32 > +++++++++++++++++++++++++++----- > NEWS | 2 ++ > lib/netdev-dpdk.c | 5 +++-- > 7 files changed, 40 insertions(+), 14 deletions(-) > > diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh index > 4d6459f..ed28ee4 100755 > --- a/.travis/linux-build.sh > +++ b/.travis/linux-build.sh > @@ -81,7 +81,7 @@ fi > > if [ "$DPDK" ]; then > if [ -z "$DPDK_VER" ]; then > - DPDK_VER="17.05.2" > + DPDK_VER="17.11" > fi > install_dpdk $DPDK_VER > if [ "$CC" = "clang" ]; then > diff --git a/Documentation/faq/releases.rst > b/Documentation/faq/releases.rst index d903b06..62a1957 100644 > --- a/Documentation/faq/releases.rst > +++ b/Documentation/faq/releases.rst > @@ -164,6 +164,7 @@ Q: What DPDK version does each Open vSwitch release > work with? > 2.6.x 16.07.2 > 2.7.x 16.11.3 > 2.8.x 17.05.2 > + 2.9.x 17.11 > ============ ======= > > Q: I get an error like this when I configure Open vSwitch: > diff --git a/Documentation/intro/install/dpdk.rst > b/Documentation/intro/install/dpdk.rst > index bb69ae5..3fecb5c 100644 > --- a/Documentation/intro/install/dpdk.rst > +++ b/Documentation/intro/install/dpdk.rst > @@ -40,7 +40,7 @@ Build requirements > In addition to the requirements described in :doc:`general`, building > Open vSwitch with DPDK will require the following: > > -- DPDK 17.05.2 > +- DPDK 17.11 > > - A `DPDK supported NIC`_ > > @@ -69,9 +69,9 @@ Install DPDK > #. Download the `DPDK sources`_, extract the file and set ``DPDK_DIR``:: > > $ cd /usr/src/ > - $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz > - $ tar xf dpdk-17.05.2.tar.xz > - $ export DPDK_DIR=/usr/src/dpdk-stable-17.05.2 > + $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz > + $ tar xf dpdk-17.11.tar.xz > + $ export DPDK_DIR=/usr/src/dpdk-17.11 > $ cd $DPDK_DIR > > #. (Optional) Configure DPDK as a shared library @@ -583,7 +583,7 @@ > Limitations > The latest list of validated firmware versions can be found in the > `DPDK > release notes`_. > > -.. _DPDK release notes: > http://dpdk.org/doc/guides/rel_notes/release_17_05.html > +.. _DPDK release notes: > +http://dpdk.org/doc/guides/rel_notes/release_17_11.html > > Reporting Bugs > -------------- > diff --git a/Documentation/topics/dpdk/ring.rst > b/Documentation/topics/dpdk/ring.rst > index ad9d7a5..8d0ede8 100644 > --- a/Documentation/topics/dpdk/ring.rst > +++ b/Documentation/topics/dpdk/ring.rst > @@ -77,4 +77,4 @@ DPDK. However, this functionality was removed because: > - :doc:`vhost-user interfaces <vhost-user>` are the defacto DPDK-based > path to > guests > > -.. _DPDK documentation: > https://dpdk.readthedocs.io/en/v17.05/prog_guide/ring_lib.html > +.. _DPDK documentation: > +https://dpdk.readthedocs.io/en/v17.11/prog_guide/ring_lib.html > diff --git a/Documentation/topics/dpdk/vhost-user.rst > b/Documentation/topics/dpdk/vhost-user.rst > index 74ac06e..a43affa 100644 > --- a/Documentation/topics/dpdk/vhost-user.rst > +++ b/Documentation/topics/dpdk/vhost-user.rst > @@ -292,9 +292,9 @@ To begin, instantiate a guest as described in > :ref:`dpdk-vhost-user` or DPDK sources to VM and build DPDK:: > > $ cd /root/dpdk/ > - $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz > - $ tar xf dpdk-17.05.2.tar.xz > - $ export DPDK_DIR=/root/dpdk/dpdk-stable-17.05.2 > + $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz > + $ tar xf dpdk-17.11.tar.xz > + $ export DPDK_DIR=/root/dpdk/dpdk-17.11 > $ export DPDK_TARGET=x86_64-native-linuxapp-gcc > $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET > $ cd $DPDK_DIR > @@ -327,6 +327,28 @@ Finally, start the application:: > > # TODO > > +.. important:: > + > + DPDK v17.11 virtio PMD contains a bug in the vectorized Rx function > + that affects testpmd/DPDK guest applications. As such, guest DPDK > + applications should use a non-vectorized Rx function. > + > +The DPDK v17.11 virtio net driver contains a bug that prevents guest > +DPDK applications from receiving packets when the vectorized Rx function > is used. > +This only occurs when guest-bound traffic is live before a DPDK > +application is started within the guest, and where two or more > +forwarding cores are used. As such, it is not recommended for guests > +which execute DPDK applications to use the virtio vectorized Rx > +function. A simple method of ensuring that a non- vectorized Rx > +function is used is to enable mergeable buffers for the guest, with the > following QEMU command line option:: > + > + mrg_rxbuf=on > + > +Additional details regarding the virtio driver bug are available on the > +`DPDK mailing list`_. > + > +.. _DPDK mailing list: > +http://dpdk.org/ml/archives/dev/2017-December/082801.html > + > .. _dpdk-vhost-user-xml: > > Sample XML > @@ -387,7 +409,7 @@ Sample XML > <source type='unix' > path='/usr/local/var/run/openvswitch/dpdkvhostuser0' mode='client'/> > <model type='virtio'/> > <driver queues='2'> > - <host mrg_rxbuf='off'/> > + <host mrg_rxbuf='on'/> > </driver> > </interface> > <interface type='vhostuser'> > @@ -395,7 +417,7 @@ Sample XML > <source type='unix' > path='/usr/local/var/run/openvswitch/dpdkvhostuser1' mode='client'/> > <model type='virtio'/> > <driver queues='2'> > - <host mrg_rxbuf='off'/> > + <host mrg_rxbuf='on'/> > </driver> > </interface> > <serial type='pty'> > diff --git a/NEWS b/NEWS > index 427c8f8..d4a1c9a 100644 > --- a/NEWS > +++ b/NEWS > @@ -13,6 +13,8 @@ Post-v2.8.0 > * ovn-ctl: New commands run_nb_ovsdb and run_sb_ovsdb. > - Linux kernel 4.13 > * Add support for compiling OVS with the latest Linux 4.13 kernel > + - DPDK: > + * Add support for DPDK v17.11 > > v2.8.0 - 31 Aug 2017 > -------------------- > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index faff842..f552444 > 100644 > --- a/lib/netdev-dpdk.c > +++ b/lib/netdev-dpdk.c > @@ -26,6 +26,7 @@ > #include <sys/socket.h> > #include <linux/if.h> > > +#include <rte_bus_pci.h> > #include <rte_config.h> > #include <rte_cycles.h> > #include <rte_errno.h> > @@ -140,8 +141,8 @@ static struct vlog_rate_limit rl = > VLOG_RATE_LIMIT_INIT(5, 20); > > #define DPDK_ETH_PORT_ID_INVALID RTE_MAX_ETHPORTS > > -/* DPDK library uses uint8_t for port_id. */ -typedef uint8_t > dpdk_port_t; > +/* DPDK library uses uint16_t for port_id. */ typedef uint16_t > +dpdk_port_t; > > #define VHOST_ENQ_RETRY_NUM 8 > #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ) > -- > 1.9.3
Sorry for the late comments. I'm still not happy with the proposed documentation changes (see below). /Jan > -----Original Message----- > From: Mark Kavanagh [mailto:mark.b.kavanagh@intel.com] > Sent: Friday, 08 December, 2017 11:54 > To: dev@openvswitch.org > Cc: ktraynor@redhat.com; maxime.coquelin@redhat.com; i.maximets@samsung.com; Jan Scheurich <jan.scheurich@ericsson.com>; > sean.k.mooney@intel.com; ian.stokes@intel.com; ligs@dtdream.com; antonio.fischetti@intel.com; tiwei.bie@intel.com; > ciara.loftus@intel.com; Mark Kavanagh <mark.b.kavanagh@intel.com> > Subject: [ovs-dev][PATCH V6 1/2] netdev-dpdk: DPDK v17.11 upgrade > > This commit adds support for DPDK v17.11: > - minor updates to accomodate DPDK API changes > - update references to DPDK version in Documentation > - update DPDK version in travis' linux-build script > - document DPDK v17.11 virtio driver bug > > Signed-off-by: Mark Kavanagh <mark.b.kavanagh@intel.com> > Acked-by: Maxime Coquelin <maxime.coquelin@redhat.com> > Acked-by: Ciara Loftus <ciara.loftus@intel.com> > Acked-by: Jan Scheurich <jan.scheurich@ericsson.com> > Tested-by: Jan Scheurich <jan.scheurich@ericsson.com> > Tested-by: Guoshuai Li <ligs@dtdream.com> > --- > .travis/linux-build.sh | 2 +- > Documentation/faq/releases.rst | 1 + > Documentation/intro/install/dpdk.rst | 10 +++++----- > Documentation/topics/dpdk/ring.rst | 2 +- > Documentation/topics/dpdk/vhost-user.rst | 32 +++++++++++++++++++++++++++----- > NEWS | 2 ++ > lib/netdev-dpdk.c | 5 +++-- > 7 files changed, 40 insertions(+), 14 deletions(-) > > diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh > index 4d6459f..ed28ee4 100755 > --- a/.travis/linux-build.sh > +++ b/.travis/linux-build.sh > @@ -81,7 +81,7 @@ fi > > if [ "$DPDK" ]; then > if [ -z "$DPDK_VER" ]; then > - DPDK_VER="17.05.2" > + DPDK_VER="17.11" > fi > install_dpdk $DPDK_VER > if [ "$CC" = "clang" ]; then > diff --git a/Documentation/faq/releases.rst b/Documentation/faq/releases.rst > index d903b06..62a1957 100644 > --- a/Documentation/faq/releases.rst > +++ b/Documentation/faq/releases.rst > @@ -164,6 +164,7 @@ Q: What DPDK version does each Open vSwitch release work with? > 2.6.x 16.07.2 > 2.7.x 16.11.3 > 2.8.x 17.05.2 > + 2.9.x 17.11 > ============ ======= > > Q: I get an error like this when I configure Open vSwitch: > diff --git a/Documentation/intro/install/dpdk.rst b/Documentation/intro/install/dpdk.rst > index bb69ae5..3fecb5c 100644 > --- a/Documentation/intro/install/dpdk.rst > +++ b/Documentation/intro/install/dpdk.rst > @@ -40,7 +40,7 @@ Build requirements > In addition to the requirements described in :doc:`general`, building Open > vSwitch with DPDK will require the following: > > -- DPDK 17.05.2 > +- DPDK 17.11 > > - A `DPDK supported NIC`_ > > @@ -69,9 +69,9 @@ Install DPDK > #. Download the `DPDK sources`_, extract the file and set ``DPDK_DIR``:: > > $ cd /usr/src/ > - $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz > - $ tar xf dpdk-17.05.2.tar.xz > - $ export DPDK_DIR=/usr/src/dpdk-stable-17.05.2 > + $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz > + $ tar xf dpdk-17.11.tar.xz > + $ export DPDK_DIR=/usr/src/dpdk-17.11 > $ cd $DPDK_DIR > > #. (Optional) Configure DPDK as a shared library > @@ -583,7 +583,7 @@ Limitations > The latest list of validated firmware versions can be found in the `DPDK > release notes`_. > > -.. _DPDK release notes: http://dpdk.org/doc/guides/rel_notes/release_17_05.html > +.. _DPDK release notes: http://dpdk.org/doc/guides/rel_notes/release_17_11.html > What happened to my request to add here that running OVS 2.9 with DPDK datapath in a VM with virtio interfaces may itself suffer from the DPDK 17.11 bug in OVS' own virtio PMD? The work-around for this could be to enable the mrg_rxbuf feature in Qemu for the OVS VM as that would trigger OVS to not use vector rx. This is essentially different from the note in the vhost_user section below that talks about DPDK apps running in a guest VM served by OVS. > Reporting Bugs > -------------- > diff --git a/Documentation/topics/dpdk/ring.rst b/Documentation/topics/dpdk/ring.rst > index ad9d7a5..8d0ede8 100644 > --- a/Documentation/topics/dpdk/ring.rst > +++ b/Documentation/topics/dpdk/ring.rst > @@ -77,4 +77,4 @@ DPDK. However, this functionality was removed because: > - :doc:`vhost-user interfaces <vhost-user>` are the defacto DPDK-based path to > guests > > -.. _DPDK documentation: https://dpdk.readthedocs.io/en/v17.05/prog_guide/ring_lib.html > +.. _DPDK documentation: https://dpdk.readthedocs.io/en/v17.11/prog_guide/ring_lib.html > diff --git a/Documentation/topics/dpdk/vhost-user.rst b/Documentation/topics/dpdk/vhost-user.rst > index 74ac06e..a43affa 100644 > --- a/Documentation/topics/dpdk/vhost-user.rst > +++ b/Documentation/topics/dpdk/vhost-user.rst > @@ -292,9 +292,9 @@ To begin, instantiate a guest as described in :ref:`dpdk-vhost-user` or > DPDK sources to VM and build DPDK:: > > $ cd /root/dpdk/ > - $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz > - $ tar xf dpdk-17.05.2.tar.xz > - $ export DPDK_DIR=/root/dpdk/dpdk-stable-17.05.2 > + $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz > + $ tar xf dpdk-17.11.tar.xz > + $ export DPDK_DIR=/root/dpdk/dpdk-17.11 > $ export DPDK_TARGET=x86_64-native-linuxapp-gcc > $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET > $ cd $DPDK_DIR > @@ -327,6 +327,28 @@ Finally, start the application:: > > # TODO > > +.. important:: > + > + DPDK v17.11 virtio PMD contains a bug in the vectorized Rx function that > + affects testpmd/DPDK guest applications. As such, guest DPDK applications > + should use a non-vectorized Rx function. > + > +The DPDK v17.11 virtio net driver contains a bug that prevents guest DPDK > +applications from receiving packets when the vectorized Rx function is used. > +This only occurs when guest-bound traffic is live before a DPDK application is > +started within the guest, and where two or more forwarding cores are used. As > +such, it is not recommended for guests which execute DPDK applications to use > +the virtio vectorized Rx function. A simple method of ensuring that a non- > +vectorized Rx function is used is to enable mergeable buffers for the guest, > +with the following QEMU command line option:: > + > + mrg_rxbuf=on For an arbitrary DPDK application in the guest that may not solve the issue! Where or not mergeable Rx buffers are negotiated to be used depends on two things: 1. Support enabled in Qemu, 2. Support signaled in the guest virtio driver and 3. Support signaled in the vhostuser backend. If the application e.g. for performance reasons doesn't want to use mergeable Rx buffers, it could still negotiate to off and apply the broken vector rx routine. > + > +Additional details regarding the virtio driver bug are available on the > +`DPDK mailing list`_. > + > +.. _DPDK mailing list: http://dpdk.org/ml/archives/dev/2017-December/082801.html > + > .. _dpdk-vhost-user-xml: > > Sample XML > @@ -387,7 +409,7 @@ Sample XML > <source type='unix' path='/usr/local/var/run/openvswitch/dpdkvhostuser0' mode='client'/> > <model type='virtio'/> > <driver queues='2'> > - <host mrg_rxbuf='off'/> > + <host mrg_rxbuf='on'/> Are you changing this example because of the current bug? Or is there some other reason? I don't think the temporary bug in DPDK 17.11 justifies to change the example VM XML to enable mrg_rxbuf. This costs performance in OVS and DPDK applications should use it only when needed (e.g. because jumbo frames must be received fragmented into smaller mbufs). > </driver> > </interface> > <interface type='vhostuser'> > @@ -395,7 +417,7 @@ Sample XML > <source type='unix' path='/usr/local/var/run/openvswitch/dpdkvhostuser1' mode='client'/> > <model type='virtio'/> > <driver queues='2'> > - <host mrg_rxbuf='off'/> > + <host mrg_rxbuf='on'/> > </driver> > </interface> > <serial type='pty'> > diff --git a/NEWS b/NEWS > index 427c8f8..d4a1c9a 100644 > --- a/NEWS > +++ b/NEWS > @@ -13,6 +13,8 @@ Post-v2.8.0 > * ovn-ctl: New commands run_nb_ovsdb and run_sb_ovsdb. > - Linux kernel 4.13 > * Add support for compiling OVS with the latest Linux 4.13 kernel > + - DPDK: > + * Add support for DPDK v17.11 > > v2.8.0 - 31 Aug 2017 > -------------------- > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c > index faff842..f552444 100644 > --- a/lib/netdev-dpdk.c > +++ b/lib/netdev-dpdk.c > @@ -26,6 +26,7 @@ > #include <sys/socket.h> > #include <linux/if.h> > > +#include <rte_bus_pci.h> > #include <rte_config.h> > #include <rte_cycles.h> > #include <rte_errno.h> > @@ -140,8 +141,8 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 20); > > #define DPDK_ETH_PORT_ID_INVALID RTE_MAX_ETHPORTS > > -/* DPDK library uses uint8_t for port_id. */ > -typedef uint8_t dpdk_port_t; > +/* DPDK library uses uint16_t for port_id. */ > +typedef uint16_t dpdk_port_t; > > #define VHOST_ENQ_RETRY_NUM 8 > #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ) > -- > 1.9.3
>From: Jan Scheurich [mailto:jan.scheurich@ericsson.com] >Sent: Friday, December 8, 2017 1:22 PM >To: Kavanagh, Mark B <mark.b.kavanagh@intel.com>; dev@openvswitch.org >Cc: ktraynor@redhat.com; maxime.coquelin@redhat.com; i.maximets@samsung.com; >Mooney, Sean K <sean.k.mooney@intel.com>; Stokes, Ian <ian.stokes@intel.com>; >ligs@dtdream.com; Fischetti, Antonio <antonio.fischetti@intel.com>; Bie, Tiwei ><tiwei.bie@intel.com>; Loftus, Ciara <ciara.loftus@intel.com> >Subject: RE: [ovs-dev][PATCH V6 1/2] netdev-dpdk: DPDK v17.11 upgrade > >Sorry for the late comments. >I'm still not happy with the proposed documentation changes (see below). > >/Jan Hi Jan, Thanks for your feedback - responses inline as usual. Best, Mark > >> -----Original Message----- >> From: Mark Kavanagh [mailto:mark.b.kavanagh@intel.com] >> Sent: Friday, 08 December, 2017 11:54 >> To: dev@openvswitch.org >> Cc: ktraynor@redhat.com; maxime.coquelin@redhat.com; i.maximets@samsung.com; >Jan Scheurich <jan.scheurich@ericsson.com>; >> sean.k.mooney@intel.com; ian.stokes@intel.com; ligs@dtdream.com; >antonio.fischetti@intel.com; tiwei.bie@intel.com; >> ciara.loftus@intel.com; Mark Kavanagh <mark.b.kavanagh@intel.com> >> Subject: [ovs-dev][PATCH V6 1/2] netdev-dpdk: DPDK v17.11 upgrade >> >> This commit adds support for DPDK v17.11: >> - minor updates to accomodate DPDK API changes >> - update references to DPDK version in Documentation >> - update DPDK version in travis' linux-build script >> - document DPDK v17.11 virtio driver bug >> >> Signed-off-by: Mark Kavanagh <mark.b.kavanagh@intel.com> >> Acked-by: Maxime Coquelin <maxime.coquelin@redhat.com> >> Acked-by: Ciara Loftus <ciara.loftus@intel.com> >> Acked-by: Jan Scheurich <jan.scheurich@ericsson.com> >> Tested-by: Jan Scheurich <jan.scheurich@ericsson.com> >> Tested-by: Guoshuai Li <ligs@dtdream.com> >> --- >> .travis/linux-build.sh | 2 +- >> Documentation/faq/releases.rst | 1 + >> Documentation/intro/install/dpdk.rst | 10 +++++----- >> Documentation/topics/dpdk/ring.rst | 2 +- >> Documentation/topics/dpdk/vhost-user.rst | 32 +++++++++++++++++++++++++++-- >--- >> NEWS | 2 ++ >> lib/netdev-dpdk.c | 5 +++-- >> 7 files changed, 40 insertions(+), 14 deletions(-) >> >> diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh >> index 4d6459f..ed28ee4 100755 >> --- a/.travis/linux-build.sh >> +++ b/.travis/linux-build.sh >> @@ -81,7 +81,7 @@ fi >> >> if [ "$DPDK" ]; then >> if [ -z "$DPDK_VER" ]; then >> - DPDK_VER="17.05.2" >> + DPDK_VER="17.11" >> fi >> install_dpdk $DPDK_VER >> if [ "$CC" = "clang" ]; then >> diff --git a/Documentation/faq/releases.rst b/Documentation/faq/releases.rst >> index d903b06..62a1957 100644 >> --- a/Documentation/faq/releases.rst >> +++ b/Documentation/faq/releases.rst >> @@ -164,6 +164,7 @@ Q: What DPDK version does each Open vSwitch release work >with? >> 2.6.x 16.07.2 >> 2.7.x 16.11.3 >> 2.8.x 17.05.2 >> + 2.9.x 17.11 >> ============ ======= >> >> Q: I get an error like this when I configure Open vSwitch: >> diff --git a/Documentation/intro/install/dpdk.rst >b/Documentation/intro/install/dpdk.rst >> index bb69ae5..3fecb5c 100644 >> --- a/Documentation/intro/install/dpdk.rst >> +++ b/Documentation/intro/install/dpdk.rst >> @@ -40,7 +40,7 @@ Build requirements >> In addition to the requirements described in :doc:`general`, building Open >> vSwitch with DPDK will require the following: >> >> -- DPDK 17.05.2 >> +- DPDK 17.11 >> >> - A `DPDK supported NIC`_ >> >> @@ -69,9 +69,9 @@ Install DPDK >> #. Download the `DPDK sources`_, extract the file and set ``DPDK_DIR``:: >> >> $ cd /usr/src/ >> - $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz >> - $ tar xf dpdk-17.05.2.tar.xz >> - $ export DPDK_DIR=/usr/src/dpdk-stable-17.05.2 >> + $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz >> + $ tar xf dpdk-17.11.tar.xz >> + $ export DPDK_DIR=/usr/src/dpdk-17.11 >> $ cd $DPDK_DIR >> >> #. (Optional) Configure DPDK as a shared library >> @@ -583,7 +583,7 @@ Limitations >> The latest list of validated firmware versions can be found in the `DPDK >> release notes`_. >> >> -.. _DPDK release notes: >http://dpdk.org/doc/guides/rel_notes/release_17_05.html >> +.. _DPDK release notes: >http://dpdk.org/doc/guides/rel_notes/release_17_11.html >> > >What happened to my request to add here that running OVS 2.9 with DPDK >datapath in a VM with virtio interfaces may itself suffer from the DPDK 17.11 >bug in OVS' own virtio PMD? The work-around for this could be to enable the >mrg_rxbuf feature in Qemu for the OVS VM as that would trigger OVS to not use >vector rx. > >This is essentially different from the note in the vhost_user section below >that talks about DPDK apps running in a guest VM served by OVS. I see - I actually didn't get that from your initial mail, but your point is clear now. Technically, OvS is still a DPDK app in the guest using the virtio PMD, so it would fall under the caveats in this patch regarding guest DPDK apps. However, I see no harm in calling out OvS in the guest specifically. > >> Reporting Bugs >> -------------- >> diff --git a/Documentation/topics/dpdk/ring.rst >b/Documentation/topics/dpdk/ring.rst >> index ad9d7a5..8d0ede8 100644 >> --- a/Documentation/topics/dpdk/ring.rst >> +++ b/Documentation/topics/dpdk/ring.rst >> @@ -77,4 +77,4 @@ DPDK. However, this functionality was removed because: >> - :doc:`vhost-user interfaces <vhost-user>` are the defacto DPDK-based path >to >> guests >> >> -.. _DPDK documentation: >https://dpdk.readthedocs.io/en/v17.05/prog_guide/ring_lib.html >> +.. _DPDK documentation: >https://dpdk.readthedocs.io/en/v17.11/prog_guide/ring_lib.html >> diff --git a/Documentation/topics/dpdk/vhost-user.rst >b/Documentation/topics/dpdk/vhost-user.rst >> index 74ac06e..a43affa 100644 >> --- a/Documentation/topics/dpdk/vhost-user.rst >> +++ b/Documentation/topics/dpdk/vhost-user.rst >> @@ -292,9 +292,9 @@ To begin, instantiate a guest as described in >:ref:`dpdk-vhost-user` or >> DPDK sources to VM and build DPDK:: >> >> $ cd /root/dpdk/ >> - $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz >> - $ tar xf dpdk-17.05.2.tar.xz >> - $ export DPDK_DIR=/root/dpdk/dpdk-stable-17.05.2 >> + $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz >> + $ tar xf dpdk-17.11.tar.xz >> + $ export DPDK_DIR=/root/dpdk/dpdk-17.11 >> $ export DPDK_TARGET=x86_64-native-linuxapp-gcc >> $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET >> $ cd $DPDK_DIR >> @@ -327,6 +327,28 @@ Finally, start the application:: >> >> # TODO >> >> +.. important:: >> + >> + DPDK v17.11 virtio PMD contains a bug in the vectorized Rx function that >> + affects testpmd/DPDK guest applications. As such, guest DPDK applications >> + should use a non-vectorized Rx function. >> + >> +The DPDK v17.11 virtio net driver contains a bug that prevents guest DPDK >> +applications from receiving packets when the vectorized Rx function is >used. >> +This only occurs when guest-bound traffic is live before a DPDK application >is >> +started within the guest, and where two or more forwarding cores are used. >As >> +such, it is not recommended for guests which execute DPDK applications to >use >> +the virtio vectorized Rx function. A simple method of ensuring that a non- >> +vectorized Rx function is used is to enable mergeable buffers for the >guest, >> +with the following QEMU command line option:: >> + >> + mrg_rxbuf=on > >For an arbitrary DPDK application in the guest that may not solve the issue! >Where or not mergeable Rx buffers are negotiated to be used depends on two >things: 1. Support enabled in Qemu, 2. Support signaled in the guest virtio >driver and 3. Support signaled in the vhostuser backend. > If I'm not mistaken, your primary concern expressed in the three numbered points above is that if the version of DPDK in the guest does not support signaling of mergeable buffer support in both the virtio driver and vhostuser backend, then the vectorized Rx function is still a possibility (point 1. is irrelevant, since QEMU v2.7.0 and up - prerequisites for vhost user client ports - support mergerable buffers). Would your concerns be alleviated if a particular known 'good' version of DPDK was specified here? (v17.05.2, for example). >If the application e.g. for performance reasons doesn't want to use mergeable >Rx buffers, it could still negotiate to off and apply the broken vector rx >routine. That's true - what would you suggest in this instance? Simply a caveat that the mergeable buffer workaround is best-effort? > >> + >> +Additional details regarding the virtio driver bug are available on the >> +`DPDK mailing list`_. >> + >> +.. _DPDK mailing list: http://dpdk.org/ml/archives/dev/2017- >December/082801.html >> + >> .. _dpdk-vhost-user-xml: >> >> Sample XML >> @@ -387,7 +409,7 @@ Sample XML >> <source type='unix' >path='/usr/local/var/run/openvswitch/dpdkvhostuser0' mode='client'/> >> <model type='virtio'/> >> <driver queues='2'> >> - <host mrg_rxbuf='off'/> >> + <host mrg_rxbuf='on'/> > >Are you changing this example because of the current bug? Or is there some >other reason? > >I don't think the temporary bug in DPDK 17.11 justifies to change the example >VM XML to enable mrg_rxbuf. This costs performance in OVS and DPDK >applications should use it only when needed (e.g. because jumbo frames must be >received fragmented into smaller mbufs). > >> </driver> >> </interface> >> <interface type='vhostuser'> >> @@ -395,7 +417,7 @@ Sample XML >> <source type='unix' >path='/usr/local/var/run/openvswitch/dpdkvhostuser1' mode='client'/> >> <model type='virtio'/> >> <driver queues='2'> >> - <host mrg_rxbuf='off'/> >> + <host mrg_rxbuf='on'/> >> </driver> >> </interface> >> <serial type='pty'> >> diff --git a/NEWS b/NEWS >> index 427c8f8..d4a1c9a 100644 >> --- a/NEWS >> +++ b/NEWS >> @@ -13,6 +13,8 @@ Post-v2.8.0 >> * ovn-ctl: New commands run_nb_ovsdb and run_sb_ovsdb. >> - Linux kernel 4.13 >> * Add support for compiling OVS with the latest Linux 4.13 kernel >> + - DPDK: >> + * Add support for DPDK v17.11 >> >> v2.8.0 - 31 Aug 2017 >> -------------------- >> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c >> index faff842..f552444 100644 >> --- a/lib/netdev-dpdk.c >> +++ b/lib/netdev-dpdk.c >> @@ -26,6 +26,7 @@ >> #include <sys/socket.h> >> #include <linux/if.h> >> >> +#include <rte_bus_pci.h> >> #include <rte_config.h> >> #include <rte_cycles.h> >> #include <rte_errno.h> >> @@ -140,8 +141,8 @@ static struct vlog_rate_limit rl = >VLOG_RATE_LIMIT_INIT(5, 20); >> >> #define DPDK_ETH_PORT_ID_INVALID RTE_MAX_ETHPORTS >> >> -/* DPDK library uses uint8_t for port_id. */ >> -typedef uint8_t dpdk_port_t; >> +/* DPDK library uses uint16_t for port_id. */ >> +typedef uint16_t dpdk_port_t; >> >> #define VHOST_ENQ_RETRY_NUM 8 >> #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ) >> -- >> 1.9.3
> > > >What happened to my request to add here that running OVS 2.9 with DPDK > >datapath in a VM with virtio interfaces may itself suffer from the DPDK 17.11 > >bug in OVS' own virtio PMD? The work-around for this could be to enable the > >mrg_rxbuf feature in Qemu for the OVS VM as that would trigger OVS to not use > >vector rx. > > > >This is essentially different from the note in the vhost_user section below > >that talks about DPDK apps running in a guest VM served by OVS. > > I see - I actually didn't get that from your initial mail, but your point is clear now. > Technically, OvS is still a DPDK app in the guest using the virtio PMD, so it would fall under the caveats in this patch regarding guest DPDK > apps. However, I see no harm in calling out OvS in the guest specifically. From OVS perspective I would say "guest" is an OS + application in a VM connected to OVS through vhost or vhostuser ports. Not being able to run OVS-DPDK in a VM is a limitation of OVS itself. I would not look for a warning in the vhost_user documentation. > >> +.. important:: > >> + > >> + DPDK v17.11 virtio PMD contains a bug in the vectorized Rx function that > >> + affects testpmd/DPDK guest applications. As such, guest DPDK applications > >> + should use a non-vectorized Rx function. > >> + > >> +The DPDK v17.11 virtio net driver contains a bug that prevents guest DPDK > >> +applications from receiving packets when the vectorized Rx function is > >used. > >> +This only occurs when guest-bound traffic is live before a DPDK application > >is > >> +started within the guest, and where two or more forwarding cores are used. > >As > >> +such, it is not recommended for guests which execute DPDK applications to > >use > >> +the virtio vectorized Rx function. A simple method of ensuring that a non- > >> +vectorized Rx function is used is to enable mergeable buffers for the > >guest, > >> +with the following QEMU command line option:: > >> + > >> + mrg_rxbuf=on > > > >For an arbitrary DPDK application in the guest that may not solve the issue! > >Where or not mergeable Rx buffers are negotiated to be used depends on two > >things: 1. Support enabled in Qemu, 2. Support signaled in the guest virtio > >driver and 3. Support signaled in the vhostuser backend. > > > > If I'm not mistaken, your primary concern expressed in the three numbered points above is that if the version of DPDK in the guest does > not support signaling of mergeable buffer support in both the virtio driver and vhostuser backend, then the vectorized Rx function is still a > possibility (point 1. is irrelevant, since QEMU v2.7.0 and up - prerequisites for vhost user client ports - support mergerable buffers). Would > your concerns be alleviated if a particular known 'good' version of DPDK was specified here? (v17.05.2, for example). I don't want to complicate things. Actually I would prefer not to discuss this aspect of the DPDK bug in OVS documentation at all! If I, as an OVS user, need to host a VM with an application built against the buggy DPDK 17.11, I can go and ask the supplier to fix it by building against 17.05.2 or against the 17.11.1 release when it comes. I think it is in general impossible to force the application to not use vector rx by enabling mrg_rxbuf on the Qemu cmdline because the application may choose to ignore that and disable mrg_rxbuf anyhow. Perhaps removing some other pre-requisite for the bug, like disabling vhost multi-queue might help. It is only in the special case that the OVS user is also building the VM herself (e.g. as a test tool) that she has a chance to correct the bug by patching DPDK or building with 17.05.2. The latter would be trivial for DPDK test apps like testpmd or pktgen. In other cases it might require application fixes due the incompatible ABI changes. BR, Jan
> > > > > >What happened to my request to add here that running OVS 2.9 with > > >DPDK datapath in a VM with virtio interfaces may itself suffer from > > >the DPDK 17.11 bug in OVS' own virtio PMD? The work-around for this > > >could be to enable the mrg_rxbuf feature in Qemu for the OVS VM as > > >that would trigger OVS to not use vector rx. > > > > > >This is essentially different from the note in the vhost_user section > > >below that talks about DPDK apps running in a guest VM served by OVS. > > > > I see - I actually didn't get that from your initial mail, but your > point is clear now. > > Technically, OvS is still a DPDK app in the guest using the virtio > > PMD, so it would fall under the caveats in this patch regarding guest > DPDK apps. However, I see no harm in calling out OvS in the guest > specifically. > > From OVS perspective I would say "guest" is an OS + application in a VM > connected to OVS through vhost or vhostuser ports. > > Not being able to run OVS-DPDK in a VM is a limitation of OVS itself. I > would not look for a warning in the vhost_user documentation. > Thanks to all for input here, after some discussion with Jan and Mark we've decided to push this patch to the DPDK_MERGE branch for upstream as is. There will be an incremental patch in the future to finalize the usecase warnings and the exact details of what needs to be flagged to the user. Thanks Ian > > >> +.. important:: > > >> + > > >> + DPDK v17.11 virtio PMD contains a bug in the vectorized Rx > > >> + function that affects testpmd/DPDK guest applications. As such, > > >> + guest DPDK applications should use a non-vectorized Rx function. > > >> + > > >> +The DPDK v17.11 virtio net driver contains a bug that prevents > > >> +guest DPDK applications from receiving packets when the vectorized > > >> +Rx function is > > >used. > > >> +This only occurs when guest-bound traffic is live before a DPDK > > >> +application > > >is > > >> +started within the guest, and where two or more forwarding cores are > used. > > >As > > >> +such, it is not recommended for guests which execute DPDK > > >> +applications to > > >use > > >> +the virtio vectorized Rx function. A simple method of ensuring > > >> +that a non- vectorized Rx function is used is to enable mergeable > > >> +buffers for the > > >guest, > > >> +with the following QEMU command line option:: > > >> + > > >> + mrg_rxbuf=on > > > > > >For an arbitrary DPDK application in the guest that may not solve the > issue! > > >Where or not mergeable Rx buffers are negotiated to be used depends > > >on two > > >things: 1. Support enabled in Qemu, 2. Support signaled in the guest > > >virtio driver and 3. Support signaled in the vhostuser backend. > > > > > > > If I'm not mistaken, your primary concern expressed in the three > > numbered points above is that if the version of DPDK in the guest does > > not support signaling of mergeable buffer support in both the virtio > > driver and vhostuser backend, then the vectorized Rx function is still a > possibility (point 1. is irrelevant, since QEMU v2.7.0 and up - > prerequisites for vhost user client ports - support mergerable buffers). > Would your concerns be alleviated if a particular known 'good' version of > DPDK was specified here? (v17.05.2, for example). > > I don't want to complicate things. Actually I would prefer not to discuss > this aspect of the DPDK bug in OVS documentation at all! > > If I, as an OVS user, need to host a VM with an application built against > the buggy DPDK 17.11, I can go and ask the supplier to fix it by building > against 17.05.2 or against the 17.11.1 release when it comes. > > I think it is in general impossible to force the application to not use > vector rx by enabling mrg_rxbuf on the Qemu cmdline because the > application may choose to ignore that and disable mrg_rxbuf anyhow. > Perhaps removing some other pre-requisite for the bug, like disabling > vhost multi-queue might help. > > It is only in the special case that the OVS user is also building the VM > herself (e.g. as a test tool) that she has a chance to correct the bug by > patching DPDK or building with 17.05.2. The latter would be trivial for > DPDK test apps like testpmd or pktgen. In other cases it might require > application fixes due the incompatible ABI changes. > > BR, Jan
diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh index 4d6459f..ed28ee4 100755 --- a/.travis/linux-build.sh +++ b/.travis/linux-build.sh @@ -81,7 +81,7 @@ fi if [ "$DPDK" ]; then if [ -z "$DPDK_VER" ]; then - DPDK_VER="17.05.2" + DPDK_VER="17.11" fi install_dpdk $DPDK_VER if [ "$CC" = "clang" ]; then diff --git a/Documentation/faq/releases.rst b/Documentation/faq/releases.rst index d903b06..62a1957 100644 --- a/Documentation/faq/releases.rst +++ b/Documentation/faq/releases.rst @@ -164,6 +164,7 @@ Q: What DPDK version does each Open vSwitch release work with? 2.6.x 16.07.2 2.7.x 16.11.3 2.8.x 17.05.2 + 2.9.x 17.11 ============ ======= Q: I get an error like this when I configure Open vSwitch: diff --git a/Documentation/intro/install/dpdk.rst b/Documentation/intro/install/dpdk.rst index bb69ae5..3fecb5c 100644 --- a/Documentation/intro/install/dpdk.rst +++ b/Documentation/intro/install/dpdk.rst @@ -40,7 +40,7 @@ Build requirements In addition to the requirements described in :doc:`general`, building Open vSwitch with DPDK will require the following: -- DPDK 17.05.2 +- DPDK 17.11 - A `DPDK supported NIC`_ @@ -69,9 +69,9 @@ Install DPDK #. Download the `DPDK sources`_, extract the file and set ``DPDK_DIR``:: $ cd /usr/src/ - $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz - $ tar xf dpdk-17.05.2.tar.xz - $ export DPDK_DIR=/usr/src/dpdk-stable-17.05.2 + $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz + $ tar xf dpdk-17.11.tar.xz + $ export DPDK_DIR=/usr/src/dpdk-17.11 $ cd $DPDK_DIR #. (Optional) Configure DPDK as a shared library @@ -583,7 +583,7 @@ Limitations The latest list of validated firmware versions can be found in the `DPDK release notes`_. -.. _DPDK release notes: http://dpdk.org/doc/guides/rel_notes/release_17_05.html +.. _DPDK release notes: http://dpdk.org/doc/guides/rel_notes/release_17_11.html Reporting Bugs -------------- diff --git a/Documentation/topics/dpdk/ring.rst b/Documentation/topics/dpdk/ring.rst index ad9d7a5..8d0ede8 100644 --- a/Documentation/topics/dpdk/ring.rst +++ b/Documentation/topics/dpdk/ring.rst @@ -77,4 +77,4 @@ DPDK. However, this functionality was removed because: - :doc:`vhost-user interfaces <vhost-user>` are the defacto DPDK-based path to guests -.. _DPDK documentation: https://dpdk.readthedocs.io/en/v17.05/prog_guide/ring_lib.html +.. _DPDK documentation: https://dpdk.readthedocs.io/en/v17.11/prog_guide/ring_lib.html diff --git a/Documentation/topics/dpdk/vhost-user.rst b/Documentation/topics/dpdk/vhost-user.rst index 74ac06e..a43affa 100644 --- a/Documentation/topics/dpdk/vhost-user.rst +++ b/Documentation/topics/dpdk/vhost-user.rst @@ -292,9 +292,9 @@ To begin, instantiate a guest as described in :ref:`dpdk-vhost-user` or DPDK sources to VM and build DPDK:: $ cd /root/dpdk/ - $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz - $ tar xf dpdk-17.05.2.tar.xz - $ export DPDK_DIR=/root/dpdk/dpdk-stable-17.05.2 + $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz + $ tar xf dpdk-17.11.tar.xz + $ export DPDK_DIR=/root/dpdk/dpdk-17.11 $ export DPDK_TARGET=x86_64-native-linuxapp-gcc $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET $ cd $DPDK_DIR @@ -327,6 +327,28 @@ Finally, start the application:: # TODO +.. important:: + + DPDK v17.11 virtio PMD contains a bug in the vectorized Rx function that + affects testpmd/DPDK guest applications. As such, guest DPDK applications + should use a non-vectorized Rx function. + +The DPDK v17.11 virtio net driver contains a bug that prevents guest DPDK +applications from receiving packets when the vectorized Rx function is used. +This only occurs when guest-bound traffic is live before a DPDK application is +started within the guest, and where two or more forwarding cores are used. As +such, it is not recommended for guests which execute DPDK applications to use +the virtio vectorized Rx function. A simple method of ensuring that a non- +vectorized Rx function is used is to enable mergeable buffers for the guest, +with the following QEMU command line option:: + + mrg_rxbuf=on + +Additional details regarding the virtio driver bug are available on the +`DPDK mailing list`_. + +.. _DPDK mailing list: http://dpdk.org/ml/archives/dev/2017-December/082801.html + .. _dpdk-vhost-user-xml: Sample XML @@ -387,7 +409,7 @@ Sample XML <source type='unix' path='/usr/local/var/run/openvswitch/dpdkvhostuser0' mode='client'/> <model type='virtio'/> <driver queues='2'> - <host mrg_rxbuf='off'/> + <host mrg_rxbuf='on'/> </driver> </interface> <interface type='vhostuser'> @@ -395,7 +417,7 @@ Sample XML <source type='unix' path='/usr/local/var/run/openvswitch/dpdkvhostuser1' mode='client'/> <model type='virtio'/> <driver queues='2'> - <host mrg_rxbuf='off'/> + <host mrg_rxbuf='on'/> </driver> </interface> <serial type='pty'> diff --git a/NEWS b/NEWS index 427c8f8..d4a1c9a 100644 --- a/NEWS +++ b/NEWS @@ -13,6 +13,8 @@ Post-v2.8.0 * ovn-ctl: New commands run_nb_ovsdb and run_sb_ovsdb. - Linux kernel 4.13 * Add support for compiling OVS with the latest Linux 4.13 kernel + - DPDK: + * Add support for DPDK v17.11 v2.8.0 - 31 Aug 2017 -------------------- diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index faff842..f552444 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -26,6 +26,7 @@ #include <sys/socket.h> #include <linux/if.h> +#include <rte_bus_pci.h> #include <rte_config.h> #include <rte_cycles.h> #include <rte_errno.h> @@ -140,8 +141,8 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 20); #define DPDK_ETH_PORT_ID_INVALID RTE_MAX_ETHPORTS -/* DPDK library uses uint8_t for port_id. */ -typedef uint8_t dpdk_port_t; +/* DPDK library uses uint16_t for port_id. */ +typedef uint16_t dpdk_port_t; #define VHOST_ENQ_RETRY_NUM 8 #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ)