| Message ID | 20200624144058.6786-1-gmuthukr@redhat.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [ovs-dev] system-dpdk: add negotiation check for userspace-tso | expand |
Hi Gowrishankar,
Thanks for working on this patch, it's a good idea to test the
TSO negotiation.
Which dpdk version are you testing? Because I am getting the same
results on 19.11.3 or master. See the details below.
Sometimes it fails with this in testpmd-dpdkvhostuser0.log:
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Cannot create lock on '/var/run/dpdk/page0/config'. Is another
primary process running?
EAL: FATAL: Cannot init config
EAL: Cannot init config
EAL: Error - exiting with code: 1
Cause: Cannot init EAL: Success
Because the previous testpmd hasn't terminated fully yet.
I added this hack and it helped:
@@ -302,6 +302,7 @@ AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
dnl Clean up the testpmd now
pkill -f -x -9 'tail -f /dev/null'
+sleep 1
dnl Add vhostuser port (server mode)
AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
It also fails regularly checking for TCP_TSO flag in testpmd output.
I did a copy of the log right before checking and compared with the
log at the end of the test. Turns out that the log is not updated yet.
So, I improved the hack above to terminate testpmd earlier and give
extra time in both places:
@@ -287,6 +287,9 @@ tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
dnl Give settling time to the testpmd processes - NOTE: this is bad form.
sleep 10
+dnl Clean up the testpmd now
+pkill -f -x -9 'tail -f /dev/null'
+sleep 1
dnl Check whether TSO is turned on (host side)
AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
@@ -300,9 +303,6 @@ AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END
dnl Clean up
AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
-dnl Clean up the testpmd now
-pkill -f -x -9 'tail -f /dev/null'
-
dnl Add vhostuser port (server mode)
AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
type=dpdkvhostuser], [],
Same change for the next chunk below that. Perhaps it can wait until
the /var/run/dpdk/page0 is gone.
Then it fails when testing vhost in server mode because OVS doesn't
pass the EXTBUF flag, so TSO gets disabled:
@@ -1349,6 +1349,12 @@ netdev_dpdk_vhost_construct(struct netdev *netdev)
/* There is no support for multi-segments buffers. */
dev->vhost_driver_flags |= RTE_VHOST_USER_LINEARBUF_SUPPORT;
+
+ /* Enable External Buffers if TCP Segmentation Offload is enabled. */
+ if (userspace_tso_enabled()) {
+ dev->vhost_driver_flags |= RTE_VHOST_USER_EXTBUF_SUPPORT;
+ }
+
err = rte_vhost_driver_register(dev->vhost_id, dev->vhost_driver_flags);
if (err) {
VLOG_ERR("vhost-user socket device setup failure for socket %s\n",
I am going to post that as a patch tomorrow.
Now it always runs until vswitchd is stopped, but then it fails because
vswitchd reports a NUMA socket error:
2020-07-03T00:11:35.575Z|00019|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
+2020-07-03T00:11:35.575Z|00021|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
+2020-07-03T00:11:35.652Z|00054|dpdk|WARN|VHOST_CONFIG: failed to connect to /home/fleitner/repo/ovs/tests/system-dpdk-testsuite.dir/6/dpdkvhostclient0: No such file or directory
+2020-07-03T00:11:45.764Z|00044|dpdk|WARN|VHOST_CONFIG: failed to connect to /home/fleitner/repo/ovs/tests/system-dpdk-testsuite.dir/6/dpdkvhostclient0: No such file or directory
+2020-07-03T00:11:46.801Z|00065|netdev_dpdk|WARN|dpdkvhostuser ports are considered deprecated; please migrate to dpdkvhostuserclient ports.
+2020-07-03T00:11:35Z|00019|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
+2020-07-03T00:11:35Z|00021|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
My system is a virtual machine running fedora 32 with a virtio NIC
for testing. It has selinux disabled, upstream kernel cd77006e01b
recompiled with fedora's kernel config + CONFIG_VFIO_NOIOMMU=y to
load vfio kernel module passing enable_unsafe_noiommu_mode=1.
fbl
On Wed, Jun 24, 2020 at 08:10:58PM +0530, Gowrishankar Muthukrishnan wrote:
> This patch adds minimal check for userspace-tso in system-dpdk
> tests, starting with verification on virtio negotiation.
>
> Signed-off-by: Gowrishankar Muthukrishnan <gmuthukr@redhat.com>
> ---
> v4:
> - configure offload settings in testpmd
> ---
> tests/system-dpdk-macros.at | 17 +++-
> tests/system-dpdk.at | 228 +++++++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 236 insertions(+), 9 deletions(-)
>
> diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at
> index c6708ca..9e55f10 100644
> --- a/tests/system-dpdk-macros.at
> +++ b/tests/system-dpdk-macros.at
> @@ -33,13 +33,11 @@ m4_define([OVS_DPDK_PRE_PHY_SKIP],
> ])
>
>
> -# OVS_DPDK_START()
> +# OVS_DB_START()
> #
> -# Create an empty database and start ovsdb-server. Add special configuration
> -# dpdk-init to enable DPDK functionality. Start ovs-vswitchd connected to that
> -# database using system devices (no dummies).
> +# Create an empty database and start ovsdb-server.
> #
> -m4_define([OVS_DPDK_START],
> +m4_define([OVS_DB_START],
> [dnl Create database.
> AT_CHECK([touch .conf.db.~lock~])
> AT_CHECK([ovsdb-tool create conf.db $abs_top_srcdir/vswitchd/vswitch.ovsschema])
> @@ -54,7 +52,16 @@ m4_define([OVS_DPDK_START],
>
> dnl Initialize database.
> AT_CHECK([ovs-vsctl --no-wait init])
> +])
> +
>
> +# OVS_DPDK_START()
> +#
> +# Add special configuration dpdk-init to enable DPDK functionality.
> +# Start ovs-vswitchd connected to that database using system devices (no dummies).
> +#
> +m4_define([OVS_DPDK_START],
> + [
> dnl Enable DPDK functionality
> AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true])
>
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index a015d52..ba9a1e7 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -1,6 +1,11 @@
> m4_define([CONFIGURE_VETH_OFFLOADS],
> [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
>
> +m4_define([SET_NUMA_NODE],
> + [
> + AT_CHECK([lscpu | awk '/NUMA node\(s\)/ {c=1; while (c++<$(3)) {printf "$1,"}; print "$1"}' > NUMA_NODE])
> +])
> +
> AT_BANNER([OVS-DPDK unit tests])
>
> dnl --------------------------------------------------------------------------
> @@ -8,6 +13,7 @@ dnl Check if EAL init is successful
> AT_SETUP([OVS-DPDK - EAL init])
> AT_KEYWORDS([dpdk])
> OVS_DPDK_PRE_CHECK()
> +OVS_DB_START()
> OVS_DPDK_START()
> AT_CHECK([grep "DPDK Enabled - initializing..." ovs-vswitchd.log], [], [stdout])
> AT_CHECK([grep "EAL" ovs-vswitchd.log], [], [stdout])
> @@ -27,6 +33,7 @@ AT_SETUP([OVS-DPDK - add standard DPDK port])
> AT_KEYWORDS([dpdk])
>
> OVS_DPDK_PRE_PHY_SKIP()
> +OVS_DB_START()
> OVS_DPDK_START()
>
> dnl Add userspace bridge and attach it to OVS
> @@ -53,6 +60,7 @@ dnl Add vhost-user-client port
> AT_SETUP([OVS-DPDK - add vhost-user-client port])
> AT_KEYWORDS([dpdk])
> OVS_DPDK_PRE_CHECK()
> +OVS_DB_START()
> OVS_DPDK_START()
>
> dnl Add userspace bridge and attach it to OVS
> @@ -86,11 +94,11 @@ AT_SETUP([OVS-DPDK - ping vhost-user ports])
> AT_KEYWORDS([dpdk])
> OVS_DPDK_PRE_CHECK()
> AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DB_START()
> OVS_DPDK_START()
>
> dnl Find number of sockets
> -AT_CHECK([lscpu], [], [stdout])
> -AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE])
> +SET_NUMA_NODE([512])
>
> dnl Add userspace bridge and attach it to OVS
> AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> @@ -163,11 +171,11 @@ AT_SETUP([OVS-DPDK - ping vhost-user-client ports])
> AT_KEYWORDS([dpdk])
> OVS_DPDK_PRE_CHECK()
> AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DB_START()
> OVS_DPDK_START()
>
> dnl Find number of sockets
> -AT_CHECK([lscpu], [], [stdout])
> -AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE])
> +SET_NUMA_NODE([512])
>
> dnl Add userspace bridge and attach it to OVS
> AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> @@ -232,3 +240,215 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is probably
> \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> AT_CLEANUP
> dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl validate tso negotiation (with userspace-tso)
> +AT_SETUP([OVS-DPDK - validate tso negotiation (with userspace-tso)])
> +AT_KEYWORDS([dpdk])
> +OVS_DPDK_PRE_CHECK()
> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DB_START()
> +AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:userspace-tso-enable=true])
> +OVS_DPDK_START()
> +AT_CHECK([grep -c 'Userspace TCP Segmentation Offloading support enabled' \
> + ovs-vswitchd.log],[ignore],[dnl
> +1
> +])
> +dnl Find number of sockets
> +SET_NUMA_NODE([512])
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +
> +dnl Add vhostuser port (client mode)
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \
> + dpdkvhostuserclient0 \
> + type=dpdkvhostuserclient \
> + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> + [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +AT_CHECK([echo "show device info all" > CMDFILE])
> +AT_CHECK([echo "stop" >> CMDFILE])
> +AT_CHECK([echo "port stop 0" >> CMDFILE])
> +AT_CHECK([echo "tso set 1500 0" >> CMDFILE])
> +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE])
> +AT_CHECK([echo "port start 0" >> CMDFILE])
> +AT_CHECK([echo "start" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE])
> +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
> + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \
> + --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> + --single-file-segments -- --cmdline-file=CMDFILE \
> + -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Check whether TSO is turned on (host side)
> +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
> + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout])
> +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[1801])
> +
> +dnl Check whether TSO is turned on (guest side)
> +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \
> + $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log],[0],[1])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Add vhostuser port (server mode)
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
> + type=dpdkvhostuser], [],
> + [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +AT_CHECK([echo "show device info all" > CMDFILE])
> +AT_CHECK([echo "stop" >> CMDFILE])
> +AT_CHECK([echo "port stop 0" >> CMDFILE])
> +AT_CHECK([echo "tso set 1500 0" >> CMDFILE])
> +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE])
> +AT_CHECK([echo "port start 0" >> CMDFILE])
> +AT_CHECK([echo "start" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE])
> +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
> + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> + --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> + --single-file-segments -- --cmdline-file=CMDFILE \
> + -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Check whether TSO is turned on (host side)
> +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
> + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout])
> +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[1801])
> +
> +dnl Check whether TSO is turned on (guest side)
> +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \
> + $OVS_RUNDIR/testpmd-dpdkvhostuser0.log],[0],[1])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP()
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl validate tso negotiation (without userspace-tso)
> +AT_SETUP([OVS-DPDK - validate tso negotiation (without userspace-tso)])
> +AT_KEYWORDS([dpdk])
> +OVS_DPDK_PRE_CHECK()
> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DB_START()
> +AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:userspace-tso-enable=false])
> +OVS_DPDK_START()
> +AT_CHECK([grep -c 'Userspace TCP Segmentation Offloading support enabled' \
> + ovs-vswitchd.log],[ignore],[dnl
> +0
> +])
> +dnl Find number of sockets
> +SET_NUMA_NODE([512])
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +
> +dnl Add vhostuser port (client mode)
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \
> + dpdkvhostuserclient0 \
> + type=dpdkvhostuserclient \
> + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> + [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +AT_CHECK([echo "show device info all" > CMDFILE])
> +AT_CHECK([echo "stop" >> CMDFILE])
> +AT_CHECK([echo "port stop 0" >> CMDFILE])
> +AT_CHECK([echo "tso set 1500 0" >> CMDFILE])
> +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE])
> +AT_CHECK([echo "port start 0" >> CMDFILE])
> +AT_CHECK([echo "start" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE])
> +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
> + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \
> + --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> + --single-file-segments -- --cmdline-file=CMDFILE \
> + -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Check whether TSO is turned off (host side)
> +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
> + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout])
> +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[0])
> +
> +dnl Check whether TSO is turned off (guest side)
> +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \
> + $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log],[0],[0])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Add vhostuser port (server mode)
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
> + type=dpdkvhostuser], [],
> + [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +AT_CHECK([echo "show device info all" > CMDFILE])
> +AT_CHECK([echo "stop" >> CMDFILE])
> +AT_CHECK([echo "port stop 0" >> CMDFILE])
> +AT_CHECK([echo "tso set 1500 0" >> CMDFILE])
> +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE])
> +AT_CHECK([echo "port start 0" >> CMDFILE])
> +AT_CHECK([echo "start" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE])
> +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
> + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> + --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> + --single-file-segments -- --cmdline-file=CMDFILE \
> + -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Check whether TSO is turned off (host side)
> +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
> + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout])
> +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[0])
> +
> +dnl Check whether TSO is turned off (guest side)
> +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \
> + $OVS_RUNDIR/testpmd-dpdkvhostuser0.log],[0],[0])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP()
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> --
> 1.8.3.1
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Hi Flavio, Thanks for reviewing it. My comments inline. On Fri, Jul 3, 2020 at 6:13 AM Flavio Leitner <fbl@sysclose.org> wrote: > > Hi Gowrishankar, > > Thanks for working on this patch, it's a good idea to test the > TSO negotiation. > > Which dpdk version are you testing? Because I am getting the same > results on 19.11.3 or master. See the details below. > > I had used 19.11 before, but now tested with 19.11.3. I could reproduce below error (but not before). So, I go with your suggestion to sleep a bit. > Sometimes it fails with this in testpmd-dpdkvhostuser0.log: > EAL: Detected 4 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Cannot create lock on '/var/run/dpdk/page0/config'. Is another > primary process running? > EAL: FATAL: Cannot init config > EAL: Cannot init config > EAL: Error - exiting with code: 1 > Cause: Cannot init EAL: Success > > Because the previous testpmd hasn't terminated fully yet. > I added this hack and it helped: > @@ -302,6 +302,7 @@ AT_CHECK([ovs-vsctl del-port br10 > dpdkvhostuserclient0], [], [stdout], [stderr]) > > dnl Clean up the testpmd now > pkill -f -x -9 'tail -f /dev/null' > +sleep 1 > > dnl Add vhostuser port (server mode) > AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface > dpdkvhostuser0 \ > > > It also fails regularly checking for TCP_TSO flag in testpmd output. > I did a copy of the log right before checking and compared with the > log at the end of the test. Turns out that the log is not updated yet. > > So, I improved the hack above to terminate testpmd earlier and give > extra time in both places: > > @@ -287,6 +287,9 @@ tail -f /dev/null | testpmd --socket-mem="$(cat > NUMA_NODE)" --no-pci\ > > dnl Give settling time to the testpmd processes - NOTE: this is bad form. > sleep 10 > +dnl Clean up the testpmd now > +pkill -f -x -9 'tail -f /dev/null' > +sleep 1 > > dnl Check whether TSO is turned on (host side) > AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ > @@ -300,9 +303,6 @@ AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && > /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END > dnl Clean up > AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], > [stderr]) > > -dnl Clean up the testpmd now > -pkill -f -x -9 'tail -f /dev/null' > - > dnl Add vhostuser port (server mode) > AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface > dpdkvhostuser0 \ > type=dpdkvhostuser], [], > > > Same change for the next chunk below that. Perhaps it can wait until the /var/run/dpdk/page0 is gone. > But I never see this folder disappear even after killing testpmd. > > Then it fails when testing vhost in server mode because OVS doesn't > pass the EXTBUF flag, so TSO gets disabled: > @@ -1349,6 +1349,12 @@ netdev_dpdk_vhost_construct(struct netdev *netdev) > > /* There is no support for multi-segments buffers. */ > dev->vhost_driver_flags |= RTE_VHOST_USER_LINEARBUF_SUPPORT; > + > + /* Enable External Buffers if TCP Segmentation Offload is enabled. */ > + if (userspace_tso_enabled()) { > + dev->vhost_driver_flags |= RTE_VHOST_USER_EXTBUF_SUPPORT; > + } > + > I had noticed test failure* due to this, but less prioritized patching it as vhostuser (server) mode is depreciated for ovs. * - https://patchwork.ozlabs.org/project/openvswitch/patch/0c4c167ad644d3dda51b992e51ec1c27b8492457.1589605823.git.gmuthukr@redhat.com/ err = rte_vhost_driver_register(dev->vhost_id, > dev->vhost_driver_flags); > if (err) { > VLOG_ERR("vhost-user socket device setup failure for socket %s\n", > > I am going to post that as a patch tomorrow. > > Thanks. > Now it always runs until vswitchd is stopped, but then it fails because > vswitchd reports a NUMA socket error: > > I had seen issues with stop and I thought I'm alone facing it, so disabled log check while running tests. I think ovs_vswitchd_stop itself would require further mask on checking logs. I would prefer fixing it separately. Thanks, I'm sending a new version with your suggestions. --- a/tests/ofproto-macros.at +++ b/tests/ofproto-macros.at @@ -312,7 +312,7 @@ m4_divert_pop([PREPARE_TESTS]) # # OVS_VSWITCHD_STOP(["/expected error/d"]) m4_define([OVS_VSWITCHD_STOP], - [AT_CHECK([check_logs $1]) + [ OVS_APP_EXIT_AND_WAIT([ovs-vswitchd]) OVS_APP_EXIT_AND_WAIT([ovsdb-server])]) > 2020-07-03T00:11:35.575Z|00019|dpdk|WARN|EAL: Invalid NUMA socket, > default to 0 > +2020-07-03T00:11:35.575Z|00021|dpdk|WARN|EAL: Invalid NUMA socket, > default to 0 > +2020-07-03T00:11:35.652Z|00054|dpdk|WARN|VHOST_CONFIG: failed to connect > to > /home/fleitner/repo/ovs/tests/system-dpdk-testsuite.dir/6/dpdkvhostclient0: > No such file or directory > +2020-07-03T00:11:45.764Z|00044|dpdk|WARN|VHOST_CONFIG: failed to connect > to > /home/fleitner/repo/ovs/tests/system-dpdk-testsuite.dir/6/dpdkvhostclient0: > No such file or directory > +2020-07-03T00:11:46.801Z|00065|netdev_dpdk|WARN|dpdkvhostuser ports are > considered deprecated; please migrate to dpdkvhostuserclient ports. > +2020-07-03T00:11:35Z|00019|dpdk|WARN|EAL: Invalid NUMA socket, default > to 0 > +2020-07-03T00:11:35Z|00021|dpdk|WARN|EAL: Invalid NUMA socket, default > to 0 > > My system is a virtual machine running fedora 32 with a virtio NIC > for testing. It has selinux disabled, upstream kernel cd77006e01b > recompiled with fedora's kernel config + CONFIG_VFIO_NOIOMMU=y to > load vfio kernel module passing enable_unsafe_noiommu_mode=1. > > fbl > > On Wed, Jun 24, 2020 at 08:10:58PM +0530, Gowrishankar Muthukrishnan wrote: > > This patch adds minimal check for userspace-tso in system-dpdk > > tests, starting with verification on virtio negotiation. > > > > Signed-off-by: Gowrishankar Muthukrishnan <gmuthukr@redhat.com> > > --- > > v4: > > - configure offload settings in testpmd > > --- > > tests/system-dpdk-macros.at | 17 +++- > > tests/system-dpdk.at | 228 > +++++++++++++++++++++++++++++++++++++++++++- > > 2 files changed, 236 insertions(+), 9 deletions(-) > > > > diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at > > index c6708ca..9e55f10 100644 > > --- a/tests/system-dpdk-macros.at > > +++ b/tests/system-dpdk-macros.at > > @@ -33,13 +33,11 @@ m4_define([OVS_DPDK_PRE_PHY_SKIP], > > ]) > > > > > > -# OVS_DPDK_START() > > +# OVS_DB_START() > > # > > -# Create an empty database and start ovsdb-server. Add special > configuration > > -# dpdk-init to enable DPDK functionality. Start ovs-vswitchd connected > to that > > -# database using system devices (no dummies). > > +# Create an empty database and start ovsdb-server. > > # > > -m4_define([OVS_DPDK_START], > > +m4_define([OVS_DB_START], > > [dnl Create database. > > AT_CHECK([touch .conf.db.~lock~]) > > AT_CHECK([ovsdb-tool create conf.db > $abs_top_srcdir/vswitchd/vswitch.ovsschema]) > > @@ -54,7 +52,16 @@ m4_define([OVS_DPDK_START], > > > > dnl Initialize database. > > AT_CHECK([ovs-vsctl --no-wait init]) > > +]) > > + > > > > +# OVS_DPDK_START() > > +# > > +# Add special configuration dpdk-init to enable DPDK functionality. > > +# Start ovs-vswitchd connected to that database using system devices > (no dummies). > > +# > > +m4_define([OVS_DPDK_START], > > + [ > > dnl Enable DPDK functionality > > AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . > other_config:dpdk-init=true]) > > > > diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at > > index a015d52..ba9a1e7 100644 > > --- a/tests/system-dpdk.at > > +++ b/tests/system-dpdk.at > > @@ -1,6 +1,11 @@ > > m4_define([CONFIGURE_VETH_OFFLOADS], > > [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])]) > > > > +m4_define([SET_NUMA_NODE], > > + [ > > + AT_CHECK([lscpu | awk '/NUMA node\(s\)/ {c=1; while (c++<$(3)) > {printf "$1,"}; print "$1"}' > NUMA_NODE]) > > +]) > > + > > AT_BANNER([OVS-DPDK unit tests]) > > > > dnl > -------------------------------------------------------------------------- > > @@ -8,6 +13,7 @@ dnl Check if EAL init is successful > > AT_SETUP([OVS-DPDK - EAL init]) > > AT_KEYWORDS([dpdk]) > > OVS_DPDK_PRE_CHECK() > > +OVS_DB_START() > > OVS_DPDK_START() > > AT_CHECK([grep "DPDK Enabled - initializing..." ovs-vswitchd.log], [], > [stdout]) > > AT_CHECK([grep "EAL" ovs-vswitchd.log], [], [stdout]) > > @@ -27,6 +33,7 @@ AT_SETUP([OVS-DPDK - add standard DPDK port]) > > AT_KEYWORDS([dpdk]) > > > > OVS_DPDK_PRE_PHY_SKIP() > > +OVS_DB_START() > > OVS_DPDK_START() > > > > dnl Add userspace bridge and attach it to OVS > > @@ -53,6 +60,7 @@ dnl Add vhost-user-client port > > AT_SETUP([OVS-DPDK - add vhost-user-client port]) > > AT_KEYWORDS([dpdk]) > > OVS_DPDK_PRE_CHECK() > > +OVS_DB_START() > > OVS_DPDK_START() > > > > dnl Add userspace bridge and attach it to OVS > > @@ -86,11 +94,11 @@ AT_SETUP([OVS-DPDK - ping vhost-user ports]) > > AT_KEYWORDS([dpdk]) > > OVS_DPDK_PRE_CHECK() > > AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null]) > > +OVS_DB_START() > > OVS_DPDK_START() > > > > dnl Find number of sockets > > -AT_CHECK([lscpu], [], [stdout]) > > -AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while > (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE]) > > +SET_NUMA_NODE([512]) > > > > dnl Add userspace bridge and attach it to OVS > > AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 > datapath_type=netdev]) > > @@ -163,11 +171,11 @@ AT_SETUP([OVS-DPDK - ping vhost-user-client ports]) > > AT_KEYWORDS([dpdk]) > > OVS_DPDK_PRE_CHECK() > > AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null]) > > +OVS_DB_START() > > OVS_DPDK_START() > > > > dnl Find number of sockets > > -AT_CHECK([lscpu], [], [stdout]) > > -AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while > (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE]) > > +SET_NUMA_NODE([512]) > > > > dnl Add userspace bridge and attach it to OVS > > AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 > datapath_type=netdev]) > > @@ -232,3 +240,215 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open > vSwitch kernel module is probably > > \@EAL: No free hugepages reported in hugepages-1048576kB@d"]) > > AT_CLEANUP > > dnl > -------------------------------------------------------------------------- > > + > > +dnl > -------------------------------------------------------------------------- > > +dnl validate tso negotiation (with userspace-tso) > > +AT_SETUP([OVS-DPDK - validate tso negotiation (with userspace-tso)]) > > +AT_KEYWORDS([dpdk]) > > +OVS_DPDK_PRE_CHECK() > > +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null]) > > +OVS_DB_START() > > +AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . > other_config:userspace-tso-enable=true]) > > +OVS_DPDK_START() > > +AT_CHECK([grep -c 'Userspace TCP Segmentation Offloading support > enabled' \ > > + ovs-vswitchd.log],[ignore],[dnl > > +1 > > +]) > > +dnl Find number of sockets > > +SET_NUMA_NODE([512]) > > + > > +dnl Add userspace bridge and attach it to OVS > > +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 > datapath_type=netdev]) > > + > > +dnl Add vhostuser port (client mode) > > +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface > \ > > + dpdkvhostuserclient0 \ > > + type=dpdkvhostuserclient \ > > + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], > > + [stdout], [stderr]) > > +AT_CHECK([ovs-vsctl show], [], [stdout]) > > + > > +dnl Execute testpmd in background > > +on_exit "pkill -f -x -9 'tail -f /dev/null'" > > +AT_CHECK([echo "show device info all" > CMDFILE]) > > +AT_CHECK([echo "stop" >> CMDFILE]) > > +AT_CHECK([echo "port stop 0" >> CMDFILE]) > > +AT_CHECK([echo "tso set 1500 0" >> CMDFILE]) > > +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE]) > > +AT_CHECK([echo "port start 0" >> CMDFILE]) > > +AT_CHECK([echo "start" >> CMDFILE]) > > +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE]) > > +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE]) > > +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ > > + > --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \ > > + --vdev="net_tap0,iface=tap0" --file-prefix page0 \ > > + --single-file-segments -- --cmdline-file=CMDFILE \ > > + -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 & > > + > > +dnl Give settling time to the testpmd processes - NOTE: this is bad > form. > > +sleep 10 > > + > > +dnl Check whether TSO is turned on (host side) > > +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ > > + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout]) > > +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) > ))],[],[1801]) > > + > > +dnl Check whether TSO is turned on (guest side) > > +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ > {n++} END{printf n}' \ > > + $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log],[0],[1]) > > + > > +dnl Clean up > > +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], > [stderr]) > > + > > +dnl Clean up the testpmd now > > +pkill -f -x -9 'tail -f /dev/null' > > + > > +dnl Add vhostuser port (server mode) > > +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface > dpdkvhostuser0 \ > > + type=dpdkvhostuser], [], > > + [stdout], [stderr]) > > +AT_CHECK([ovs-vsctl show], [], [stdout]) > > + > > +dnl Execute testpmd in background > > +on_exit "pkill -f -x -9 'tail -f /dev/null'" > > +AT_CHECK([echo "show device info all" > CMDFILE]) > > +AT_CHECK([echo "stop" >> CMDFILE]) > > +AT_CHECK([echo "port stop 0" >> CMDFILE]) > > +AT_CHECK([echo "tso set 1500 0" >> CMDFILE]) > > +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE]) > > +AT_CHECK([echo "port start 0" >> CMDFILE]) > > +AT_CHECK([echo "start" >> CMDFILE]) > > +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE]) > > +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE]) > > +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ > > + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \ > > + --vdev="net_tap0,iface=tap0" --file-prefix page0 \ > > + --single-file-segments -- --cmdline-file=CMDFILE \ > > + -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 & > > + > > +dnl Give settling time to the testpmd processes - NOTE: this is bad > form. > > +sleep 10 > > + > > +dnl Check whether TSO is turned on (host side) > > +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ > > + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout]) > > +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) > ))],[],[1801]) > > + > > +dnl Check whether TSO is turned on (guest side) > > +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ > {n++} END{printf n}' \ > > + $OVS_RUNDIR/testpmd-dpdkvhostuser0.log],[0],[1]) > > + > > +dnl Clean up the testpmd now > > +pkill -f -x -9 'tail -f /dev/null' > > + > > +dnl Clean up > > +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], > [stderr]) > > +OVS_VSWITCHD_STOP() > > +AT_CLEANUP > > +dnl > -------------------------------------------------------------------------- > > + > > +dnl > -------------------------------------------------------------------------- > > +dnl validate tso negotiation (without userspace-tso) > > +AT_SETUP([OVS-DPDK - validate tso negotiation (without userspace-tso)]) > > +AT_KEYWORDS([dpdk]) > > +OVS_DPDK_PRE_CHECK() > > +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null]) > > +OVS_DB_START() > > +AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . > other_config:userspace-tso-enable=false]) > > +OVS_DPDK_START() > > +AT_CHECK([grep -c 'Userspace TCP Segmentation Offloading support > enabled' \ > > + ovs-vswitchd.log],[ignore],[dnl > > +0 > > +]) > > +dnl Find number of sockets > > +SET_NUMA_NODE([512]) > > + > > +dnl Add userspace bridge and attach it to OVS > > +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 > datapath_type=netdev]) > > + > > +dnl Add vhostuser port (client mode) > > +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface > \ > > + dpdkvhostuserclient0 \ > > + type=dpdkvhostuserclient \ > > + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], > > + [stdout], [stderr]) > > +AT_CHECK([ovs-vsctl show], [], [stdout]) > > + > > +dnl Execute testpmd in background > > +on_exit "pkill -f -x -9 'tail -f /dev/null'" > > +AT_CHECK([echo "show device info all" > CMDFILE]) > > +AT_CHECK([echo "stop" >> CMDFILE]) > > +AT_CHECK([echo "port stop 0" >> CMDFILE]) > > +AT_CHECK([echo "tso set 1500 0" >> CMDFILE]) > > +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE]) > > +AT_CHECK([echo "port start 0" >> CMDFILE]) > > +AT_CHECK([echo "start" >> CMDFILE]) > > +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE]) > > +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE]) > > +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ > > + > --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \ > > + --vdev="net_tap0,iface=tap0" --file-prefix page0 \ > > + --single-file-segments -- --cmdline-file=CMDFILE \ > > + -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 & > > + > > +dnl Give settling time to the testpmd processes - NOTE: this is bad > form. > > +sleep 10 > > + > > +dnl Check whether TSO is turned off (host side) > > +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ > > + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout]) > > +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) > ))],[],[0]) > > + > > +dnl Check whether TSO is turned off (guest side) > > +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ > {n++} END{printf n}' \ > > + $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log],[0],[0]) > > + > > +dnl Clean up > > +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], > [stderr]) > > + > > +dnl Clean up the testpmd now > > +pkill -f -x -9 'tail -f /dev/null' > > + > > +dnl Add vhostuser port (server mode) > > +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface > dpdkvhostuser0 \ > > + type=dpdkvhostuser], [], > > + [stdout], [stderr]) > > +AT_CHECK([ovs-vsctl show], [], [stdout]) > > + > > +dnl Execute testpmd in background > > +on_exit "pkill -f -x -9 'tail -f /dev/null'" > > +AT_CHECK([echo "show device info all" > CMDFILE]) > > +AT_CHECK([echo "stop" >> CMDFILE]) > > +AT_CHECK([echo "port stop 0" >> CMDFILE]) > > +AT_CHECK([echo "tso set 1500 0" >> CMDFILE]) > > +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE]) > > +AT_CHECK([echo "port start 0" >> CMDFILE]) > > +AT_CHECK([echo "start" >> CMDFILE]) > > +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE]) > > +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE]) > > +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ > > + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \ > > + --vdev="net_tap0,iface=tap0" --file-prefix page0 \ > > + --single-file-segments -- --cmdline-file=CMDFILE \ > > + -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 & > > + > > +dnl Give settling time to the testpmd processes - NOTE: this is bad > form. > > +sleep 10 > > + > > +dnl Check whether TSO is turned off (host side) > > +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ > > + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout]) > > +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) > ))],[],[0]) > > + > > +dnl Check whether TSO is turned off (guest side) > > +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ > {n++} END{printf n}' \ > > + $OVS_RUNDIR/testpmd-dpdkvhostuser0.log],[0],[0]) > > + > > +dnl Clean up the testpmd now > > +pkill -f -x -9 'tail -f /dev/null' > > + > > +dnl Clean up > > +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], > [stderr]) > > +OVS_VSWITCHD_STOP() > > +AT_CLEANUP > > +dnl > -------------------------------------------------------------------------- > > -- > > 1.8.3.1 > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > -- > fbl > >
diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at index c6708ca..9e55f10 100644 --- a/tests/system-dpdk-macros.at +++ b/tests/system-dpdk-macros.at @@ -33,13 +33,11 @@ m4_define([OVS_DPDK_PRE_PHY_SKIP], ]) -# OVS_DPDK_START() +# OVS_DB_START() # -# Create an empty database and start ovsdb-server. Add special configuration -# dpdk-init to enable DPDK functionality. Start ovs-vswitchd connected to that -# database using system devices (no dummies). +# Create an empty database and start ovsdb-server. # -m4_define([OVS_DPDK_START], +m4_define([OVS_DB_START], [dnl Create database. AT_CHECK([touch .conf.db.~lock~]) AT_CHECK([ovsdb-tool create conf.db $abs_top_srcdir/vswitchd/vswitch.ovsschema]) @@ -54,7 +52,16 @@ m4_define([OVS_DPDK_START], dnl Initialize database. AT_CHECK([ovs-vsctl --no-wait init]) +]) + +# OVS_DPDK_START() +# +# Add special configuration dpdk-init to enable DPDK functionality. +# Start ovs-vswitchd connected to that database using system devices (no dummies). +# +m4_define([OVS_DPDK_START], + [ dnl Enable DPDK functionality AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true]) diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at index a015d52..ba9a1e7 100644 --- a/tests/system-dpdk.at +++ b/tests/system-dpdk.at @@ -1,6 +1,11 @@ m4_define([CONFIGURE_VETH_OFFLOADS], [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])]) +m4_define([SET_NUMA_NODE], + [ + AT_CHECK([lscpu | awk '/NUMA node\(s\)/ {c=1; while (c++<$(3)) {printf "$1,"}; print "$1"}' > NUMA_NODE]) +]) + AT_BANNER([OVS-DPDK unit tests]) dnl -------------------------------------------------------------------------- @@ -8,6 +13,7 @@ dnl Check if EAL init is successful AT_SETUP([OVS-DPDK - EAL init]) AT_KEYWORDS([dpdk]) OVS_DPDK_PRE_CHECK() +OVS_DB_START() OVS_DPDK_START() AT_CHECK([grep "DPDK Enabled - initializing..." ovs-vswitchd.log], [], [stdout]) AT_CHECK([grep "EAL" ovs-vswitchd.log], [], [stdout]) @@ -27,6 +33,7 @@ AT_SETUP([OVS-DPDK - add standard DPDK port]) AT_KEYWORDS([dpdk]) OVS_DPDK_PRE_PHY_SKIP() +OVS_DB_START() OVS_DPDK_START() dnl Add userspace bridge and attach it to OVS @@ -53,6 +60,7 @@ dnl Add vhost-user-client port AT_SETUP([OVS-DPDK - add vhost-user-client port]) AT_KEYWORDS([dpdk]) OVS_DPDK_PRE_CHECK() +OVS_DB_START() OVS_DPDK_START() dnl Add userspace bridge and attach it to OVS @@ -86,11 +94,11 @@ AT_SETUP([OVS-DPDK - ping vhost-user ports]) AT_KEYWORDS([dpdk]) OVS_DPDK_PRE_CHECK() AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null]) +OVS_DB_START() OVS_DPDK_START() dnl Find number of sockets -AT_CHECK([lscpu], [], [stdout]) -AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE]) +SET_NUMA_NODE([512]) dnl Add userspace bridge and attach it to OVS AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev]) @@ -163,11 +171,11 @@ AT_SETUP([OVS-DPDK - ping vhost-user-client ports]) AT_KEYWORDS([dpdk]) OVS_DPDK_PRE_CHECK() AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null]) +OVS_DB_START() OVS_DPDK_START() dnl Find number of sockets -AT_CHECK([lscpu], [], [stdout]) -AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE]) +SET_NUMA_NODE([512]) dnl Add userspace bridge and attach it to OVS AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev]) @@ -232,3 +240,215 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is probably \@EAL: No free hugepages reported in hugepages-1048576kB@d"]) AT_CLEANUP dnl -------------------------------------------------------------------------- + +dnl -------------------------------------------------------------------------- +dnl validate tso negotiation (with userspace-tso) +AT_SETUP([OVS-DPDK - validate tso negotiation (with userspace-tso)]) +AT_KEYWORDS([dpdk]) +OVS_DPDK_PRE_CHECK() +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null]) +OVS_DB_START() +AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:userspace-tso-enable=true]) +OVS_DPDK_START() +AT_CHECK([grep -c 'Userspace TCP Segmentation Offloading support enabled' \ + ovs-vswitchd.log],[ignore],[dnl +1 +]) +dnl Find number of sockets +SET_NUMA_NODE([512]) + +dnl Add userspace bridge and attach it to OVS +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev]) + +dnl Add vhostuser port (client mode) +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \ + dpdkvhostuserclient0 \ + type=dpdkvhostuserclient \ + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], + [stdout], [stderr]) +AT_CHECK([ovs-vsctl show], [], [stdout]) + +dnl Execute testpmd in background +on_exit "pkill -f -x -9 'tail -f /dev/null'" +AT_CHECK([echo "show device info all" > CMDFILE]) +AT_CHECK([echo "stop" >> CMDFILE]) +AT_CHECK([echo "port stop 0" >> CMDFILE]) +AT_CHECK([echo "tso set 1500 0" >> CMDFILE]) +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE]) +AT_CHECK([echo "port start 0" >> CMDFILE]) +AT_CHECK([echo "start" >> CMDFILE]) +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE]) +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE]) +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \ + --vdev="net_tap0,iface=tap0" --file-prefix page0 \ + --single-file-segments -- --cmdline-file=CMDFILE \ + -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 & + +dnl Give settling time to the testpmd processes - NOTE: this is bad form. +sleep 10 + +dnl Check whether TSO is turned on (host side) +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout]) +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[1801]) + +dnl Check whether TSO is turned on (guest side) +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \ + $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log],[0],[1]) + +dnl Clean up +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr]) + +dnl Clean up the testpmd now +pkill -f -x -9 'tail -f /dev/null' + +dnl Add vhostuser port (server mode) +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \ + type=dpdkvhostuser], [], + [stdout], [stderr]) +AT_CHECK([ovs-vsctl show], [], [stdout]) + +dnl Execute testpmd in background +on_exit "pkill -f -x -9 'tail -f /dev/null'" +AT_CHECK([echo "show device info all" > CMDFILE]) +AT_CHECK([echo "stop" >> CMDFILE]) +AT_CHECK([echo "port stop 0" >> CMDFILE]) +AT_CHECK([echo "tso set 1500 0" >> CMDFILE]) +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE]) +AT_CHECK([echo "port start 0" >> CMDFILE]) +AT_CHECK([echo "start" >> CMDFILE]) +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE]) +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE]) +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \ + --vdev="net_tap0,iface=tap0" --file-prefix page0 \ + --single-file-segments -- --cmdline-file=CMDFILE \ + -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 & + +dnl Give settling time to the testpmd processes - NOTE: this is bad form. +sleep 10 + +dnl Check whether TSO is turned on (host side) +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout]) +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[1801]) + +dnl Check whether TSO is turned on (guest side) +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \ + $OVS_RUNDIR/testpmd-dpdkvhostuser0.log],[0],[1]) + +dnl Clean up the testpmd now +pkill -f -x -9 'tail -f /dev/null' + +dnl Clean up +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr]) +OVS_VSWITCHD_STOP() +AT_CLEANUP +dnl -------------------------------------------------------------------------- + +dnl -------------------------------------------------------------------------- +dnl validate tso negotiation (without userspace-tso) +AT_SETUP([OVS-DPDK - validate tso negotiation (without userspace-tso)]) +AT_KEYWORDS([dpdk]) +OVS_DPDK_PRE_CHECK() +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null]) +OVS_DB_START() +AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:userspace-tso-enable=false]) +OVS_DPDK_START() +AT_CHECK([grep -c 'Userspace TCP Segmentation Offloading support enabled' \ + ovs-vswitchd.log],[ignore],[dnl +0 +]) +dnl Find number of sockets +SET_NUMA_NODE([512]) + +dnl Add userspace bridge and attach it to OVS +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev]) + +dnl Add vhostuser port (client mode) +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \ + dpdkvhostuserclient0 \ + type=dpdkvhostuserclient \ + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [], + [stdout], [stderr]) +AT_CHECK([ovs-vsctl show], [], [stdout]) + +dnl Execute testpmd in background +on_exit "pkill -f -x -9 'tail -f /dev/null'" +AT_CHECK([echo "show device info all" > CMDFILE]) +AT_CHECK([echo "stop" >> CMDFILE]) +AT_CHECK([echo "port stop 0" >> CMDFILE]) +AT_CHECK([echo "tso set 1500 0" >> CMDFILE]) +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE]) +AT_CHECK([echo "port start 0" >> CMDFILE]) +AT_CHECK([echo "start" >> CMDFILE]) +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE]) +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE]) +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \ + --vdev="net_tap0,iface=tap0" --file-prefix page0 \ + --single-file-segments -- --cmdline-file=CMDFILE \ + -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 & + +dnl Give settling time to the testpmd processes - NOTE: this is bad form. +sleep 10 + +dnl Check whether TSO is turned off (host side) +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout]) +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[0]) + +dnl Check whether TSO is turned off (guest side) +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \ + $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log],[0],[0]) + +dnl Clean up +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr]) + +dnl Clean up the testpmd now +pkill -f -x -9 'tail -f /dev/null' + +dnl Add vhostuser port (server mode) +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \ + type=dpdkvhostuser], [], + [stdout], [stderr]) +AT_CHECK([ovs-vsctl show], [], [stdout]) + +dnl Execute testpmd in background +on_exit "pkill -f -x -9 'tail -f /dev/null'" +AT_CHECK([echo "show device info all" > CMDFILE]) +AT_CHECK([echo "stop" >> CMDFILE]) +AT_CHECK([echo "port stop 0" >> CMDFILE]) +AT_CHECK([echo "tso set 1500 0" >> CMDFILE]) +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE]) +AT_CHECK([echo "port start 0" >> CMDFILE]) +AT_CHECK([echo "start" >> CMDFILE]) +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE]) +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE]) +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\ + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \ + --vdev="net_tap0,iface=tap0" --file-prefix page0 \ + --single-file-segments -- --cmdline-file=CMDFILE \ + -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 & + +dnl Give settling time to the testpmd processes - NOTE: this is bad form. +sleep 10 + +dnl Check whether TSO is turned off (host side) +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \ + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout]) +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[0]) + +dnl Check whether TSO is turned off (guest side) +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \ + $OVS_RUNDIR/testpmd-dpdkvhostuser0.log],[0],[0]) + +dnl Clean up the testpmd now +pkill -f -x -9 'tail -f /dev/null' + +dnl Clean up +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr]) +OVS_VSWITCHD_STOP() +AT_CLEANUP +dnl --------------------------------------------------------------------------
This patch adds minimal check for userspace-tso in system-dpdk tests, starting with verification on virtio negotiation. Signed-off-by: Gowrishankar Muthukrishnan <gmuthukr@redhat.com> --- v4: - configure offload settings in testpmd --- tests/system-dpdk-macros.at | 17 +++- tests/system-dpdk.at | 228 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 236 insertions(+), 9 deletions(-)