[ovs-dev,v4,6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports

This adds a few unit tests to the 'check-dpdk' subsystem that will
exercise allocations of two network namespaces, PMDs, and the
vhost-user and vhost-user-client code path(separate tests).
As part of the tests, userspace bridge is added and attached to OVS.
Also, the tap devices created are added to the network namespaces.

The ultimatum is to connect the two network namespaces by pinging them.

Signed-off-by: Bala Sankaran <bsankara@redhat.com>
Co-authored-by: Aaron Conole <aconole@redhat.com>
Signed-off-by: Aaron Conole <aconole@redhat.com>
---
 tests/system-dpdk.at | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 151 insertions(+), 1 deletion(-)

On 31/08/2018 16:55, Bala Sankaran wrote:
> This adds a few unit tests to the 'check-dpdk' subsystem that will
> exercise allocations of two network namespaces, PMDs, and the
> vhost-user and vhost-user-client code path(separate tests).
> As part of the tests, userspace bridge is added and attached to OVS.
> Also, the tap devices created are added to the network namespaces.
> 
> The ultimatum is to connect the two network namespaces by pinging them.
> 
> Signed-off-by: Bala Sankaran <bsankara@redhat.com>
> Co-authored-by: Aaron Conole <aconole@redhat.com>
> Signed-off-by: Aaron Conole <aconole@redhat.com>
> ---

Hi Bala,

Thanks for the new version and including the vhostuserclient test - that
test runs OK on my end, and passes successfully (just a note below).

As I mentioned on v3, I've faced some issues getting the vhostuser test
to run on my system. I've tried testpmd binaries compiled against DPDK
18.02, 18.05, 18.08 and 18.11-rc0. Only 18.02 seemed to work.

After spending some time bisecting this, it comes down to the
"--socket-mem" option, and providing some memory for all available
sockets in the system [1]. If one provides "--socket-mem=512" (like
you're doing on the tests below), pre DPDK 18.05, testpmd would fail
with an error mentioning it can't allocate memory for a socket 1 (if
your system has 2 sockets). Post DPDK 18.05, it goes on and doesn't
complain, but in my case it also fails to forward traffic.

So, this begs the question, does the first test pass for you as is,
without a second argument to "--socket-mem"? How many sockets does your
system have?

[1] Note that all ports are allocated, in my system, in socket 0. But
still, I have to allocate as little as "1" to get traffic through.

>  tests/system-dpdk.at | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 151 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index 58dc8aa..baa74da 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -1,7 +1,10 @@
> +m4_define([CONFIGURE_VETH_OFFLOADS],
> +   [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
> +
>  AT_BANNER([OVS-DPDK unit tests])
>  
>  dnl --------------------------------------------------------------------------
> -dnl Check if EAL init is successfull
> +dnl Check if EAL init is successful
>  AT_SETUP([OVS-DPDK datapath - EAL init])
>  AT_KEYWORDS([dpdk])
>  OVS_DPDK_PRE_CHECK()
> @@ -74,3 +77,150 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is probably
>  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
>  AT_CLEANUP
>  dnl --------------------------------------------------------------------------
> +
> +
> +
> +dnl --------------------------------------------------------------------------
> +dnl Ping vhost-user port
> +AT_SETUP([OVS-DPDK datapath - ping vhost-user ports])
> +AT_KEYWORDS([dpdk])
> +OVS_DPDK_PRE_CHECK()
> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
> +          type=dpdkvhostuser], [],
> +         [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Parse log file
> +AT_CHECK([grep "VHOST_CONFIG: vhost-user server: socket created" \
> +          ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "Socket $OVS_RUNDIR/dpdkvhostuser0 created for vhost-user port dpdkvhostuser0" \
> +          ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "VHOST_CONFIG: bind to $OVS_RUNDIR/dpdkvhostuser0" ovs-vswitchd.log], [],
> +         [stdout])
> +
> +dnl Set up namespaces
> +ADD_NAMESPACES(ns1, ns2)
> +
> +dnl Add veth device
> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +tail -f /dev/null | testpmd --socket-mem=512 \
> +           --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> +           --single-file-segments -- -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &

Since this test doesn't require any physical devices, I would use either
the "--no-pci" option or check if any physical devices is bound to DPDK
any skip the test if so. Otherwise DPDK will try to init the driver
there and testpmd will hang as a consequence. Same for the other test below.

Tiago.

> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Move the tap devices to the namespaces
> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> +AT_CHECK([ip link show], [], [stdout], [stderr])
> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> +
> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
> +         [stdout], [stderr])
> +
> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [], [stdout],
> +         [stderr])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is probably not loaded.@d
> +\@Failed to enable flow control@d
> +\@VHOST_CONFIG: recvmsg failed@d
> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostuser0: No such file or directory@d
> +\@Global register is changed during@d
> +\@dpdkvhostuser ports are considered deprecated;  please migrate to dpdkvhostuserclient ports.@d
> +\@failed to enumerate system datapaths: No such file or directory@d
> +\@EAL:   Invalid NUMA socket, default to 0@d
> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !@d
> +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl Ping vhost-user-client port
> +AT_SETUP([OVS-DPDK datapath - ping vhost-user-client ports])
> +AT_KEYWORDS([dpdk])
> +OVS_DPDK_PRE_CHECK()
> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \
> +          dpdkvhostuserclient0 \
> +          type=dpdkvhostuserclient \
> +          options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> +         [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Parse log file
> +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0: reconnecting..." ovs-vswitchd.log], [], [stdout])
> +
> +dnl Set up namespaces
> +ADD_NAMESPACES(ns1, ns2)
> +
> +dnl Add veth device
> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +tail -f /dev/null | testpmd --socket-mem=512 \
> +           --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \
> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> +           --single-file-segments -- -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Move the tap devices to the namespaces
> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> +AT_CHECK([ip link show], [], [stdout], [stderr])
> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> +
> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
> +         [stdout], [stderr])
> +
> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [], [stdout],
> +         [stderr])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is probably not loaded.@d
> +\@Failed to enable flow control@d
> +\@VHOST_CONFIG: recvmsg failed@d
> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No such file or directory@d
> +\@Global register is changed during@d
> +\@dpdkvhostuser ports are considered deprecated;  please migrate to dpdkvhostuserclient ports.@d
> +\@failed to enumerate system datapaths: No such file or directory@d
> +\@EAL:   Invalid NUMA socket, default to 0@d
> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !@d
> +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
>

----- Original Message -----
> From: "Tiago Lam" <tiago.lam@intel.com>
> To: "Bala Sankaran" <bsankara@redhat.com>, dev@openvswitch.org
> Cc: "Aaron Conole" <aconole@redhat.com>, "Ian Stokes" <ian.stokes@intel.com>, "Ciara Loftus" <ciara.loftus@intel.com>
> Sent: Tuesday, 4 September, 2018 6:32:39 AM
> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports
> 
> On 31/08/2018 16:55, Bala Sankaran wrote:
> > This adds a few unit tests to the 'check-dpdk' subsystem that will
> > exercise allocations of two network namespaces, PMDs, and the
> > vhost-user and vhost-user-client code path(separate tests).
> > As part of the tests, userspace bridge is added and attached to OVS.
> > Also, the tap devices created are added to the network namespaces.
> > 
> > The ultimatum is to connect the two network namespaces by pinging them.
> > 
> > Signed-off-by: Bala Sankaran <bsankara@redhat.com>
> > Co-authored-by: Aaron Conole <aconole@redhat.com>
> > Signed-off-by: Aaron Conole <aconole@redhat.com>
> > ---
> 
> Hi Bala,
> 
> Thanks for the new version and including the vhostuserclient test - that
> test runs OK on my end, and passes successfully (just a note below).
> 
> As I mentioned on v3, I've faced some issues getting the vhostuser test
> to run on my system. I've tried testpmd binaries compiled against DPDK
> 18.02, 18.05, 18.08 and 18.11-rc0. Only 18.02 seemed to work.
> 
> After spending some time bisecting this, it comes down to the
> "--socket-mem" option, and providing some memory for all available
> sockets in the system [1]. If one provides "--socket-mem=512" (like
> you're doing on the tests below), pre DPDK 18.05, testpmd would fail
> with an error mentioning it can't allocate memory for a socket 1 (if
> your system has 2 sockets). Post DPDK 18.05, it goes on and doesn't
> complain, but in my case it also fails to forward traffic.
> 
> So, this begs the question, does the first test pass for you as is,
> without a second argument to "--socket-mem"? How many sockets does your
> system have?

Hello Tiago,

I've got 4 sockets in the VM that I tested on. I have been using the
--socket-mem=512 as an argument for testpmd each time I test, I don't
reckon that my tests would work without the argument.


Best,
Bala.

> 
> [1] Note that all ports are allocated, in my system, in socket 0. But
> still, I have to allocate as little as "1" to get traffic through.
> 
> >  tests/system-dpdk.at | 152
> >  ++++++++++++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 151 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> > index 58dc8aa..baa74da 100644
> > --- a/tests/system-dpdk.at
> > +++ b/tests/system-dpdk.at
> > @@ -1,7 +1,10 @@
> > +m4_define([CONFIGURE_VETH_OFFLOADS],
> > +   [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
> > +
> >  AT_BANNER([OVS-DPDK unit tests])
> >  
> >  dnl
> >  --------------------------------------------------------------------------
> > -dnl Check if EAL init is successfull
> > +dnl Check if EAL init is successful
> >  AT_SETUP([OVS-DPDK datapath - EAL init])
> >  AT_KEYWORDS([dpdk])
> >  OVS_DPDK_PRE_CHECK()
> > @@ -74,3 +77,150 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch
> > kernel module is probably
> >  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> >  AT_CLEANUP
> >  dnl
> >  --------------------------------------------------------------------------
> > +
> > +
> > +
> > +dnl
> > --------------------------------------------------------------------------
> > +dnl Ping vhost-user port
> > +AT_SETUP([OVS-DPDK datapath - ping vhost-user ports])
> > +AT_KEYWORDS([dpdk])
> > +OVS_DPDK_PRE_CHECK()
> > +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> > +OVS_DPDK_START()
> > +
> > +dnl Add userspace bridge and attach it to OVS
> > +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> > +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface
> > dpdkvhostuser0 \
> > +          type=dpdkvhostuser], [],
> > +         [stdout], [stderr])
> > +AT_CHECK([ovs-vsctl show], [], [stdout])
> > +
> > +dnl Parse log file
> > +AT_CHECK([grep "VHOST_CONFIG: vhost-user server: socket created" \
> > +          ovs-vswitchd.log], [], [stdout])
> > +AT_CHECK([grep "Socket $OVS_RUNDIR/dpdkvhostuser0 created for vhost-user
> > port dpdkvhostuser0" \
> > +          ovs-vswitchd.log], [], [stdout])
> > +AT_CHECK([grep "VHOST_CONFIG: bind to $OVS_RUNDIR/dpdkvhostuser0"
> > ovs-vswitchd.log], [],
> > +         [stdout])
> > +
> > +dnl Set up namespaces
> > +ADD_NAMESPACES(ns1, ns2)
> > +
> > +dnl Add veth device
> > +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> > +
> > +dnl Execute testpmd in background
> > +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> > +tail -f /dev/null | testpmd --socket-mem=512 \
> > +           --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> > +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> > +           --single-file-segments -- -a
> > >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> 
> Since this test doesn't require any physical devices, I would use either
> the "--no-pci" option or check if any physical devices is bound to DPDK
> any skip the test if so. Otherwise DPDK will try to init the driver
> there and testpmd will hang as a consequence. Same for the other test below.
> 
> Tiago.
> 
> > +
> > +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> > +sleep 10
> > +
> > +dnl Move the tap devices to the namespaces
> > +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> > +AT_CHECK([ip link show], [], [stdout], [stderr])
> > +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> > +
> > +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> > [stderr])
> > +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout], [stderr])
> > +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
> > +         [stdout], [stderr])
> > +
> > +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> > +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> > [stdout],
> > +         [stderr])
> > +
> > +dnl Clean up the testpmd now
> > +pkill -f -x -9 'tail -f /dev/null'
> > +
> > +dnl Clean up
> > +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr])
> > +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is
> > probably not loaded.@d
> > +\@Failed to enable flow control@d
> > +\@VHOST_CONFIG: recvmsg failed@d
> > +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostuser0: No such
> > file or directory@d
> > +\@Global register is changed during@d
> > +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> > dpdkvhostuserclient ports.@d
> > +\@failed to enumerate system datapaths: No such file or directory@d
> > +\@EAL:   Invalid NUMA socket, default to 0@d
> > +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > unreliable clock cycles !@d
> > +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> > +AT_CLEANUP
> > +dnl
> > --------------------------------------------------------------------------
> > +
> > +dnl
> > --------------------------------------------------------------------------
> > +dnl Ping vhost-user-client port
> > +AT_SETUP([OVS-DPDK datapath - ping vhost-user-client ports])
> > +AT_KEYWORDS([dpdk])
> > +OVS_DPDK_PRE_CHECK()
> > +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> > +OVS_DPDK_START()
> > +
> > +dnl Add userspace bridge and attach it to OVS
> > +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> > +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \
> > +          dpdkvhostuserclient0 \
> > +          type=dpdkvhostuserclient \
> > +          options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> > +         [stdout], [stderr])
> > +AT_CHECK([ovs-vsctl show], [], [stdout])
> > +
> > +dnl Parse log file
> > +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created"
> > ovs-vswitchd.log], [], [stdout])
> > +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in
> > 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> > +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0:
> > reconnecting..." ovs-vswitchd.log], [], [stdout])
> > +
> > +dnl Set up namespaces
> > +ADD_NAMESPACES(ns1, ns2)
> > +
> > +dnl Add veth device
> > +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> > +
> > +dnl Execute testpmd in background
> > +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> > +tail -f /dev/null | testpmd --socket-mem=512 \
> > +
> > --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1"
> > \
> > +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> > +           --single-file-segments -- -a
> > >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> > +
> > +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> > +sleep 10
> > +
> > +dnl Move the tap devices to the namespaces
> > +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> > +AT_CHECK([ip link show], [], [stdout], [stderr])
> > +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> > +
> > +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> > [stderr])
> > +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout], [stderr])
> > +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
> > +         [stdout], [stderr])
> > +
> > +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> > +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> > [stdout],
> > +         [stderr])
> > +
> > +dnl Clean up the testpmd now
> > +pkill -f -x -9 'tail -f /dev/null'
> > +
> > +dnl Clean up
> > +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout],
> > [stderr])
> > +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is
> > probably not loaded.@d
> > +\@Failed to enable flow control@d
> > +\@VHOST_CONFIG: recvmsg failed@d
> > +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No such
> > file or directory@d
> > +\@Global register is changed during@d
> > +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> > dpdkvhostuserclient ports.@d
> > +\@failed to enumerate system datapaths: No such file or directory@d
> > +\@EAL:   Invalid NUMA socket, default to 0@d
> > +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > unreliable clock cycles !@d
> > +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> > +AT_CLEANUP
> > +dnl
> > --------------------------------------------------------------------------
> > 
>

On 04/09/2018 18:46, Bala Sankaran wrote:
> ----- Original Message -----
>> From: "Tiago Lam" <tiago.lam@intel.com>
>> To: "Bala Sankaran" <bsankara@redhat.com>, dev@openvswitch.org
>> Cc: "Aaron Conole" <aconole@redhat.com>, "Ian Stokes" <ian.stokes@intel.com>, "Ciara Loftus" <ciara.loftus@intel.com>
>> Sent: Tuesday, 4 September, 2018 6:32:39 AM
>> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports
>>
>> On 31/08/2018 16:55, Bala Sankaran wrote:
>>> This adds a few unit tests to the 'check-dpdk' subsystem that will
>>> exercise allocations of two network namespaces, PMDs, and the
>>> vhost-user and vhost-user-client code path(separate tests).
>>> As part of the tests, userspace bridge is added and attached to OVS.
>>> Also, the tap devices created are added to the network namespaces.
>>>
>>> The ultimatum is to connect the two network namespaces by pinging them.
>>>
>>> Signed-off-by: Bala Sankaran <bsankara@redhat.com>
>>> Co-authored-by: Aaron Conole <aconole@redhat.com>
>>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>>> ---
>>
>> Hi Bala,
>>
>> Thanks for the new version and including the vhostuserclient test - that
>> test runs OK on my end, and passes successfully (just a note below).
>>
>> As I mentioned on v3, I've faced some issues getting the vhostuser test
>> to run on my system. I've tried testpmd binaries compiled against DPDK
>> 18.02, 18.05, 18.08 and 18.11-rc0. Only 18.02 seemed to work.
>>
>> After spending some time bisecting this, it comes down to the
>> "--socket-mem" option, and providing some memory for all available
>> sockets in the system [1]. If one provides "--socket-mem=512" (like
>> you're doing on the tests below), pre DPDK 18.05, testpmd would fail
>> with an error mentioning it can't allocate memory for a socket 1 (if
>> your system has 2 sockets). Post DPDK 18.05, it goes on and doesn't
>> complain, but in my case it also fails to forward traffic.
>>
>> So, this begs the question, does the first test pass for you as is,
>> without a second argument to "--socket-mem"? How many sockets does your
>> system have?
> 
> Hello Tiago,
> 
> I've got 4 sockets in the VM that I tested on. I have been using the
> --socket-mem=512 as an argument for testpmd each time I test, I don't
> reckon that my tests would work without the argument.

Is that what `lscpu` reports under "NUMA nodes(s)", if executed inside
the VM?

My argument is not around removing "--socket-mem=", but assigning a
value to each available socket. If you look into the OVS_DPDK_START
macro, in tests/system-dpdk-macros.at, it is doing exactly that by using
`lscpu` and grep'ing for "NUMA node(s)". Otherwise I think this test
will fail on several systems and pass on some others.

Also, I think you've missed my other comment below, about using "--no-pci".

Tiago.

> 
> 
> Best,
> Bala.
> 
>>
>> [1] Note that all ports are allocated, in my system, in socket 0. But
>> still, I have to allocate as little as "1" to get traffic through.
>>
>>>  tests/system-dpdk.at | 152
>>>  ++++++++++++++++++++++++++++++++++++++++++++++++++-
>>>  1 file changed, 151 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
>>> index 58dc8aa..baa74da 100644
>>> --- a/tests/system-dpdk.at
>>> +++ b/tests/system-dpdk.at
>>> @@ -1,7 +1,10 @@
>>> +m4_define([CONFIGURE_VETH_OFFLOADS],
>>> +   [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
>>> +
>>>  AT_BANNER([OVS-DPDK unit tests])
>>>  
>>>  dnl
>>>  --------------------------------------------------------------------------
>>> -dnl Check if EAL init is successfull
>>> +dnl Check if EAL init is successful
>>>  AT_SETUP([OVS-DPDK datapath - EAL init])
>>>  AT_KEYWORDS([dpdk])
>>>  OVS_DPDK_PRE_CHECK()
>>> @@ -74,3 +77,150 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch
>>> kernel module is probably
>>>  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
>>>  AT_CLEANUP
>>>  dnl
>>>  --------------------------------------------------------------------------
>>> +
>>> +
>>> +
>>> +dnl
>>> --------------------------------------------------------------------------
>>> +dnl Ping vhost-user port
>>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user ports])
>>> +AT_KEYWORDS([dpdk])
>>> +OVS_DPDK_PRE_CHECK()
>>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
>>> +OVS_DPDK_START()
>>> +
>>> +dnl Add userspace bridge and attach it to OVS
>>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
>>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface
>>> dpdkvhostuser0 \
>>> +          type=dpdkvhostuser], [],
>>> +         [stdout], [stderr])
>>> +AT_CHECK([ovs-vsctl show], [], [stdout])
>>> +
>>> +dnl Parse log file
>>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user server: socket created" \
>>> +          ovs-vswitchd.log], [], [stdout])
>>> +AT_CHECK([grep "Socket $OVS_RUNDIR/dpdkvhostuser0 created for vhost-user
>>> port dpdkvhostuser0" \
>>> +          ovs-vswitchd.log], [], [stdout])
>>> +AT_CHECK([grep "VHOST_CONFIG: bind to $OVS_RUNDIR/dpdkvhostuser0"
>>> ovs-vswitchd.log], [],
>>> +         [stdout])
>>> +
>>> +dnl Set up namespaces
>>> +ADD_NAMESPACES(ns1, ns2)
>>> +
>>> +dnl Add veth device
>>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
>>> +
>>> +dnl Execute testpmd in background
>>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
>>> +tail -f /dev/null | testpmd --socket-mem=512 \
>>> +           --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
>>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
>>> +           --single-file-segments -- -a
>>>> $OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
>>
>> Since this test doesn't require any physical devices, I would use either
>> the "--no-pci" option or check if any physical devices is bound to DPDK
>> any skip the test if so. Otherwise DPDK will try to init the driver
>> there and testpmd will hang as a consequence. Same for the other test below.
>>
>> Tiago.
>>
>>> +
>>> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
>>> +sleep 10
>>> +
>>> +dnl Move the tap devices to the namespaces
>>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
>>> +AT_CHECK([ip link show], [], [stdout], [stderr])
>>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
>>> +
>>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
>>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
>>> [stderr])
>>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout], [stderr])
>>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
>>> +         [stdout], [stderr])
>>> +
>>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
>>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
>>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
>>> [stdout],
>>> +         [stderr])
>>> +
>>> +dnl Clean up the testpmd now
>>> +pkill -f -x -9 'tail -f /dev/null'
>>> +
>>> +dnl Clean up
>>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr])
>>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is
>>> probably not loaded.@d
>>> +\@Failed to enable flow control@d
>>> +\@VHOST_CONFIG: recvmsg failed@d
>>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostuser0: No such
>>> file or directory@d
>>> +\@Global register is changed during@d
>>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
>>> dpdkvhostuserclient ports.@d
>>> +\@failed to enumerate system datapaths: No such file or directory@d
>>> +\@EAL:   Invalid NUMA socket, default to 0@d
>>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>>> unreliable clock cycles !@d
>>> +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
>>> +AT_CLEANUP
>>> +dnl
>>> --------------------------------------------------------------------------
>>> +
>>> +dnl
>>> --------------------------------------------------------------------------
>>> +dnl Ping vhost-user-client port
>>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user-client ports])
>>> +AT_KEYWORDS([dpdk])
>>> +OVS_DPDK_PRE_CHECK()
>>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
>>> +OVS_DPDK_START()
>>> +
>>> +dnl Add userspace bridge and attach it to OVS
>>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
>>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \
>>> +          dpdkvhostuserclient0 \
>>> +          type=dpdkvhostuserclient \
>>> +          options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
>>> +         [stdout], [stderr])
>>> +AT_CHECK([ovs-vsctl show], [], [stdout])
>>> +
>>> +dnl Parse log file
>>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created"
>>> ovs-vswitchd.log], [], [stdout])
>>> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in
>>> 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
>>> +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0:
>>> reconnecting..." ovs-vswitchd.log], [], [stdout])
>>> +
>>> +dnl Set up namespaces
>>> +ADD_NAMESPACES(ns1, ns2)
>>> +
>>> +dnl Add veth device
>>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
>>> +
>>> +dnl Execute testpmd in background
>>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
>>> +tail -f /dev/null | testpmd --socket-mem=512 \
>>> +
>>> --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1"
>>> \
>>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
>>> +           --single-file-segments -- -a
>>>> $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
>>> +
>>> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
>>> +sleep 10
>>> +
>>> +dnl Move the tap devices to the namespaces
>>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
>>> +AT_CHECK([ip link show], [], [stdout], [stderr])
>>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
>>> +
>>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
>>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
>>> [stderr])
>>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout], [stderr])
>>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
>>> +         [stdout], [stderr])
>>> +
>>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
>>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
>>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
>>> [stdout],
>>> +         [stderr])
>>> +
>>> +dnl Clean up the testpmd now
>>> +pkill -f -x -9 'tail -f /dev/null'
>>> +
>>> +dnl Clean up
>>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout],
>>> [stderr])
>>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is
>>> probably not loaded.@d
>>> +\@Failed to enable flow control@d
>>> +\@VHOST_CONFIG: recvmsg failed@d
>>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No such
>>> file or directory@d
>>> +\@Global register is changed during@d
>>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
>>> dpdkvhostuserclient ports.@d
>>> +\@failed to enumerate system datapaths: No such file or directory@d
>>> +\@EAL:   Invalid NUMA socket, default to 0@d
>>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>>> unreliable clock cycles !@d
>>> +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
>>> +AT_CLEANUP
>>> +dnl
>>> --------------------------------------------------------------------------
>>>
>>

----- Original Message -----
> From: "Tiago Lam" <tiago.lam@intel.com>
> To: "Bala Sankaran" <bsankara@redhat.com>
> Cc: dev@openvswitch.org, "Aaron Conole" <aconole@redhat.com>, "Ian Stokes" <ian.stokes@intel.com>, "Ciara Loftus"
> <ciara.loftus@intel.com>
> Sent: Wednesday, 5 September, 2018 3:24:46 AM
> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports
> 
> 
> 
> On 04/09/2018 18:46, Bala Sankaran wrote:
> > ----- Original Message -----
> >> From: "Tiago Lam" <tiago.lam@intel.com>
> >> To: "Bala Sankaran" <bsankara@redhat.com>, dev@openvswitch.org
> >> Cc: "Aaron Conole" <aconole@redhat.com>, "Ian Stokes"
> >> <ian.stokes@intel.com>, "Ciara Loftus" <ciara.loftus@intel.com>
> >> Sent: Tuesday, 4 September, 2018 6:32:39 AM
> >> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via
> >> dpdkvhostuser ports
> >>
> >> On 31/08/2018 16:55, Bala Sankaran wrote:
> >>> This adds a few unit tests to the 'check-dpdk' subsystem that will
> >>> exercise allocations of two network namespaces, PMDs, and the
> >>> vhost-user and vhost-user-client code path(separate tests).
> >>> As part of the tests, userspace bridge is added and attached to OVS.
> >>> Also, the tap devices created are added to the network namespaces.
> >>>
> >>> The ultimatum is to connect the two network namespaces by pinging them.
> >>>
> >>> Signed-off-by: Bala Sankaran <bsankara@redhat.com>
> >>> Co-authored-by: Aaron Conole <aconole@redhat.com>
> >>> Signed-off-by: Aaron Conole <aconole@redhat.com>
> >>> ---
> >>
> >> Hi Bala,
> >>
> >> Thanks for the new version and including the vhostuserclient test - that
> >> test runs OK on my end, and passes successfully (just a note below).
> >>
> >> As I mentioned on v3, I've faced some issues getting the vhostuser test
> >> to run on my system. I've tried testpmd binaries compiled against DPDK
> >> 18.02, 18.05, 18.08 and 18.11-rc0. Only 18.02 seemed to work.
> >>
> >> After spending some time bisecting this, it comes down to the
> >> "--socket-mem" option, and providing some memory for all available
> >> sockets in the system [1]. If one provides "--socket-mem=512" (like
> >> you're doing on the tests below), pre DPDK 18.05, testpmd would fail
> >> with an error mentioning it can't allocate memory for a socket 1 (if
> >> your system has 2 sockets). Post DPDK 18.05, it goes on and doesn't
> >> complain, but in my case it also fails to forward traffic.
> >>
> >> So, this begs the question, does the first test pass for you as is,
> >> without a second argument to "--socket-mem"? How many sockets does your
> >> system have?
> > 
> > Hello Tiago,
> > 
> > I've got 4 sockets in the VM that I tested on. I have been using the
> > --socket-mem=512 as an argument for testpmd each time I test, I don't
> > reckon that my tests would work without the argument.
Hi Tiago,

> 
> Is that what `lscpu` reports under "NUMA nodes(s)", if executed inside
> the VM?
> 
> My argument is not around removing "--socket-mem=", but assigning a
> value to each available socket. If you look into the OVS_DPDK_START
> macro, in tests/system-dpdk-macros.at, it is doing exactly that by using
> `lscpu` and grep'ing for "NUMA node(s)". Otherwise I think this test
> will fail on several systems and pass on some others.

Apologies for the delayed response. I understand this, I will add this up.

> 
> Also, I think you've missed my other comment below, about using "--no-pci".
> 
> Tiago.
> 
> > 
> > 
> > Best,
> > Bala.
> > 
> >>
> >> [1] Note that all ports are allocated, in my system, in socket 0. But
> >> still, I have to allocate as little as "1" to get traffic through.
> >>
> >>>  tests/system-dpdk.at | 152
> >>>  ++++++++++++++++++++++++++++++++++++++++++++++++++-
> >>>  1 file changed, 151 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> >>> index 58dc8aa..baa74da 100644
> >>> --- a/tests/system-dpdk.at
> >>> +++ b/tests/system-dpdk.at
> >>> @@ -1,7 +1,10 @@
> >>> +m4_define([CONFIGURE_VETH_OFFLOADS],
> >>> +   [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
> >>> +
> >>>  AT_BANNER([OVS-DPDK unit tests])
> >>>  
> >>>  dnl
> >>>  --------------------------------------------------------------------------
> >>> -dnl Check if EAL init is successfull
> >>> +dnl Check if EAL init is successful
> >>>  AT_SETUP([OVS-DPDK datapath - EAL init])
> >>>  AT_KEYWORDS([dpdk])
> >>>  OVS_DPDK_PRE_CHECK()
> >>> @@ -74,3 +77,150 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open
> >>> vSwitch
> >>> kernel module is probably
> >>>  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> >>>  AT_CLEANUP
> >>>  dnl
> >>>  --------------------------------------------------------------------------
> >>> +
> >>> +
> >>> +
> >>> +dnl
> >>> --------------------------------------------------------------------------
> >>> +dnl Ping vhost-user port
> >>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user ports])
> >>> +AT_KEYWORDS([dpdk])
> >>> +OVS_DPDK_PRE_CHECK()
> >>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> >>> +OVS_DPDK_START()
> >>> +
> >>> +dnl Add userspace bridge and attach it to OVS
> >>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10
> >>> datapath_type=netdev])
> >>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface
> >>> dpdkvhostuser0 \
> >>> +          type=dpdkvhostuser], [],
> >>> +         [stdout], [stderr])
> >>> +AT_CHECK([ovs-vsctl show], [], [stdout])
> >>> +
> >>> +dnl Parse log file
> >>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user server: socket created" \
> >>> +          ovs-vswitchd.log], [], [stdout])
> >>> +AT_CHECK([grep "Socket $OVS_RUNDIR/dpdkvhostuser0 created for vhost-user
> >>> port dpdkvhostuser0" \
> >>> +          ovs-vswitchd.log], [], [stdout])
> >>> +AT_CHECK([grep "VHOST_CONFIG: bind to $OVS_RUNDIR/dpdkvhostuser0"
> >>> ovs-vswitchd.log], [],
> >>> +         [stdout])
> >>> +
> >>> +dnl Set up namespaces
> >>> +ADD_NAMESPACES(ns1, ns2)
> >>> +
> >>> +dnl Add veth device
> >>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> >>> +
> >>> +dnl Execute testpmd in background
> >>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> >>> +tail -f /dev/null | testpmd --socket-mem=512 \
> >>> +           --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> >>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> >>> +           --single-file-segments -- -a
> >>>> $OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> >>
> >> Since this test doesn't require any physical devices, I would use either
> >> the "--no-pci" option or check if any physical devices is bound to DPDK
> >> any skip the test if so. Otherwise DPDK will try to init the driver
> >> there and testpmd will hang as a consequence. Same for the other test
> >> below.

I will add the --no-pci option as well to my tests and submit a v5.

Thank you.

Best,
Bala.

> >>
> >> Tiago.
> >>
> >>> +
> >>> +dnl Give settling time to the testpmd processes - NOTE: this is bad
> >>> form.
> >>> +sleep 10
> >>> +
> >>> +dnl Move the tap devices to the namespaces
> >>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> >>> +AT_CHECK([ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> >>> +
> >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> >>> [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout],
> >>> [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
> >>> +         [stdout], [stderr])
> >>> +
> >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> >>> [stdout],
> >>> +         [stderr])
> >>> +
> >>> +dnl Clean up the testpmd now
> >>> +pkill -f -x -9 'tail -f /dev/null'
> >>> +
> >>> +dnl Clean up
> >>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout],
> >>> [stderr])
> >>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is
> >>> probably not loaded.@d
> >>> +\@Failed to enable flow control@d
> >>> +\@VHOST_CONFIG: recvmsg failed@d
> >>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostuser0: No such
> >>> file or directory@d
> >>> +\@Global register is changed during@d
> >>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> >>> dpdkvhostuserclient ports.@d
> >>> +\@failed to enumerate system datapaths: No such file or directory@d
> >>> +\@EAL:   Invalid NUMA socket, default to 0@d
> >>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> >>> unreliable clock cycles !@d
> >>> +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> >>> +AT_CLEANUP
> >>> +dnl
> >>> --------------------------------------------------------------------------
> >>> +
> >>> +dnl
> >>> --------------------------------------------------------------------------
> >>> +dnl Ping vhost-user-client port
> >>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user-client ports])
> >>> +AT_KEYWORDS([dpdk])
> >>> +OVS_DPDK_PRE_CHECK()
> >>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> >>> +OVS_DPDK_START()
> >>> +
> >>> +dnl Add userspace bridge and attach it to OVS
> >>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10
> >>> datapath_type=netdev])
> >>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface
> >>> \
> >>> +          dpdkvhostuserclient0 \
> >>> +          type=dpdkvhostuserclient \
> >>> +          options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> >>> +         [stdout], [stderr])
> >>> +AT_CHECK([ovs-vsctl show], [], [stdout])
> >>> +
> >>> +dnl Parse log file
> >>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created"
> >>> ovs-vswitchd.log], [], [stdout])
> >>> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in
> >>> 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> >>> +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0:
> >>> reconnecting..." ovs-vswitchd.log], [], [stdout])
> >>> +
> >>> +dnl Set up namespaces
> >>> +ADD_NAMESPACES(ns1, ns2)
> >>> +
> >>> +dnl Add veth device
> >>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> >>> +
> >>> +dnl Execute testpmd in background
> >>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> >>> +tail -f /dev/null | testpmd --socket-mem=512 \
> >>> +
> >>> --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1"
> >>> \
> >>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> >>> +           --single-file-segments -- -a
> >>>> $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> >>> +
> >>> +dnl Give settling time to the testpmd processes - NOTE: this is bad
> >>> form.
> >>> +sleep 10
> >>> +
> >>> +dnl Move the tap devices to the namespaces
> >>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> >>> +AT_CHECK([ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> >>> +
> >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> >>> [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout],
> >>> [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
> >>> +         [stdout], [stderr])
> >>> +
> >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> >>> [stdout],
> >>> +         [stderr])
> >>> +
> >>> +dnl Clean up the testpmd now
> >>> +pkill -f -x -9 'tail -f /dev/null'
> >>> +
> >>> +dnl Clean up
> >>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout],
> >>> [stderr])
> >>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is
> >>> probably not loaded.@d
> >>> +\@Failed to enable flow control@d
> >>> +\@VHOST_CONFIG: recvmsg failed@d
> >>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No
> >>> such
> >>> file or directory@d
> >>> +\@Global register is changed during@d
> >>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> >>> dpdkvhostuserclient ports.@d
> >>> +\@failed to enumerate system datapaths: No such file or directory@d
> >>> +\@EAL:   Invalid NUMA socket, default to 0@d
> >>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> >>> unreliable clock cycles !@d
> >>> +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> >>> +AT_CLEANUP
> >>> +dnl
> >>> --------------------------------------------------------------------------
> >>>
> >>
>

Hello Tiago,

Here's an incremental diff of the patch 6 which I tested. It looks to pass the tests. I would be submitting the 
new version (v5) applying these changes:

diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
index baa74da..a9247f8 100644
--- a/tests/system-dpdk.at
+++ b/tests/system-dpdk.at
@@ -88,6 +88,10 @@ OVS_DPDK_PRE_CHECK()
 AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
 OVS_DPDK_START()
 
+dnl Find number of sockets
+AT_CHECK([lscpu], [], [stdout])
+AT_CHECK([cat stdout | grep "Socket(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > SOCKET_MEM])
+
 dnl Add userspace bridge and attach it to OVS
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
 AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
@@ -111,7 +115,7 @@ ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
 
 dnl Execute testpmd in background
 on_exit "pkill -f -x -9 'tail -f /dev/null'"
-tail -f /dev/null | testpmd --socket-mem=512 \
+tail -f /dev/null | testpmd --socket-mem="$(cat SOCKET_MEM)" --no-pci\
            --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
            --vdev="net_tap0,iface=tap0" --file-prefix page0 \
            --single-file-segments -- -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
@@ -183,7 +187,7 @@ ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
 
 dnl Execute testpmd in background
 on_exit "pkill -f -x -9 'tail -f /dev/null'"
-tail -f /dev/null | testpmd --socket-mem=512 \
+tail -f /dev/null | testpmd --socket-mem="$(cat SOCKET_MEM)" --no-pci\
            --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \
            --vdev="net_tap0,iface=tap0" --file-prefix page0 \
            --single-file-segments -- -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &


Before I do so, I had a question for you.

Do you suggest that we have the socket-mem option at all? Because it could vary for each system, and also, the latest 
versions of DPDK would dynamically allocate the socket-memory, but earlier versions might have had the socket-memory
option hard-coded - which could in turn be a different value in each system that the tests run on.

Thanks.
Bala.

----- Original Message -----
> From: "Bala Sankaran" <bsankara@redhat.com>
> To: "Tiago Lam" <tiago.lam@intel.com>
> Cc: dev@openvswitch.org, "Aaron Conole" <aconole@redhat.com>, "Ian Stokes" <ian.stokes@intel.com>, "Ciara Loftus"
> <ciara.loftus@intel.com>
> Sent: Monday, 10 September, 2018 9:22:13 AM
> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports
> 
> ----- Original Message -----
> > From: "Tiago Lam" <tiago.lam@intel.com>
> > To: "Bala Sankaran" <bsankara@redhat.com>
> > Cc: dev@openvswitch.org, "Aaron Conole" <aconole@redhat.com>, "Ian Stokes"
> > <ian.stokes@intel.com>, "Ciara Loftus"
> > <ciara.loftus@intel.com>
> > Sent: Wednesday, 5 September, 2018 3:24:46 AM
> > Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via
> > dpdkvhostuser ports
> > 
> > 
> > 
> > On 04/09/2018 18:46, Bala Sankaran wrote:
> > > ----- Original Message -----
> > >> From: "Tiago Lam" <tiago.lam@intel.com>
> > >> To: "Bala Sankaran" <bsankara@redhat.com>, dev@openvswitch.org
> > >> Cc: "Aaron Conole" <aconole@redhat.com>, "Ian Stokes"
> > >> <ian.stokes@intel.com>, "Ciara Loftus" <ciara.loftus@intel.com>
> > >> Sent: Tuesday, 4 September, 2018 6:32:39 AM
> > >> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via
> > >> dpdkvhostuser ports
> > >>
> > >> On 31/08/2018 16:55, Bala Sankaran wrote:
> > >>> This adds a few unit tests to the 'check-dpdk' subsystem that will
> > >>> exercise allocations of two network namespaces, PMDs, and the
> > >>> vhost-user and vhost-user-client code path(separate tests).
> > >>> As part of the tests, userspace bridge is added and attached to OVS.
> > >>> Also, the tap devices created are added to the network namespaces.
> > >>>
> > >>> The ultimatum is to connect the two network namespaces by pinging them.
> > >>>
> > >>> Signed-off-by: Bala Sankaran <bsankara@redhat.com>
> > >>> Co-authored-by: Aaron Conole <aconole@redhat.com>
> > >>> Signed-off-by: Aaron Conole <aconole@redhat.com>
> > >>> ---
> > >>
> > >> Hi Bala,
> > >>
> > >> Thanks for the new version and including the vhostuserclient test - that
> > >> test runs OK on my end, and passes successfully (just a note below).
> > >>
> > >> As I mentioned on v3, I've faced some issues getting the vhostuser test
> > >> to run on my system. I've tried testpmd binaries compiled against DPDK
> > >> 18.02, 18.05, 18.08 and 18.11-rc0. Only 18.02 seemed to work.
> > >>
> > >> After spending some time bisecting this, it comes down to the
> > >> "--socket-mem" option, and providing some memory for all available
> > >> sockets in the system [1]. If one provides "--socket-mem=512" (like
> > >> you're doing on the tests below), pre DPDK 18.05, testpmd would fail
> > >> with an error mentioning it can't allocate memory for a socket 1 (if
> > >> your system has 2 sockets). Post DPDK 18.05, it goes on and doesn't
> > >> complain, but in my case it also fails to forward traffic.
> > >>
> > >> So, this begs the question, does the first test pass for you as is,
> > >> without a second argument to "--socket-mem"? How many sockets does your
> > >> system have?
> > > 
> > > Hello Tiago,
> > > 
> > > I've got 4 sockets in the VM that I tested on. I have been using the
> > > --socket-mem=512 as an argument for testpmd each time I test, I don't
> > > reckon that my tests would work without the argument.
> Hi Tiago,
> 
> > 
> > Is that what `lscpu` reports under "NUMA nodes(s)", if executed inside
> > the VM?
> > 
> > My argument is not around removing "--socket-mem=", but assigning a
> > value to each available socket. If you look into the OVS_DPDK_START
> > macro, in tests/system-dpdk-macros.at, it is doing exactly that by using
> > `lscpu` and grep'ing for "NUMA node(s)". Otherwise I think this test
> > will fail on several systems and pass on some others.
> 
> Apologies for the delayed response. I understand this, I will add this up.
> 
> > 
> > Also, I think you've missed my other comment below, about using "--no-pci".
> > 
> > Tiago.
> > 
> > > 
> > > 
> > > Best,
> > > Bala.
> > > 
> > >>
> > >> [1] Note that all ports are allocated, in my system, in socket 0. But
> > >> still, I have to allocate as little as "1" to get traffic through.
> > >>
> > >>>  tests/system-dpdk.at | 152
> > >>>  ++++++++++++++++++++++++++++++++++++++++++++++++++-
> > >>>  1 file changed, 151 insertions(+), 1 deletion(-)
> > >>>
> > >>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> > >>> index 58dc8aa..baa74da 100644
> > >>> --- a/tests/system-dpdk.at
> > >>> +++ b/tests/system-dpdk.at
> > >>> @@ -1,7 +1,10 @@
> > >>> +m4_define([CONFIGURE_VETH_OFFLOADS],
> > >>> +   [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
> > >>> +
> > >>>  AT_BANNER([OVS-DPDK unit tests])
> > >>>  
> > >>>  dnl
> > >>>  --------------------------------------------------------------------------
> > >>> -dnl Check if EAL init is successfull
> > >>> +dnl Check if EAL init is successful
> > >>>  AT_SETUP([OVS-DPDK datapath - EAL init])
> > >>>  AT_KEYWORDS([dpdk])
> > >>>  OVS_DPDK_PRE_CHECK()
> > >>> @@ -74,3 +77,150 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open
> > >>> vSwitch
> > >>> kernel module is probably
> > >>>  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> > >>>  AT_CLEANUP
> > >>>  dnl
> > >>>  --------------------------------------------------------------------------
> > >>> +
> > >>> +
> > >>> +
> > >>> +dnl
> > >>> --------------------------------------------------------------------------
> > >>> +dnl Ping vhost-user port
> > >>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user ports])
> > >>> +AT_KEYWORDS([dpdk])
> > >>> +OVS_DPDK_PRE_CHECK()
> > >>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> > >>> +OVS_DPDK_START()
> > >>> +
> > >>> +dnl Add userspace bridge and attach it to OVS
> > >>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10
> > >>> datapath_type=netdev])
> > >>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface
> > >>> dpdkvhostuser0 \
> > >>> +          type=dpdkvhostuser], [],
> > >>> +         [stdout], [stderr])
> > >>> +AT_CHECK([ovs-vsctl show], [], [stdout])
> > >>> +
> > >>> +dnl Parse log file
> > >>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user server: socket created" \
> > >>> +          ovs-vswitchd.log], [], [stdout])
> > >>> +AT_CHECK([grep "Socket $OVS_RUNDIR/dpdkvhostuser0 created for
> > >>> vhost-user
> > >>> port dpdkvhostuser0" \
> > >>> +          ovs-vswitchd.log], [], [stdout])
> > >>> +AT_CHECK([grep "VHOST_CONFIG: bind to $OVS_RUNDIR/dpdkvhostuser0"
> > >>> ovs-vswitchd.log], [],
> > >>> +         [stdout])
> > >>> +
> > >>> +dnl Set up namespaces
> > >>> +ADD_NAMESPACES(ns1, ns2)
> > >>> +
> > >>> +dnl Add veth device
> > >>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> > >>> +
> > >>> +dnl Execute testpmd in background
> > >>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> > >>> +tail -f /dev/null | testpmd --socket-mem=512 \
> > >>> +           --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> > >>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> > >>> +           --single-file-segments -- -a
> > >>>> $OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> > >>
> > >> Since this test doesn't require any physical devices, I would use either
> > >> the "--no-pci" option or check if any physical devices is bound to DPDK
> > >> any skip the test if so. Otherwise DPDK will try to init the driver
> > >> there and testpmd will hang as a consequence. Same for the other test
> > >> below.
> 
> I will add the --no-pci option as well to my tests and submit a v5.
> 
> Thank you.
> 
> Best,
> Bala.
> 
> > >>
> > >> Tiago.
> > >>
> > >>> +
> > >>> +dnl Give settling time to the testpmd processes - NOTE: this is bad
> > >>> form.
> > >>> +sleep 10
> > >>> +
> > >>> +dnl Move the tap devices to the namespaces
> > >>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> > >>> +
> > >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> > >>> [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout],
> > >>> [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0],
> > >>> [],
> > >>> +         [stdout], [stderr])
> > >>> +
> > >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> > >>> [stdout],
> > >>> +         [stderr])
> > >>> +
> > >>> +dnl Clean up the testpmd now
> > >>> +pkill -f -x -9 'tail -f /dev/null'
> > >>> +
> > >>> +dnl Clean up
> > >>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout],
> > >>> [stderr])
> > >>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module
> > >>> is
> > >>> probably not loaded.@d
> > >>> +\@Failed to enable flow control@d
> > >>> +\@VHOST_CONFIG: recvmsg failed@d
> > >>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostuser0: No
> > >>> such
> > >>> file or directory@d
> > >>> +\@Global register is changed during@d
> > >>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> > >>> dpdkvhostuserclient ports.@d
> > >>> +\@failed to enumerate system datapaths: No such file or directory@d
> > >>> +\@EAL:   Invalid NUMA socket, default to 0@d
> > >>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > >>> unreliable clock cycles !@d
> > >>> +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> > >>> +AT_CLEANUP
> > >>> +dnl
> > >>> --------------------------------------------------------------------------
> > >>> +
> > >>> +dnl
> > >>> --------------------------------------------------------------------------
> > >>> +dnl Ping vhost-user-client port
> > >>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user-client ports])
> > >>> +AT_KEYWORDS([dpdk])
> > >>> +OVS_DPDK_PRE_CHECK()
> > >>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> > >>> +OVS_DPDK_START()
> > >>> +
> > >>> +dnl Add userspace bridge and attach it to OVS
> > >>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10
> > >>> datapath_type=netdev])
> > >>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set
> > >>> Interface
> > >>> \
> > >>> +          dpdkvhostuserclient0 \
> > >>> +          type=dpdkvhostuserclient \
> > >>> +          options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> > >>> +         [stdout], [stderr])
> > >>> +AT_CHECK([ovs-vsctl show], [], [stdout])
> > >>> +
> > >>> +dnl Parse log file
> > >>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created"
> > >>> ovs-vswitchd.log], [], [stdout])
> > >>> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in
> > >>> 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> > >>> +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0:
> > >>> reconnecting..." ovs-vswitchd.log], [], [stdout])
> > >>> +
> > >>> +dnl Set up namespaces
> > >>> +ADD_NAMESPACES(ns1, ns2)
> > >>> +
> > >>> +dnl Add veth device
> > >>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> > >>> +
> > >>> +dnl Execute testpmd in background
> > >>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> > >>> +tail -f /dev/null | testpmd --socket-mem=512 \
> > >>> +
> > >>> --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1"
> > >>> \
> > >>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> > >>> +           --single-file-segments -- -a
> > >>>> $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> > >>> +
> > >>> +dnl Give settling time to the testpmd processes - NOTE: this is bad
> > >>> form.
> > >>> +sleep 10
> > >>> +
> > >>> +dnl Move the tap devices to the namespaces
> > >>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> > >>> +
> > >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> > >>> [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout],
> > >>> [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0],
> > >>> [],
> > >>> +         [stdout], [stderr])
> > >>> +
> > >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> > >>> [stdout],
> > >>> +         [stderr])
> > >>> +
> > >>> +dnl Clean up the testpmd now
> > >>> +pkill -f -x -9 'tail -f /dev/null'
> > >>> +
> > >>> +dnl Clean up
> > >>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout],
> > >>> [stderr])
> > >>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module
> > >>> is
> > >>> probably not loaded.@d
> > >>> +\@Failed to enable flow control@d
> > >>> +\@VHOST_CONFIG: recvmsg failed@d
> > >>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No
> > >>> such
> > >>> file or directory@d
> > >>> +\@Global register is changed during@d
> > >>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> > >>> dpdkvhostuserclient ports.@d
> > >>> +\@failed to enumerate system datapaths: No such file or directory@d
> > >>> +\@EAL:   Invalid NUMA socket, default to 0@d
> > >>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > >>> unreliable clock cycles !@d
> > >>> +\@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> > >>> +AT_CLEANUP
> > >>> +dnl
> > >>> --------------------------------------------------------------------------
> > >>>
> > >>
> >

On 10/09/2018 16:47, Bala Sankaran wrote:
> Hello Tiago,
> 
> Here's an incremental diff of the patch 6 which I tested. It looks to pass the tests. I would be submitting the 
> new version (v5) applying these changes:
>
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index baa74da..a9247f8 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -88,6 +88,10 @@ OVS_DPDK_PRE_CHECK()
>  AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
>  OVS_DPDK_START()
>  
> +dnl Find number of sockets
> +AT_CHECK([lscpu], [], [stdout])
> +AT_CHECK([cat stdout | grep "Socket(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > SOCKET_MEM])

Hi Bala,

Thanks for the incremental.

Any specific reason to use the result of "Socket(s)" here? I'd use the
same "NUMA node(s)" instead, as the OVS_DPDK_START is doing (in both
cases the result is going to be passed to the "--socket-mem" option).

Also, I think it would be preferable if you'd use a different file to
store the information, instead of overriding the same SOCKET_MEM file
that OVS_DPDK_START sets.

> +
>  dnl Add userspace bridge and attach it to OVS
>  AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
>  AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
> @@ -111,7 +115,7 @@ ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
>  
>  dnl Execute testpmd in background
>  on_exit "pkill -f -x -9 'tail -f /dev/null'"
> -tail -f /dev/null | testpmd --socket-mem=512 \
> +tail -f /dev/null | testpmd --socket-mem="$(cat SOCKET_MEM)" --no-pci\
>             --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
>             --vdev="net_tap0,iface=tap0" --file-prefix page0 \
>             --single-file-segments -- -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> @@ -183,7 +187,7 @@ ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
>  
>  dnl Execute testpmd in background
>  on_exit "pkill -f -x -9 'tail -f /dev/null'"
> -tail -f /dev/null | testpmd --socket-mem=512 \
> +tail -f /dev/null | testpmd --socket-mem="$(cat SOCKET_MEM)" --no-pci\
>             --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \
>             --vdev="net_tap0,iface=tap0" --file-prefix page0 \
>             --single-file-segments -- -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> 
> 
> Before I do so, I had a question for you.
> 
> Do you suggest that we have the socket-mem option at all? Because it could vary for each system, and also, the latest 
> versions of DPDK would dynamically allocate the socket-memory, but earlier versions might have had the socket-memory
> option hard-coded - which could in turn be a different value in each system that the tests run on.

I had the same question myself, because it would make a lot more sense
to just drop it (the "--socket-mem" option, that is). However, it
doesn't seem to work either. The only way I could get it to work on my
system was to provide an argument to all available nodes (I'm not sure
this is the intended behavior though, from DPDK 18.05+, it would require
further investigation). Does it work on your end, if running on a system
with multiple NUMA nodes?

Hope this helps,
Tiago.

----- Original Message -----
> From: "Tiago Lam" <tiago.lam@intel.com>
> To: "Bala Sankaran" <bsankara@redhat.com>
> Cc: dev@openvswitch.org, "Aaron Conole" <aconole@redhat.com>, "Ian Stokes" <ian.stokes@intel.com>, "Ciara Loftus"
> <ciara.loftus@intel.com>, "anatoly burakov" <anatoly.burakov@intel.com>
> Sent: Tuesday, 11 September, 2018 6:56:00 PM
> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports
> 
> On 10/09/2018 16:47, Bala Sankaran wrote:
> > Hello Tiago,
> > 
> > Here's an incremental diff of the patch 6 which I tested. It looks to pass
> > the tests. I would be submitting the
> > new version (v5) applying these changes:
> >
> > diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> > index baa74da..a9247f8 100644
> > --- a/tests/system-dpdk.at
> > +++ b/tests/system-dpdk.at
> > @@ -88,6 +88,10 @@ OVS_DPDK_PRE_CHECK()
> >  AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> >  OVS_DPDK_START()
> >  
> > +dnl Find number of sockets
> > +AT_CHECK([lscpu], [], [stdout])
> > +AT_CHECK([cat stdout | grep "Socket(s)" | awk '{c=1; while (c++<$(3))
> > {printf "512,"}; print "512"}' > SOCKET_MEM])
> 
> Hi Bala,
Hello Tiago,

> 
> Thanks for the incremental.
> 
> Any specific reason to use the result of "Socket(s)" here? I'd use the
> same "NUMA node(s)" instead, as the OVS_DPDK_START is doing (in both
> cases the result is going to be passed to the "--socket-mem" option).
> 
> Also, I think it would be preferable if you'd use a different file to
> store the information, instead of overriding the same SOCKET_MEM file
> that OVS_DPDK_START sets.

I have submitted a v5 of the patches, including these changes you specified.

> 
> > +
> >  dnl Add userspace bridge and attach it to OVS
> >  AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> >  AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface
> >  dpdkvhostuser0 \
> > @@ -111,7 +115,7 @@ ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> >  
> >  dnl Execute testpmd in background
> >  on_exit "pkill -f -x -9 'tail -f /dev/null'"
> > -tail -f /dev/null | testpmd --socket-mem=512 \
> > +tail -f /dev/null | testpmd --socket-mem="$(cat SOCKET_MEM)" --no-pci\
> >             --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> >             --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> >             --single-file-segments -- -a
> >             >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> > @@ -183,7 +187,7 @@ ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> >  
> >  dnl Execute testpmd in background
> >  on_exit "pkill -f -x -9 'tail -f /dev/null'"
> > -tail -f /dev/null | testpmd --socket-mem=512 \
> > +tail -f /dev/null | testpmd --socket-mem="$(cat SOCKET_MEM)" --no-pci\
> >             --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1"
> >             \
> >             --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> >             --single-file-segments -- -a
> >             >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> > 
> > 
> > Before I do so, I had a question for you.
> > 
> > Do you suggest that we have the socket-mem option at all? Because it could
> > vary for each system, and also, the latest
> > versions of DPDK would dynamically allocate the socket-memory, but earlier
> > versions might have had the socket-memory
> > option hard-coded - which could in turn be a different value in each system
> > that the tests run on.
> 
> I had the same question myself, because it would make a lot more sense
> to just drop it (the "--socket-mem" option, that is). However, it
> doesn't seem to work either. The only way I could get it to work on my
> system was to provide an argument to all available nodes (I'm not sure
> this is the intended behavior though, from DPDK 18.05+, it would require
> further investigation). Does it work on your end, if running on a system
> with multiple NUMA nodes?

The tests pass on a single NUMA node system, however, either due to a problem with my 
configurations on the multi-NUMA node system or with the testpmd application, the 
tests do not pass on that system. Inspecting the testpmd-dpdkvhostuser0.log file,
I find a, "Creation of mbuf pool for socket 0 failed: Cannot allocate memory"
error.

Can you test it on a multi-NUMA node system that you have and let me know if the
tests pass at your end?

Thanks,
Bala.
 
> 
> Hope this helps,
> Tiago.
>