[ovs-dev,v4,07/12] test/sytem-dpdk: Add unit test for mfex autovalidator

Tests:
  6: OVS-DPDK - MFEX Autovalidator
  7: OVS-DPDK - MFEX Autovalidator Fuzzy

Added a new directory to store the PCAP file used
in the tests and a script to generate the fuzzy traffic
type pcap to be used in fuzzy unit test.

Signed-off-by: Kumar Amber <kumar.amber@intel.com>
---
 tests/automake.mk    |   5 +++++
 tests/pcap/fuzzy.py  |  32 ++++++++++++++++++++++++++++++
 tests/pcap/mfex_test | Bin 0 -> 416 bytes
 tests/system-dpdk.at |  46 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 83 insertions(+)
 create mode 100755 tests/pcap/fuzzy.py
 create mode 100644 tests/pcap/mfex_test

Hi,


On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> Tests:
>   6: OVS-DPDK - MFEX Autovalidator
>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
> 
> Added a new directory to store the PCAP file used
> in the tests and a script to generate the fuzzy traffic
> type pcap to be used in fuzzy unit test.


I haven't tried this yet but am I right that these tests are
going to pass a pcap to send traffic in a busy loop for 5
seconds in the first case and 20 seconds in the second case?

I see that when autovalidator is set OVS will crash if one
implementation returns a different value, so I wonder why
we need to run for that long.

It is storing a python tool in the pcap directory. I think the
fuzzy tool could be called 'mfex_fuzzy.py' and stay in tests/
with other similar testing tools.

Also, I don't think the test environment sets OVS_DIR. The
'tests/' is actually $srcdir, but I could be wrong here.

BTW, scapy is not mandatory to build or test OVS, so if that
tool is not available, the test should be skipped and not fail.

Thanks,
fbl


> 
> Signed-off-by: Kumar Amber <kumar.amber@intel.com>
> ---
>  tests/automake.mk    |   5 +++++
>  tests/pcap/fuzzy.py  |  32 ++++++++++++++++++++++++++++++
>  tests/pcap/mfex_test | Bin 0 -> 416 bytes
>  tests/system-dpdk.at |  46 +++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 83 insertions(+)
>  create mode 100755 tests/pcap/fuzzy.py
>  create mode 100644 tests/pcap/mfex_test
> 
> diff --git a/tests/automake.mk b/tests/automake.mk
> index 1a528aa39..532875971 100644
> --- a/tests/automake.mk
> +++ b/tests/automake.mk
> @@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at: tests/automake.mk
>  	    echo "TEST_FUZZ_REGRESSION([$$basename])"; \
>  	done > $@.tmp && mv $@.tmp $@
>  
> +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
> +MFEX_AUTOVALIDATOR_TESTS = \
> +	tests/pcap/mfex_test \
> +	tests/pcap/fuzzy.py
> +
>  OVSDB_CLUSTER_TESTSUITE_AT = \
>  	tests/ovsdb-cluster-testsuite.at \
>  	tests/ovsdb-execution.at \
> diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py
> new file mode 100755
> index 000000000..a8051ba2b
> --- /dev/null
> +++ b/tests/pcap/fuzzy.py
> @@ -0,0 +1,32 @@
> +#!/usr/bin/python3
> +try:
> +   from scapy.all import *
> +except ModuleNotFoundError as err:
> +   print(err + ": Scapy")
> +import sys
> +import os
> +
> +path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy"
> +pktdump = PcapWriter(path, append=False, sync=True)
> +
> +for i in range(0, 2000):
> +
> +   # Generate random protocol bases, use a fuzz() over the combined packet for full fuzzing.
> +   eth = Ether(src=RandMAC(), dst=RandMAC())
> +   vlan = Dot1Q()
> +   ipv4 = IP(src=RandIP(), dst=RandIP())
> +   ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
> +   udp = UDP()
> +   tcp = TCP()
> +
> +   # IPv4 packets with fuzzing
> +   pktdump.write(fuzz(eth/ipv4/udp))
> +   pktdump.write(fuzz(eth/ipv4/tcp))
> +   pktdump.write(fuzz(eth/vlan/ipv4/udp))
> +   pktdump.write(fuzz(eth/vlan/ipv4/tcp))
> +
> +    # IPv6 packets with fuzzing
> +   pktdump.write(fuzz(eth/ipv6/udp))
> +   pktdump.write(fuzz(eth/ipv6/tcp))
> +   pktdump.write(fuzz(eth/vlan/ipv6/udp))
> +   pktdump.write(fuzz(eth/vlan/ipv6/tcp))
> \ No newline at end of file
> diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test
> new file mode 100644
> index 0000000000000000000000000000000000000000..1aac67b8d643ecb016c758cba4cc32212a80f52a
> GIT binary patch
> literal 416
> zcmca|c+)~A1{MYw`2U}Qff2}Q<eHVR>K`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}o|
> z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$<F=a4i;9x8Q*<ZrSM6Ufz
> xK>KST_NTIwYriok6N4Vm)gX-Q@<yO<!C`>c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
> 
> literal 0
> HcmV?d00001
> 
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index 802895488..46eaea35a 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -232,3 +232,49 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is probably
>  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
>  AT_CLEANUP
>  dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl Add standard DPDK PHY port
> +AT_SETUP([OVS-DPDK - MFEX Autovalidator])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk options:dpdk-devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/mfex_test,infinite_rx=1], [], [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +
> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], [dnl
> +Miniflow implementation set to autovalidator.
> +])
> +sleep 5
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl Add standard DPDK PHY port
> +AT_SETUP([OVS-DPDK - MFEX Autovalidator Fuzzy])
> +AT_KEYWORDS([dpdk])
> +AT_CHECK([$PYTHON3 $OVS_DIR/tests/pcap/fuzzy.py], [], [stdout])
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk options:dpdk-devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/fuzzy,infinite_rx=1], [], [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +
> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], [dnl
> +Miniflow implementation set to autovalidator.
> +])
> +sleep 20
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> -- 
> 2.25.1
> 
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Hi Flavio,

Thanks again for excellent reviews and my replies are inline.

> -----Original Message-----
> From: Flavio Leitner <fbl@sysclose.org>
> Sent: Monday, June 28, 2021 8:28 AM
> To: Amber, Kumar <kumar.amber@intel.com>
> Cc: dev@openvswitch.org; i.maximets@ovn.org
> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
> autovalidator
> 
> Hi,
> 
> 
> On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> > Tests:
> >   6: OVS-DPDK - MFEX Autovalidator
> >   7: OVS-DPDK - MFEX Autovalidator Fuzzy
> >
> > Added a new directory to store the PCAP file used in the tests and a
> > script to generate the fuzzy traffic type pcap to be used in fuzzy
> > unit test.
> 
> 
> I haven't tried this yet but am I right that these tests are going to pass a pcap
> to send traffic in a busy loop for 5 seconds in the first case and 20 seconds in
> the second case?

Yes ,  The second test is supposed to send traffic via a PCAP which holds over 10k
Different traffic type and thus we ensure by a time delay that each one of them is send
Before exiting the test.

> 
> I see that when autovalidator is set OVS will crash if one implementation
> returns a different value, so I wonder why we need to run for that long.
> 
> It is storing a python tool in the pcap directory. I think the fuzzy tool could
> be called 'mfex_fuzzy.py' and stay in tests/ with other similar testing tools.
> 

Make sense renamed and moved to test in v5.

> Also, I don't think the test environment sets OVS_DIR. The 'tests/' is actually
> $srcdir, but I could be wrong here.
> 

No correct changed to $srcdir and it makes test robust too thanks 😊

> BTW, scapy is not mandatory to build or test OVS, so if that tool is not
> available, the test should be skipped and not fail.
> 

Done and fixed in v5.

> Thanks,
> fbl
> 
> 
> >
> > Signed-off-by: Kumar Amber <kumar.amber@intel.com>
> > ---
> >  tests/automake.mk    |   5 +++++
> >  tests/pcap/fuzzy.py  |  32 ++++++++++++++++++++++++++++++
> > tests/pcap/mfex_test | Bin 0 -> 416 bytes  tests/system-dpdk.at |  46
> > +++++++++++++++++++++++++++++++++++++++++++
> >  4 files changed, 83 insertions(+)
> >  create mode 100755 tests/pcap/fuzzy.py  create mode 100644
> > tests/pcap/mfex_test
> >
> > diff --git a/tests/automake.mk b/tests/automake.mk index
> > 1a528aa39..532875971 100644
> > --- a/tests/automake.mk
> > +++ b/tests/automake.mk
> > @@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at:
> tests/automake.mk
> >  	    echo "TEST_FUZZ_REGRESSION([$$basename])"; \
> >  	done > $@.tmp && mv $@.tmp $@
> >
> > +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
> MFEX_AUTOVALIDATOR_TESTS =
> > +\
> > +	tests/pcap/mfex_test \
> > +	tests/pcap/fuzzy.py
> > +
> >  OVSDB_CLUSTER_TESTSUITE_AT = \
> >  	tests/ovsdb-cluster-testsuite.at \
> >  	tests/ovsdb-execution.at \
> > diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py new file mode
> > 100755 index 000000000..a8051ba2b
> > --- /dev/null
> > +++ b/tests/pcap/fuzzy.py
> > @@ -0,0 +1,32 @@
> > +#!/usr/bin/python3
> > +try:
> > +   from scapy.all import *
> > +except ModuleNotFoundError as err:
> > +   print(err + ": Scapy")
> > +import sys
> > +import os
> > +
> > +path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy"
> > +pktdump = PcapWriter(path, append=False, sync=True)
> > +
> > +for i in range(0, 2000):
> > +
> > +   # Generate random protocol bases, use a fuzz() over the combined
> packet for full fuzzing.
> > +   eth = Ether(src=RandMAC(), dst=RandMAC())
> > +   vlan = Dot1Q()
> > +   ipv4 = IP(src=RandIP(), dst=RandIP())
> > +   ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
> > +   udp = UDP()
> > +   tcp = TCP()
> > +
> > +   # IPv4 packets with fuzzing
> > +   pktdump.write(fuzz(eth/ipv4/udp))
> > +   pktdump.write(fuzz(eth/ipv4/tcp))
> > +   pktdump.write(fuzz(eth/vlan/ipv4/udp))
> > +   pktdump.write(fuzz(eth/vlan/ipv4/tcp))
> > +
> > +    # IPv6 packets with fuzzing
> > +   pktdump.write(fuzz(eth/ipv6/udp))
> > +   pktdump.write(fuzz(eth/ipv6/tcp))
> > +   pktdump.write(fuzz(eth/vlan/ipv6/udp))
> > +   pktdump.write(fuzz(eth/vlan/ipv6/tcp))
> > \ No newline at end of file
> > diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test new file mode
> > 100644 index
> >
> 0000000000000000000000000000000000000000..1aac67b8d643ecb016c758cb
> a4cc
> > 32212a80f52a
> > GIT binary patch
> > literal 416
> >
> zcmca|c+)~A1{MYw`2U}Qff2}Q<eHVR>K`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}
> o|
> >
> z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$<F=a4i;9x8Q*<ZrSM6U
> fz
> > xK>KST_NTIwYriok6N4Vm)gX-
> Q@<yO<!C`>c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
> >
> > literal 0
> > HcmV?d00001
> >
> > diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at index
> > 802895488..46eaea35a 100644
> > --- a/tests/system-dpdk.at
> > +++ b/tests/system-dpdk.at
> > @@ -232,3 +232,49 @@ OVS_VSWITCHD_STOP(["\@does not exist. The
> Open
> > vSwitch kernel module is probably
> >  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
> > AT_CLEANUP  dnl
> > ----------------------------------------------------------------------
> > ----
> > +
> > +dnl
> > +---------------------------------------------------------------------
> > +-----
> > +dnl Add standard DPDK PHY port
> > +AT_SETUP([OVS-DPDK - MFEX Autovalidator])
> > +AT_KEYWORDS([dpdk])
> > +
> > +OVS_DPDK_START()
> > +
> > +dnl Add userspace bridge and attach it to OVS AT_CHECK([ovs-vsctl
> > +add-br br0 -- set bridge br0 datapath_type=netdev])
> > +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
> > +options:dpdk-
> devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/mfex_test,
> > +infinite_rx=1], [], [stdout], [stderr]) AT_CHECK([ovs-vsctl show],
> > +[], [stdout])
> > +
> > +
> > +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator],
> > +[0], [dnl Miniflow implementation set to autovalidator.
> > +])
> > +sleep 5
> > +
> > +dnl Clean up
> > +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
> > +AT_CLEANUP dnl
> > +---------------------------------------------------------------------
> > +-----
> > +
> > +dnl
> > +---------------------------------------------------------------------
> > +-----
> > +dnl Add standard DPDK PHY port
> > +AT_SETUP([OVS-DPDK - MFEX Autovalidator Fuzzy])
> > +AT_KEYWORDS([dpdk])
> > +AT_CHECK([$PYTHON3 $OVS_DIR/tests/pcap/fuzzy.py], [], [stdout])
> > +OVS_DPDK_START()
> > +
> > +dnl Add userspace bridge and attach it to OVS AT_CHECK([ovs-vsctl
> > +add-br br0 -- set bridge br0 datapath_type=netdev])
> > +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
> > +options:dpdk-devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/fuzzy,infi
> > +nite_rx=1], [], [stdout], [stderr]) AT_CHECK([ovs-vsctl show], [],
> > +[stdout])
> > +
> > +
> > +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator],
> > +[0], [dnl Miniflow implementation set to autovalidator.
> > +])
> > +sleep 20
> > +
> > +dnl Clean up
> > +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
> > +AT_CLEANUP dnl
> > +---------------------------------------------------------------------
> > +-----
> > --
> > 2.25.1
> >
> > _______________________________________________
> > dev mailing list
> > dev@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 
> --
> fbl

On 28 Jun 2021, at 4:57, Flavio Leitner wrote:

> Hi,
>
>
> On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
>> Tests:
>>   6: OVS-DPDK - MFEX Autovalidator
>>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
>>
>> Added a new directory to store the PCAP file used
>> in the tests and a script to generate the fuzzy traffic
>> type pcap to be used in fuzzy unit test.
>
>
> I haven't tried this yet but am I right that these tests are
> going to pass a pcap to send traffic in a busy loop for 5
> seconds in the first case and 20 seconds in the second case?
>
> I see that when autovalidator is set OVS will crash if one
> implementation returns a different value, so I wonder why
> we need to run for that long.

I think we should remove the assert (already suggested by Harry),  so it will not crass by accident if someone selects autovalidator in the field (and runs into an issue).
Failure will then be detected by the ERROR log entries on shutdown.

I’m wondering if there is another way than a simple delay, as these tend to cause issues later on. Can we check packets processed or something?

> It is storing a python tool in the pcap directory. I think the
> fuzzy tool could be called 'mfex_fuzzy.py' and stay in tests/
> with other similar testing tools.
>
> Also, I don't think the test environment sets OVS_DIR. The
> 'tests/' is actually $srcdir, but I could be wrong here.
>
> BTW, scapy is not mandatory to build or test OVS, so if that
> tool is not available, the test should be skipped and not fail.
>
> Thanks,
> fbl
>
>
>>
>> Signed-off-by: Kumar Amber <kumar.amber@intel.com>
>> ---
>>  tests/automake.mk    |   5 +++++
>>  tests/pcap/fuzzy.py  |  32 ++++++++++++++++++++++++++++++
>>  tests/pcap/mfex_test | Bin 0 -> 416 bytes
>>  tests/system-dpdk.at |  46 +++++++++++++++++++++++++++++++++++++++++++
>>  4 files changed, 83 insertions(+)
>>  create mode 100755 tests/pcap/fuzzy.py
>>  create mode 100644 tests/pcap/mfex_test
>>
>> diff --git a/tests/automake.mk b/tests/automake.mk
>> index 1a528aa39..532875971 100644
>> --- a/tests/automake.mk
>> +++ b/tests/automake.mk
>> @@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at: tests/automake.mk
>>  	    echo "TEST_FUZZ_REGRESSION([$$basename])"; \
>>  	done > $@.tmp && mv $@.tmp $@
>>
>> +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
>> +MFEX_AUTOVALIDATOR_TESTS = \
>> +	tests/pcap/mfex_test \
>> +	tests/pcap/fuzzy.py
>> +
>>  OVSDB_CLUSTER_TESTSUITE_AT = \
>>  	tests/ovsdb-cluster-testsuite.at \
>>  	tests/ovsdb-execution.at \
>> diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py
>> new file mode 100755
>> index 000000000..a8051ba2b
>> --- /dev/null
>> +++ b/tests/pcap/fuzzy.py
>> @@ -0,0 +1,32 @@
>> +#!/usr/bin/python3
>> +try:
>> +   from scapy.all import *
>> +except ModuleNotFoundError as err:
>> +   print(err + ": Scapy")
>> +import sys
>> +import os
>> +
>> +path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy"
>> +pktdump = PcapWriter(path, append=False, sync=True)
>> +
>> +for i in range(0, 2000):
>> +
>> +   # Generate random protocol bases, use a fuzz() over the combined packet for full fuzzing.
>> +   eth = Ether(src=RandMAC(), dst=RandMAC())
>> +   vlan = Dot1Q()
>> +   ipv4 = IP(src=RandIP(), dst=RandIP())
>> +   ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
>> +   udp = UDP()
>> +   tcp = TCP()
>> +
>> +   # IPv4 packets with fuzzing
>> +   pktdump.write(fuzz(eth/ipv4/udp))
>> +   pktdump.write(fuzz(eth/ipv4/tcp))
>> +   pktdump.write(fuzz(eth/vlan/ipv4/udp))
>> +   pktdump.write(fuzz(eth/vlan/ipv4/tcp))
>> +
>> +    # IPv6 packets with fuzzing
>> +   pktdump.write(fuzz(eth/ipv6/udp))
>> +   pktdump.write(fuzz(eth/ipv6/tcp))
>> +   pktdump.write(fuzz(eth/vlan/ipv6/udp))
>> +   pktdump.write(fuzz(eth/vlan/ipv6/tcp))
>> \ No newline at end of file
>> diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..1aac67b8d643ecb016c758cba4cc32212a80f52a
>> GIT binary patch
>> literal 416
>> zcmca|c+)~A1{MYw`2U}Qff2}Q<eHVR>K`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}o|
>> z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$<F=a4i;9x8Q*<ZrSM6Ufz
>> xK>KST_NTIwYriok6N4Vm)gX-Q@<yO<!C`>c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
>>
>> literal 0
>> HcmV?d00001
>>
>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
>> index 802895488..46eaea35a 100644
>> --- a/tests/system-dpdk.at
>> +++ b/tests/system-dpdk.at
>> @@ -232,3 +232,49 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is probably
>>  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
>>  AT_CLEANUP
>>  dnl --------------------------------------------------------------------------
>> +
>> +dnl --------------------------------------------------------------------------
>> +dnl Add standard DPDK PHY port
>> +AT_SETUP([OVS-DPDK - MFEX Autovalidator])
>> +AT_KEYWORDS([dpdk])
>> +
>> +OVS_DPDK_START()
>> +
>> +dnl Add userspace bridge and attach it to OVS
>> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
>> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk options:dpdk-devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/mfex_test,infinite_rx=1], [], [stdout], [stderr])
>> +AT_CHECK([ovs-vsctl show], [], [stdout])
>> +
>> +
>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], [dnl
>> +Miniflow implementation set to autovalidator.
>> +])
>> +sleep 5
>> +
>> +dnl Clean up
>> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
>> +AT_CLEANUP
>> +dnl --------------------------------------------------------------------------
>> +
>> +dnl --------------------------------------------------------------------------
>> +dnl Add standard DPDK PHY port
>> +AT_SETUP([OVS-DPDK - MFEX Autovalidator Fuzzy])
>> +AT_KEYWORDS([dpdk])
>> +AT_CHECK([$PYTHON3 $OVS_DIR/tests/pcap/fuzzy.py], [], [stdout])
>> +OVS_DPDK_START()
>> +
>> +dnl Add userspace bridge and attach it to OVS
>> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
>> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk options:dpdk-devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/fuzzy,infinite_rx=1], [], [stdout], [stderr])
>> +AT_CHECK([ovs-vsctl show], [], [stdout])
>> +
>> +
>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], [dnl
>> +Miniflow implementation set to autovalidator.
>> +])
>> +sleep 20
>> +
>> +dnl Clean up
>> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
>> +AT_CLEANUP
>> +dnl --------------------------------------------------------------------------
>> -- 
>> 2.25.1
>>
>> _______________________________________________
>> dev mailing list
>> dev@openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
> -- 
> fbl
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
> 
> 
> On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> 
> > Hi,
> >
> >
> > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> >> Tests:
> >>   6: OVS-DPDK - MFEX Autovalidator
> >>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
> >>
> >> Added a new directory to store the PCAP file used
> >> in the tests and a script to generate the fuzzy traffic
> >> type pcap to be used in fuzzy unit test.
> >
> >
> > I haven't tried this yet but am I right that these tests are
> > going to pass a pcap to send traffic in a busy loop for 5
> > seconds in the first case and 20 seconds in the second case?
> >
> > I see that when autovalidator is set OVS will crash if one
> > implementation returns a different value, so I wonder why
> > we need to run for that long.
> 
> I think we should remove the assert (already suggested by Harry),
> so it will not crass by accident if someone selects autovalidator
> in the field (and runs into an issue).
> Failure will then be detected by the ERROR log entries on shutdown.

That's true for the testsuite, but not in production as there is
nothing to disable that.

Perhaps if autovalidator detects an issue, it should log an ERROR
level log to report to testsuite, disable the failing mode and make
sure OVS is either in default or in another functional mode.

> I’m wondering if there is another way than a simple delay, as these tend to cause issues later on. Can we check packets processed or something?

Yeah, maybe we can pass all packets like 5x at least.

fbl


> 
> > It is storing a python tool in the pcap directory. I think the
> > fuzzy tool could be called 'mfex_fuzzy.py' and stay in tests/
> > with other similar testing tools.
> >
> > Also, I don't think the test environment sets OVS_DIR. The
> > 'tests/' is actually $srcdir, but I could be wrong here.
> >
> > BTW, scapy is not mandatory to build or test OVS, so if that
> > tool is not available, the test should be skipped and not fail.
> >
> > Thanks,
> > fbl
<SNIP>

> Tests:
>   6: OVS-DPDK - MFEX Autovalidator
>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
> 
> Added a new directory to store the PCAP file used
> in the tests and a script to generate the fuzzy traffic
> type pcap to be used in fuzzy unit test.

Hi Amber,

I don't have much to add here that Flavio and Eelco have already flagged. Looking forward to reviewing/testing the v5.

Regards
Ian
> 
> Signed-off-by: Kumar Amber <kumar.amber@intel.com>
> ---
>  tests/automake.mk    |   5 +++++
>  tests/pcap/fuzzy.py  |  32 ++++++++++++++++++++++++++++++
>  tests/pcap/mfex_test | Bin 0 -> 416 bytes
>  tests/system-dpdk.at |  46 +++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 83 insertions(+)
>  create mode 100755 tests/pcap/fuzzy.py
>  create mode 100644 tests/pcap/mfex_test
> 
> diff --git a/tests/automake.mk b/tests/automake.mk
> index 1a528aa39..532875971 100644
> --- a/tests/automake.mk
> +++ b/tests/automake.mk
> @@ -142,6 +142,11 @@ $(srcdir)/tests/fuzz-regression-list.at:
> tests/automake.mk
>  	    echo "TEST_FUZZ_REGRESSION([$$basename])"; \
>  	done > $@.tmp && mv $@.tmp $@
> 
> +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
> +MFEX_AUTOVALIDATOR_TESTS = \
> +	tests/pcap/mfex_test \
> +	tests/pcap/fuzzy.py
> +
>  OVSDB_CLUSTER_TESTSUITE_AT = \
>  	tests/ovsdb-cluster-testsuite.at \
>  	tests/ovsdb-execution.at \
> diff --git a/tests/pcap/fuzzy.py b/tests/pcap/fuzzy.py
> new file mode 100755
> index 000000000..a8051ba2b
> --- /dev/null
> +++ b/tests/pcap/fuzzy.py
> @@ -0,0 +1,32 @@
> +#!/usr/bin/python3
> +try:
> +   from scapy.all import *
> +except ModuleNotFoundError as err:
> +   print(err + ": Scapy")
> +import sys
> +import os
> +
> +path = os.environ['OVS_DIR'] + "/tests/pcap/fuzzy"
> +pktdump = PcapWriter(path, append=False, sync=True)
> +
> +for i in range(0, 2000):
> +
> +   # Generate random protocol bases, use a fuzz() over the combined packet for
> full fuzzing.
> +   eth = Ether(src=RandMAC(), dst=RandMAC())
> +   vlan = Dot1Q()
> +   ipv4 = IP(src=RandIP(), dst=RandIP())
> +   ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
> +   udp = UDP()
> +   tcp = TCP()
> +
> +   # IPv4 packets with fuzzing
> +   pktdump.write(fuzz(eth/ipv4/udp))
> +   pktdump.write(fuzz(eth/ipv4/tcp))
> +   pktdump.write(fuzz(eth/vlan/ipv4/udp))
> +   pktdump.write(fuzz(eth/vlan/ipv4/tcp))
> +
> +    # IPv6 packets with fuzzing
> +   pktdump.write(fuzz(eth/ipv6/udp))
> +   pktdump.write(fuzz(eth/ipv6/tcp))
> +   pktdump.write(fuzz(eth/vlan/ipv6/udp))
> +   pktdump.write(fuzz(eth/vlan/ipv6/tcp))
> \ No newline at end of file
> diff --git a/tests/pcap/mfex_test b/tests/pcap/mfex_test
> new file mode 100644
> index
> 0000000000000000000000000000000000000000..1aac67b8d643ecb016c758cb
> a4cc32212a80f52a
> GIT binary patch
> literal 416
> zcmca|c+)~A1{MYw`2U}Qff2}Q<eHVR>K`M68ITRa|G@yFii5$Gfk6YL%z>@uY&}o
> |
> z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$<F=a4i;9x8Q*<ZrSM6Uf
> z
> xK>KST_NTIwYriok6N4Vm)gX-
> Q@<yO<!C`>c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
> 
> literal 0
> HcmV?d00001
> 
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index 802895488..46eaea35a 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -232,3 +232,49 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open
> vSwitch kernel module is probably
>  \@EAL: No free hugepages reported in hugepages-1048576kB@d"])
>  AT_CLEANUP
>  dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl Add standard DPDK PHY port
> +AT_SETUP([OVS-DPDK - MFEX Autovalidator])
> +AT_KEYWORDS([dpdk])
> +
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
> options:dpdk-
> devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/mfex_test,infinite_rx=1], [],
> [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +
> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], [dnl
> +Miniflow implementation set to autovalidator.
> +])
> +sleep 5
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl Add standard DPDK PHY port
> +AT_SETUP([OVS-DPDK - MFEX Autovalidator Fuzzy])
> +AT_KEYWORDS([dpdk])
> +AT_CHECK([$PYTHON3 $OVS_DIR/tests/pcap/fuzzy.py], [], [stdout])
> +OVS_DPDK_START()
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
> options:dpdk-
> devargs=net_pcap1,rx_pcap=$OVS_DIR/tests/pcap/fuzzy,infinite_rx=1], [],
> [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +
> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0], [dnl
> +Miniflow implementation set to autovalidator.
> +])
> +sleep 20
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> --
> 2.25.1
> 
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Hi Eelco, Flavio,

Pls find my replies Inline

> -----Original Message-----
> From: Flavio Leitner <fbl@sysclose.org>
> Sent: Tuesday, June 29, 2021 7:51 PM
> To: Eelco Chaudron <echaudro@redhat.com>
> Cc: Amber, Kumar <kumar.amber@intel.com>; Van Haaren, Harry
> <harry.van.haaren@intel.com>; dev@openvswitch.org; i.maximets@ovn.org
> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
> autovalidator
> 
> On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
> >
> >
> > On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> >
> > > Hi,
> > >
> > >
> > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> > >> Tests:
> > >>   6: OVS-DPDK - MFEX Autovalidator
> > >>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
> > >>
> > >> Added a new directory to store the PCAP file used in the tests and
> > >> a script to generate the fuzzy traffic type pcap to be used in
> > >> fuzzy unit test.
> > >
> > >
> > > I haven't tried this yet but am I right that these tests are going
> > > to pass a pcap to send traffic in a busy loop for 5 seconds in the
> > > first case and 20 seconds in the second case?
> > >
> > > I see that when autovalidator is set OVS will crash if one
> > > implementation returns a different value, so I wonder why we need to
> > > run for that long.
> >
> > I think we should remove the assert (already suggested by Harry), so
> > it will not crass by accident if someone selects autovalidator in the
> > field (and runs into an issue).
> > Failure will then be detected by the ERROR log entries on shutdown.
> 
> That's true for the testsuite, but not in production as there is nothing to
> disable that.
> 
> Perhaps if autovalidator detects an issue, it should log an ERROR level log to
> report to testsuite, disable the failing mode and make sure OVS is either in
> default or in another functional mode.

So I have put the following : 
	Removed the assert 
	Allow the Auto-validator to run for all implementation and for a full batch
	Document error via Vlog_Error
	Set the auto-validator to default {Scalar} when returning out in case of failure. 
> 
> > I’m wondering if there is another way than a simple delay, as these tend to
> cause issues later on. Can we check packets processed or something?
> 
> Yeah, maybe we can pass all packets like 5x at least.

Sure I will try to find something to do it more nicely.
But just a thought keeping it 20sec allows for a full-stabilization and also thorough testing of stability as well.
So keeping it may not be just a bad idea.
> 
> fbl
> 
> 
> >
> > > It is storing a python tool in the pcap directory. I think the fuzzy
> > > tool could be called 'mfex_fuzzy.py' and stay in tests/ with other
> > > similar testing tools.
> > >
> > > Also, I don't think the test environment sets OVS_DIR. The 'tests/'
> > > is actually $srcdir, but I could be wrong here.
> > >
> > > BTW, scapy is not mandatory to build or test OVS, so if that tool is
> > > not available, the test should be skipped and not fail.
> > >
> > > Thanks,
> > > fbl
> <SNIP>

On Tue, Jun 29, 2021 at 05:11:00PM +0000, Amber, Kumar wrote:
> Hi Eelco, Flavio,
> 
> Pls find my replies Inline
> 
> > -----Original Message-----
> > From: Flavio Leitner <fbl@sysclose.org>
> > Sent: Tuesday, June 29, 2021 7:51 PM
> > To: Eelco Chaudron <echaudro@redhat.com>
> > Cc: Amber, Kumar <kumar.amber@intel.com>; Van Haaren, Harry
> > <harry.van.haaren@intel.com>; dev@openvswitch.org; i.maximets@ovn.org
> > Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
> > autovalidator
> > 
> > On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
> > >
> > >
> > > On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> > > >> Tests:
> > > >>   6: OVS-DPDK - MFEX Autovalidator
> > > >>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
> > > >>
> > > >> Added a new directory to store the PCAP file used in the tests and
> > > >> a script to generate the fuzzy traffic type pcap to be used in
> > > >> fuzzy unit test.
> > > >
> > > >
> > > > I haven't tried this yet but am I right that these tests are going
> > > > to pass a pcap to send traffic in a busy loop for 5 seconds in the
> > > > first case and 20 seconds in the second case?
> > > >
> > > > I see that when autovalidator is set OVS will crash if one
> > > > implementation returns a different value, so I wonder why we need to
> > > > run for that long.
> > >
> > > I think we should remove the assert (already suggested by Harry), so
> > > it will not crass by accident if someone selects autovalidator in the
> > > field (and runs into an issue).
> > > Failure will then be detected by the ERROR log entries on shutdown.
> > 
> > That's true for the testsuite, but not in production as there is nothing to
> > disable that.
> > 
> > Perhaps if autovalidator detects an issue, it should log an ERROR level log to
> > report to testsuite, disable the failing mode and make sure OVS is either in
> > default or in another functional mode.
> 
> So I have put the following : 
> 	Removed the assert 
> 	Allow the Auto-validator to run for all implementation and for a full batch
> 	Document error via Vlog_Error
> 	Set the auto-validator to default {Scalar} when returning out in case of failure. 

Sounds like a plan to me.
Is that okay with you Eelco?


> > > I’m wondering if there is another way than a simple delay, as these tend to
> > cause issues later on. Can we check packets processed or something?
> > 
> > Yeah, maybe we can pass all packets like 5x at least.
> 
> Sure I will try to find something to do it more nicely.
> But just a thought keeping it 20sec allows for a full-stabilization and also thorough testing of stability as well.
> So keeping it may not be just a bad idea.

The issue is that if every test decides to delay seconds, the testsuite
becomes impractical. We have removed 'sleep' over time. Instead, we
have functions to wait for a certain cmdline output, or some event.
Yes, there are still some left to be fixed.

Back to the point, maybe there is a signal of some sort we can get
that indicates the stability you're looking for.

Hi Flavio,

Comments Inline.

> -----Original Message-----
> From: Flavio Leitner <fbl@sysclose.org>
> Sent: Tuesday, June 29, 2021 11:49 PM
> To: Amber, Kumar <kumar.amber@intel.com>
> Cc: Eelco Chaudron <echaudro@redhat.com>; dev@openvswitch.org;
> i.maximets@ovn.org
> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
> autovalidator
> 
> On Tue, Jun 29, 2021 at 05:11:00PM +0000, Amber, Kumar wrote:
> > Hi Eelco, Flavio,
> >
> > Pls find my replies Inline
> >
> > > -----Original Message-----
> > > From: Flavio Leitner <fbl@sysclose.org>
> > > Sent: Tuesday, June 29, 2021 7:51 PM
> > > To: Eelco Chaudron <echaudro@redhat.com>
> > > Cc: Amber, Kumar <kumar.amber@intel.com>; Van Haaren, Harry
> > > <harry.van.haaren@intel.com>; dev@openvswitch.org;
> > > i.maximets@ovn.org
> > > Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for
> > > mfex autovalidator
> > >
> > > On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
> > > >
> > > >
> > > > On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> > > > >> Tests:
> > > > >>   6: OVS-DPDK - MFEX Autovalidator
> > > > >>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
> > > > >>
> > > > >> Added a new directory to store the PCAP file used in the tests
> > > > >> and a script to generate the fuzzy traffic type pcap to be used
> > > > >> in fuzzy unit test.
> > > > >
> > > > >
> > > > > I haven't tried this yet but am I right that these tests are
> > > > > going to pass a pcap to send traffic in a busy loop for 5
> > > > > seconds in the first case and 20 seconds in the second case?
> > > > >
> > > > > I see that when autovalidator is set OVS will crash if one
> > > > > implementation returns a different value, so I wonder why we
> > > > > need to run for that long.
> > > >
> > > > I think we should remove the assert (already suggested by Harry),
> > > > so it will not crass by accident if someone selects autovalidator
> > > > in the field (and runs into an issue).
> > > > Failure will then be detected by the ERROR log entries on shutdown.
> > >
> > > That's true for the testsuite, but not in production as there is
> > > nothing to disable that.
> > >
> > > Perhaps if autovalidator detects an issue, it should log an ERROR
> > > level log to report to testsuite, disable the failing mode and make
> > > sure OVS is either in default or in another functional mode.
> >
> > So I have put the following :
> > 	Removed the assert
> > 	Allow the Auto-validator to run for all implementation and for a full
> batch
> > 	Document error via Vlog_Error
> > 	Set the auto-validator to default {Scalar} when returning out in case
> of failure.
> 
> Sounds like a plan to me.
> Is that okay with you Eelco?
> 
> 
> > > > I’m wondering if there is another way than a simple delay, as
> > > > these tend to
> > > cause issues later on. Can we check packets processed or something?
> > >
> > > Yeah, maybe we can pass all packets like 5x at least.
> >
> > Sure I will try to find something to do it more nicely.
> > But just a thought keeping it 20sec allows for a full-stabilization and also
> thorough testing of stability as well.
> > So keeping it may not be just a bad idea.
> 
> The issue is that if every test decides to delay seconds, the testsuite
> becomes impractical. We have removed 'sleep' over time. Instead, we have
> functions to wait for a certain cmdline output, or some event.
> Yes, there are still some left to be fixed.
> 
> Back to the point, maybe there is a signal of some sort we can get that
> indicates the stability you're looking for.
> 

I agree to the point and I am looking for a singal but currently due to assert removal we don’t have any marker.
To Minimize the time, I did analysis of the time taken in each test-case :

1) for the simple test-case we don’t need the 5sec wait time as PCAP only contains one traffic or each type.
2) for fuzzy we do need at least 5sec for all 10k packets to be sent at-least 2x and also stability.

Will the reductions suffice for now until I find a way to remove it completely from Fuzzy ?

Regards
Amber
> --
> fbl

On 29 Jun 2021, at 20:49, Amber, Kumar wrote:

> Hi Flavio,
>
> Comments Inline.
>
>> -----Original Message-----
>> From: Flavio Leitner <fbl@sysclose.org>
>> Sent: Tuesday, June 29, 2021 11:49 PM
>> To: Amber, Kumar <kumar.amber@intel.com>
>> Cc: Eelco Chaudron <echaudro@redhat.com>; dev@openvswitch.org;
>> i.maximets@ovn.org
>> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
>> autovalidator
>>
>> On Tue, Jun 29, 2021 at 05:11:00PM +0000, Amber, Kumar wrote:
>>> Hi Eelco, Flavio,
>>>
>>> Pls find my replies Inline
>>>
>>>> -----Original Message-----
>>>> From: Flavio Leitner <fbl@sysclose.org>
>>>> Sent: Tuesday, June 29, 2021 7:51 PM
>>>> To: Eelco Chaudron <echaudro@redhat.com>
>>>> Cc: Amber, Kumar <kumar.amber@intel.com>; Van Haaren, Harry
>>>> <harry.van.haaren@intel.com>; dev@openvswitch.org;
>>>> i.maximets@ovn.org
>>>> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for
>>>> mfex autovalidator
>>>>
>>>> On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
>>>>>
>>>>>
>>>>> On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
>>>>>>> Tests:
>>>>>>>   6: OVS-DPDK - MFEX Autovalidator
>>>>>>>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
>>>>>>>
>>>>>>> Added a new directory to store the PCAP file used in the tests
>>>>>>> and a script to generate the fuzzy traffic type pcap to be used
>>>>>>> in fuzzy unit test.
>>>>>>
>>>>>>
>>>>>> I haven't tried this yet but am I right that these tests are
>>>>>> going to pass a pcap to send traffic in a busy loop for 5
>>>>>> seconds in the first case and 20 seconds in the second case?
>>>>>>
>>>>>> I see that when autovalidator is set OVS will crash if one
>>>>>> implementation returns a different value, so I wonder why we
>>>>>> need to run for that long.
>>>>>
>>>>> I think we should remove the assert (already suggested by Harry),
>>>>> so it will not crass by accident if someone selects autovalidator
>>>>> in the field (and runs into an issue).
>>>>> Failure will then be detected by the ERROR log entries on shutdown.
>>>>
>>>> That's true for the testsuite, but not in production as there is
>>>> nothing to disable that.
>>>>
>>>> Perhaps if autovalidator detects an issue, it should log an ERROR
>>>> level log to report to testsuite, disable the failing mode and make
>>>> sure OVS is either in default or in another functional mode.
>>>
>>> So I have put the following :
>>> 	Removed the assert
>>> 	Allow the Auto-validator to run for all implementation and for a full
>> batch
>>> 	Document error via Vlog_Error
>>> 	Set the auto-validator to default {Scalar} when returning out in case
>> of failure.
>>
>> Sounds like a plan to me.
>> Is that okay with you Eelco?

ACK, this sounds good to me.

>>>>> I’m wondering if there is another way than a simple delay, as
>>>>> these tend to
>>>> cause issues later on. Can we check packets processed or something?
>>>>
>>>> Yeah, maybe we can pass all packets like 5x at least.
>>>
>>> Sure I will try to find something to do it more nicely.
>>> But just a thought keeping it 20sec allows for a full-stabilization and also
>> thorough testing of stability as well.
>>> So keeping it may not be just a bad idea.
>>
>> The issue is that if every test decides to delay seconds, the testsuite
>> becomes impractical. We have removed 'sleep' over time. Instead, we have
>> functions to wait for a certain cmdline output, or some event.
>> Yes, there are still some left to be fixed.
>>
>> Back to the point, maybe there is a signal of some sort we can get that
>> indicates the stability you're looking for.
>>
>
> I agree to the point and I am looking for a singal but currently due to assert removal we don’t have any marker.
> To Minimize the time, I did analysis of the time taken in each test-case :
>
> 1) for the simple test-case we don’t need the 5sec wait time as PCAP only contains one traffic or each type.
> 2) for fuzzy we do need at least 5sec for all 10k packets to be sent at-least 2x and also stability.

Guess you know the number of packets, so you could check in a 1sec delay loop the number of received packets for the flows, and exit if you saw enough. And just to be sure, you terminate after 5~10 seconds?