[ovs-dev,v2] ovs-macros: An option to suspend test execution on error
diff mbox series

Message ID 20190715211501.88213-1-vdasari@gmail.com
State New
Headers show
Series
  • [ovs-dev,v2] ovs-macros: An option to suspend test execution on error
Related show

Commit Message

Vasu Dasari July 15, 2019, 9:15 p.m. UTC
Origins for this patch are captured at
https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html.

Summarizing here, when a test fails, it would be good to pause test execution
and let the developer poke around the system to see current status of system.

As part of this patch, made a small tweaks to ovs-macros.at, so that when test
suite fails, ovs_on_exit() function will be called. And in this function, a check
is made to see if an environment variable to OVS_PAUSE_TEST is set. If it is
set, then test suite is paused and will continue to wait for user input
Ctrl-D. Meanwhile user can poke around the system to see why test case has
failed. Once done with investigation, user can press ctrl-d to cleanup the
test suite.

For example, to re-run test case 139:

export OVS_PAUSE_TEST=1
cd tests/system-userspace-testsuite.dir/139
sudo -E ./run

When error occurs, above command would display something like this:
=====================================================
Set environment variable to use various ovs utilities
export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
Press ENTER to continue:

=====================================================
And from another window, one can execute ovs-xxx commands like:
export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
$ ovs-ofctl dump-ports br0
.
.

To be able to pause while performing `make check`, one can do:
$ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'

Signed-off-by: Vasu Dasari <vdasari@gmail.com>
Reviewed-by: Ben Pfaff <blp@ovn.org>
Reviewed-by: Aaron Conole <aconole@redhat.com>
---
v0 -> v1:
Discussion at: https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360589.html
1. Incorporated review commanets from Ben.
2. Changed topics/testing.rst to document this trick
3. Made this trick work via "make check" as well, by issuing command: "OVS_PAUSE_TEST=1
make check TESTSUITEFLAGS='-v'"

v1 -> v2:
Discussion at: https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360705.html
1. Added a subsection, "Debugging unit tests"
2. Renamed $as_echo to just echo. AS_ECHO is not suported in autoconf 2.69.11. And, we are not gainin
g anything from AS_ECHO in this code, just using echo.

---
 Documentation/topics/testing.rst | 31 +++++++++++++++++++++++++++++++
 tests/ovs-macros.at              | 24 +++++++++++++++++++++++-
 2 files changed, 54 insertions(+), 1 deletion(-)

Comments

0-day Robot July 15, 2019, 9:58 p.m. UTC | #1
Bleep bloop.  Greetings Vasu Dasari, I am a robot and I have tried out your patch.
Thanks for your contribution.

I encountered some error that I wasn't expecting.  See the details below.


checkpatch:
WARNING: Line is 82 characters long (recommended limit is 79)
#71 FILE: Documentation/topics/testing.rst:97:
   export OVS_RUNDIR=<dir>/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139

WARNING: Line is 99 characters long (recommended limit is 79)
#76 FILE: Documentation/topics/testing.rst:102:
   export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139

Lines checked: 133, Warnings: 2, Errors: 0


Please check this out.  If you feel there has been an error, please email aconole@bytheb.org

Thanks,
0-day Robot
Aaron Conole July 16, 2019, 1:11 p.m. UTC | #2
Vasu Dasari <vdasari@gmail.com> writes:

> Origins for this patch are captured at
> https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html.
>
> Summarizing here, when a test fails, it would be good to pause test execution
> and let the developer poke around the system to see current status of system.
>
> As part of this patch, made a small tweaks to ovs-macros.at, so that when test
> suite fails, ovs_on_exit() function will be called. And in this function, a check
> is made to see if an environment variable to OVS_PAUSE_TEST is set. If it is
> set, then test suite is paused and will continue to wait for user input
> Ctrl-D. Meanwhile user can poke around the system to see why test case has
> failed. Once done with investigation, user can press ctrl-d to cleanup the
> test suite.
>
> For example, to re-run test case 139:
>
> export OVS_PAUSE_TEST=1
> cd tests/system-userspace-testsuite.dir/139
> sudo -E ./run
>
> When error occurs, above command would display something like this:
> =====================================================
> Set environment variable to use various ovs utilities
> export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> Press ENTER to continue:
>
> =====================================================
> And from another window, one can execute ovs-xxx commands like:
> export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> $ ovs-ofctl dump-ports br0
> .
> .
>
> To be able to pause while performing `make check`, one can do:
> $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'
>
> Signed-off-by: Vasu Dasari <vdasari@gmail.com>
> Reviewed-by: Ben Pfaff <blp@ovn.org>
> Reviewed-by: Aaron Conole <aconole@redhat.com>
> ---
> v0 -> v1:
> Discussion at: https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360589.html
> 1. Incorporated review commanets from Ben.
> 2. Changed topics/testing.rst to document this trick
> 3. Made this trick work via "make check" as well, by issuing command: "OVS_PAUSE_TEST=1
> make check TESTSUITEFLAGS='-v'"
>
> v1 -> v2:
> Discussion at: https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360705.html
> 1. Added a subsection, "Debugging unit tests"
> 2. Renamed $as_echo to just echo. AS_ECHO is not suported in autoconf 2.69.11. And, we are not gainin
> g anything from AS_ECHO in this code, just using echo.
>
> ---

Acked-by: Aaron Conole <aconole@redhat.com>

Thanks for the cool feature, Vasu!
Ben Pfaff July 16, 2019, 4:55 p.m. UTC | #3
On Mon, Jul 15, 2019 at 05:15:01PM -0400, Vasu Dasari wrote:
> Origins for this patch are captured at
> https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html.
> 
> Summarizing here, when a test fails, it would be good to pause test execution
> and let the developer poke around the system to see current status of system.

Thanks a lot!

I updated the documentation a bit because it talked about the userspace
datapath testsuite even though it was part of the documentation for the
main set of unit tests.

I applied this to master, thanks again!
Vasu Dasari July 16, 2019, 5:01 p.m. UTC | #4
Thanks Ben and Aaron for your review and comments.


*Vasu Dasari*


On Tue, Jul 16, 2019 at 12:57 PM Ben Pfaff <blp@ovn.org> wrote:

> On Mon, Jul 15, 2019 at 05:15:01PM -0400, Vasu Dasari wrote:
> > Origins for this patch are captured at
> > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html
> .
> >
> > Summarizing here, when a test fails, it would be good to pause test
> execution
> > and let the developer poke around the system to see current status of
> system.
>
> Thanks a lot!
>
> I updated the documentation a bit because it talked about the userspace
> datapath testsuite even though it was part of the documentation for the
> main set of unit tests.
>
> I applied this to master, thanks again!
>

Patch
diff mbox series

diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst
index a8892e1c1..73077042c 100644
--- a/Documentation/topics/testing.rst
+++ b/Documentation/topics/testing.rst
@@ -81,6 +81,37 @@  To see a complete list of test options, run::
 The results of a testing run are reported in ``tests/testsuite.log``. Report
 report test failures as bugs and include the ``testsuite.log`` in your report.
 
+Debugging unit tests
+++++++++++++++++++++
+
+To initiate debugging from artifacts generated from `make check` run, use
+following instructions. For example, to re-run test case 139::
+
+   export OVS_PAUSE_TEST=1
+   cd tests/system-userspace-testsuite.dir/139
+   sudo -E ./run
+
+When error occurs, above command would display something like this::
+
+   Set environment variable to use various ovs utilities
+   export OVS_RUNDIR=<dir>/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
+   Press ENTER to continue:
+
+And from another window, one can execute ovs-xxx commands like::
+
+   export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
+   $ ovs-ofctl dump-ports br0
+   .
+   .
+
+Once done with investigation, press ENTER to perform cleanup operation.
+
+To be able to pause on error while performing `make check`, one can do::
+
+  $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'
+
+and then follow similar on screen instructions
+
 .. note::
   Sometimes a few tests may fail on some runs but not others. This is usually a
   bug in the testsuite, not a bug in Open vSwitch itself. If you find that a
diff --git a/tests/ovs-macros.at b/tests/ovs-macros.at
index 10593429d..b6add7fda 100644
--- a/tests/ovs-macros.at
+++ b/tests/ovs-macros.at
@@ -35,11 +35,33 @@  m4_divert_push([PREPARE_TESTS])
 # directory.
 ovs_init() {
     ovs_base=`pwd`
-    trap '. "$ovs_base/cleanup"' 0
+    trap ovs_on_exit 0
     : > cleanup
     ovs_setenv
 }
 
+# Catch testsuite error condition and cleanup test environment by tearing down
+# all interfaces and processes spawned.
+# User has an option to leave the test environment in error state so that system
+# can be poked around to get more information. User can enable this option by setting
+# environment variable OVS_PAUSE_TEST=1. User needs to press CTRL-D to resume the
+# cleanup operation.
+ovs_pause() {
+    echo "====================================================="
+    echo "Set following environment variable to use various ovs utilities"
+    echo "export OVS_RUNDIR=$ovs_base"
+    echo "Press ENTER to continue: "
+    read
+}
+
+ovs_on_exit () {
+    if [ ! -z "${OVS_PAUSE_TEST}" ] && [ -z $at_verbose ]; then
+        trap '' INT
+        ovs_pause
+    fi
+    . "$ovs_base/cleanup"
+}
+
 # With no parameter or an empty parameter, sets the OVS_*DIR
 # environment variables to point to $ovs_base, the base directory in
 # which the test is running.