[ovs-dev,v1] ovs-macros: An option to suspend test execution on error
diff mbox series

Message ID 20190715192242.82742-1-vdasari@gmail.com
State Superseded
Headers show
Series
  • [ovs-dev,v1] ovs-macros: An option to suspend test execution on error
Related show

Commit Message

Vasu Dasari July 15, 2019, 7:22 p.m. UTC
Origins for this patch are captured at
https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html.

Summarizing here, when a test fails, it would be good to pause test execution
and let the developer poke around the system to see current status of system.

As part of this patch, made a small tweaks to ovs-macros.at, so that when test
suite fails, ovs_on_exit() function will be called. And in this function, a check
is made to see if an environment variable to OVS_PAUSE_TEST is set. If it is
set, then test suite is paused and will continue to wait for user input
Ctrl-D. Meanwhile user can poke around the system to see why test case has
failed. Once done with investigation, user can press ctrl-d to cleanup the
test suite.

For example, to re-run test case 139:

export OVS_PAUSE_TEST=1
cd tests/system-userspace-testsuite.dir/139
sudo -E ./run

When error occurs, above command would display something like this:
=====================================================
Set environment variable to use various ovs utilities
export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
Press ctrl-d to continue:

=====================================================
And from another window, one can execute ovs-xxx commands like:
export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
$ ovs-ofctl dump-ports br0
.
.

To be able to pause while performing `make check`, one can do:
$ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'

Signed-off-by: Vasu Dasari <vdasari@gmail.com>
Reviewed-by: Ben Pfaff <blp@ovn.org>
---
v0 -> v1:
Discussion at: https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360589.html
1. Incorporated review commanets from Ben.
2. Changed topics/testing.rst to document this trick
3. Made this trick work via "make check" as well, by issuing command: "OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'"

---
 Documentation/topics/testing.rst |  6 ++++++
 tests/ovs-macros.at              | 24 +++++++++++++++++++++++-
 2 files changed, 29 insertions(+), 1 deletion(-)

Comments

Aaron Conole July 15, 2019, 7:48 p.m. UTC | #1
Hi Vasu,

Vasu Dasari <vdasari@gmail.com> writes:

> Origins for this patch are captured at
> https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html.
>
> Summarizing here, when a test fails, it would be good to pause test execution
> and let the developer poke around the system to see current status of system.
>
> As part of this patch, made a small tweaks to ovs-macros.at, so that when test
> suite fails, ovs_on_exit() function will be called. And in this function, a check
> is made to see if an environment variable to OVS_PAUSE_TEST is set. If it is
> set, then test suite is paused and will continue to wait for user input
> Ctrl-D. Meanwhile user can poke around the system to see why test case has
> failed. Once done with investigation, user can press ctrl-d to cleanup the
> test suite.
>

From this point  
   vvvvvvvvvvvvvvvvvv

> For example, to re-run test case 139:
>
> export OVS_PAUSE_TEST=1
> cd tests/system-userspace-testsuite.dir/139
> sudo -E ./run
>
> When error occurs, above command would display something like this:
> =====================================================
> Set environment variable to use various ovs utilities
> export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> Press ctrl-d to continue:
>
> =====================================================
> And from another window, one can execute ovs-xxx commands like:
> export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> $ ovs-ofctl dump-ports br0
> .
> .
>
> To be able to pause while performing `make check`, one can do:
> $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'
>

To here ^^^^^^^^^^

I would propose to put this information in the documentation.  A new
developer may not actually trawl through the commits, but this
information would help such a developer (or even a not-so-new developer,
like me).

> Signed-off-by: Vasu Dasari <vdasari@gmail.com>
> Reviewed-by: Ben Pfaff <blp@ovn.org>
> ---
> v0 -> v1:
> Discussion at: https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360589.html
> 1. Incorporated review commanets from Ben.
> 2. Changed topics/testing.rst to document this trick
> 3. Made this trick work via "make check" as well, by issuing command: "OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'"
>
> ---
>  Documentation/topics/testing.rst |  6 ++++++
>  tests/ovs-macros.at              | 24 +++++++++++++++++++++++-
>  2 files changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst
> index a8892e1c1..058b3986f 100644
> --- a/Documentation/topics/testing.rst
> +++ b/Documentation/topics/testing.rst
> @@ -74,6 +74,12 @@ To run tests matching a keyword, e.g. ``ovsdb``, run::
>  
>      $ make check TESTSUITEFLAGS='-k ovsdb'
>  
> +To pause at a failing test (e.g. ``ovsdb``) and inspect the system in failed
> +state, follow the instructions given out by issuing following command to use 
> +various ovs-* utilities::
> +
> +    $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v -k ovsdb'
> +
>  To see a complete list of test options, run::
>  
>      $ make check TESTSUITEFLAGS=--help
> diff --git a/tests/ovs-macros.at b/tests/ovs-macros.at
> index 10593429d..e58b0a1a0 100644
> --- a/tests/ovs-macros.at
> +++ b/tests/ovs-macros.at
> @@ -35,11 +35,33 @@ m4_divert_push([PREPARE_TESTS])
>  # directory.
>  ovs_init() {
>      ovs_base=`pwd`
> -    trap '. "$ovs_base/cleanup"' 0
> +    trap ovs_on_exit 0
>      : > cleanup
>      ovs_setenv
>  }
>  
> +# Catch testsuite error condition and cleanup test environment by tearing down
> +# all interfaces and processes spawned.
> +# User has an option to leave the test environment in error state so that system
> +# can be poked around to get more information. User can enable this option by setting
> +# environment variable OVS_PAUSE_TEST=1. User needs to press CTRL-D to resume the
> +# cleanup operation.
> +ovs_pause() {
> +    $as_echo "====================================================="
> +    $as_echo "Set following environment variable to use various ovs utilities"
> +    $as_echo "export OVS_RUNDIR=$ovs_base"
> +    $as_echo "Press ENTER to continue: "
> +    read 
> +}
> +
> +ovs_on_exit () {
> +    if [ ! -z "${OVS_PAUSE_TEST}" ] && [ -z $at_verbose ]; then
> +        trap '' INT
> +        ovs_pause
> +    fi
> +    . "$ovs_base/cleanup"
> +}
> +
>  # With no parameter or an empty parameter, sets the OVS_*DIR
>  # environment variables to point to $ovs_base, the base directory in
>  # which the test is running.
Ben Pfaff July 15, 2019, 7:58 p.m. UTC | #2
On Mon, Jul 15, 2019 at 03:22:42PM -0400, Vasu Dasari wrote:
> Origins for this patch are captured at
> https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html.

...

Aaron provided helpful comments.  In addition:

> +ovs_pause() {
> +    $as_echo "====================================================="
> +    $as_echo "Set following environment variable to use various ovs utilities"
> +    $as_echo "export OVS_RUNDIR=$ovs_base"
> +    $as_echo "Press ENTER to continue: "
> +    read 
> +}

The above uses $as_echo, which is what the AS_ECHO([...]) macro expands
into.  It is better to use AS_ECHO itself, since in theory the
implementation could change in a later version of Autoconf.

Thanks,

Ben.
Vasu Dasari July 15, 2019, 8:01 p.m. UTC | #3
Aaron,

I see that documentation for topics/testing is mostly one liners and
precise. And hence I chose one liner as well. If I have to put all your
recommended test, I think it is better to have a subsection, "Debugging
unit tests". What do you think?

-Vasu

*Vasu Dasari*


On Mon, Jul 15, 2019 at 3:48 PM Aaron Conole <aconole@redhat.com> wrote:

> Hi Vasu,
>
> Vasu Dasari <vdasari@gmail.com> writes:
>
> > Origins for this patch are captured at
> > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html
> .
> >
> > Summarizing here, when a test fails, it would be good to pause test
> execution
> > and let the developer poke around the system to see current status of
> system.
> >
> > As part of this patch, made a small tweaks to ovs-macros.at, so that
> when test
> > suite fails, ovs_on_exit() function will be called. And in this
> function, a check
> > is made to see if an environment variable to OVS_PAUSE_TEST is set. If
> it is
> > set, then test suite is paused and will continue to wait for user input
> > Ctrl-D. Meanwhile user can poke around the system to see why test case
> has
> > failed. Once done with investigation, user can press ctrl-d to cleanup
> the
> > test suite.
> >
>
> From this point
>    vvvvvvvvvvvvvvvvvv
>
> > For example, to re-run test case 139:
> >
> > export OVS_PAUSE_TEST=1
> > cd tests/system-userspace-testsuite.dir/139
> > sudo -E ./run
> >
> > When error occurs, above command would display something like this:
> > =====================================================
> > Set environment variable to use various ovs utilities
> > export
> OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> > Press ctrl-d to continue:
> >
> > =====================================================
> > And from another window, one can execute ovs-xxx commands like:
> > export
> OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> > $ ovs-ofctl dump-ports br0
> > .
> > .
> >
> > To be able to pause while performing `make check`, one can do:
> > $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'
> >
>
> To here ^^^^^^^^^^
>
> I would propose to put this information in the documentation.  A new
> developer may not actually trawl through the commits, but this
> information would help such a developer (or even a not-so-new developer,
> like me).
>
> > Signed-off-by: Vasu Dasari <vdasari@gmail.com>
> > Reviewed-by: Ben Pfaff <blp@ovn.org>
> > ---
> > v0 -> v1:
> > Discussion at:
> https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360589.html
> > 1. Incorporated review commanets from Ben.
> > 2. Changed topics/testing.rst to document this trick
> > 3. Made this trick work via "make check" as well, by issuing command:
> "OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'"
> >
> > ---
> >  Documentation/topics/testing.rst |  6 ++++++
> >  tests/ovs-macros.at              | 24 +++++++++++++++++++++++-
> >  2 files changed, 29 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/topics/testing.rst
> b/Documentation/topics/testing.rst
> > index a8892e1c1..058b3986f 100644
> > --- a/Documentation/topics/testing.rst
> > +++ b/Documentation/topics/testing.rst
> > @@ -74,6 +74,12 @@ To run tests matching a keyword, e.g. ``ovsdb``, run::
> >
> >      $ make check TESTSUITEFLAGS='-k ovsdb'
> >
> > +To pause at a failing test (e.g. ``ovsdb``) and inspect the system in
> failed
> > +state, follow the instructions given out by issuing following command
> to use
> > +various ovs-* utilities::
> > +
> > +    $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v -k ovsdb'
> > +
> >  To see a complete list of test options, run::
> >
> >      $ make check TESTSUITEFLAGS=--help
> > diff --git a/tests/ovs-macros.at b/tests/ovs-macros.at
> > index 10593429d..e58b0a1a0 100644
> > --- a/tests/ovs-macros.at
> > +++ b/tests/ovs-macros.at
> > @@ -35,11 +35,33 @@ m4_divert_push([PREPARE_TESTS])
> >  # directory.
> >  ovs_init() {
> >      ovs_base=`pwd`
> > -    trap '. "$ovs_base/cleanup"' 0
> > +    trap ovs_on_exit 0
> >      : > cleanup
> >      ovs_setenv
> >  }
> >
> > +# Catch testsuite error condition and cleanup test environment by
> tearing down
> > +# all interfaces and processes spawned.
> > +# User has an option to leave the test environment in error state so
> that system
> > +# can be poked around to get more information. User can enable this
> option by setting
> > +# environment variable OVS_PAUSE_TEST=1. User needs to press CTRL-D to
> resume the
> > +# cleanup operation.
> > +ovs_pause() {
> > +    $as_echo "====================================================="
> > +    $as_echo "Set following environment variable to use various ovs
> utilities"
> > +    $as_echo "export OVS_RUNDIR=$ovs_base"
> > +    $as_echo "Press ENTER to continue: "
> > +    read
> > +}
> > +
> > +ovs_on_exit () {
> > +    if [ ! -z "${OVS_PAUSE_TEST}" ] && [ -z $at_verbose ]; then
> > +        trap '' INT
> > +        ovs_pause
> > +    fi
> > +    . "$ovs_base/cleanup"
> > +}
> > +
> >  # With no parameter or an empty parameter, sets the OVS_*DIR
> >  # environment variables to point to $ovs_base, the base directory in
> >  # which the test is running.
>
Vasu Dasari July 15, 2019, 8:05 p.m. UTC | #4
Aaron,

I see that documentation for topics/testing is mostly one liners and
precise. And hence I chose one liner as well. If I have to put all your
recommended test, I think it is better to have a subsection, "Debugging
unit tests". What do you think?

Ben,

Will do. Also, note that I am using at_verbose variable if we are in
verbose mode or not. If user has not run the test suite in verbose mode,
none of the instructions to set the OVS_RUNDIR variable, etc will appear on
screen, and hence user will not have a clue why script is pausing. So, I am
also qualifying the check for at_verbose variable. I hope it is ok to use
the at_verbose variable.

-Vasu

*Vasu Dasari*


On Mon, Jul 15, 2019 at 3:59 PM Ben Pfaff <blp@ovn.org> wrote:

> On Mon, Jul 15, 2019 at 03:22:42PM -0400, Vasu Dasari wrote:
> > Origins for this patch are captured at
> > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html
> .
>
> ...
>
> Aaron provided helpful comments.  In addition:
>
> > +ovs_pause() {
> > +    $as_echo "====================================================="
> > +    $as_echo "Set following environment variable to use various ovs
> utilities"
> > +    $as_echo "export OVS_RUNDIR=$ovs_base"
> > +    $as_echo "Press ENTER to continue: "
> > +    read
> > +}
>
> The above uses $as_echo, which is what the AS_ECHO([...]) macro expands
> into.  It is better to use AS_ECHO itself, since in theory the
> implementation could change in a later version of Autoconf.
>
> Thanks,
>
> Ben.
>
Ben Pfaff July 15, 2019, 8:11 p.m. UTC | #5
I'm not Aaron but that does seems reasonable to me.

On Mon, Jul 15, 2019 at 04:01:26PM -0400, Vasu Dasari wrote:
> Aaron,
> 
> I see that documentation for topics/testing is mostly one liners and
> precise. And hence I chose one liner as well. If I have to put all your
> recommended test, I think it is better to have a subsection, "Debugging
> unit tests". What do you think?
> 
> -Vasu
> 
> *Vasu Dasari*
> 
> 
> On Mon, Jul 15, 2019 at 3:48 PM Aaron Conole <aconole@redhat.com> wrote:
> 
> > Hi Vasu,
> >
> > Vasu Dasari <vdasari@gmail.com> writes:
> >
> > > Origins for this patch are captured at
> > > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html
> > .
> > >
> > > Summarizing here, when a test fails, it would be good to pause test
> > execution
> > > and let the developer poke around the system to see current status of
> > system.
> > >
> > > As part of this patch, made a small tweaks to ovs-macros.at, so that
> > when test
> > > suite fails, ovs_on_exit() function will be called. And in this
> > function, a check
> > > is made to see if an environment variable to OVS_PAUSE_TEST is set. If
> > it is
> > > set, then test suite is paused and will continue to wait for user input
> > > Ctrl-D. Meanwhile user can poke around the system to see why test case
> > has
> > > failed. Once done with investigation, user can press ctrl-d to cleanup
> > the
> > > test suite.
> > >
> >
> > From this point
> >    vvvvvvvvvvvvvvvvvv
> >
> > > For example, to re-run test case 139:
> > >
> > > export OVS_PAUSE_TEST=1
> > > cd tests/system-userspace-testsuite.dir/139
> > > sudo -E ./run
> > >
> > > When error occurs, above command would display something like this:
> > > =====================================================
> > > Set environment variable to use various ovs utilities
> > > export
> > OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> > > Press ctrl-d to continue:
> > >
> > > =====================================================
> > > And from another window, one can execute ovs-xxx commands like:
> > > export
> > OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> > > $ ovs-ofctl dump-ports br0
> > > .
> > > .
> > >
> > > To be able to pause while performing `make check`, one can do:
> > > $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'
> > >
> >
> > To here ^^^^^^^^^^
> >
> > I would propose to put this information in the documentation.  A new
> > developer may not actually trawl through the commits, but this
> > information would help such a developer (or even a not-so-new developer,
> > like me).
> >
> > > Signed-off-by: Vasu Dasari <vdasari@gmail.com>
> > > Reviewed-by: Ben Pfaff <blp@ovn.org>
> > > ---
> > > v0 -> v1:
> > > Discussion at:
> > https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360589.html
> > > 1. Incorporated review commanets from Ben.
> > > 2. Changed topics/testing.rst to document this trick
> > > 3. Made this trick work via "make check" as well, by issuing command:
> > "OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'"
> > >
> > > ---
> > >  Documentation/topics/testing.rst |  6 ++++++
> > >  tests/ovs-macros.at              | 24 +++++++++++++++++++++++-
> > >  2 files changed, 29 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/Documentation/topics/testing.rst
> > b/Documentation/topics/testing.rst
> > > index a8892e1c1..058b3986f 100644
> > > --- a/Documentation/topics/testing.rst
> > > +++ b/Documentation/topics/testing.rst
> > > @@ -74,6 +74,12 @@ To run tests matching a keyword, e.g. ``ovsdb``, run::
> > >
> > >      $ make check TESTSUITEFLAGS='-k ovsdb'
> > >
> > > +To pause at a failing test (e.g. ``ovsdb``) and inspect the system in
> > failed
> > > +state, follow the instructions given out by issuing following command
> > to use
> > > +various ovs-* utilities::
> > > +
> > > +    $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v -k ovsdb'
> > > +
> > >  To see a complete list of test options, run::
> > >
> > >      $ make check TESTSUITEFLAGS=--help
> > > diff --git a/tests/ovs-macros.at b/tests/ovs-macros.at
> > > index 10593429d..e58b0a1a0 100644
> > > --- a/tests/ovs-macros.at
> > > +++ b/tests/ovs-macros.at
> > > @@ -35,11 +35,33 @@ m4_divert_push([PREPARE_TESTS])
> > >  # directory.
> > >  ovs_init() {
> > >      ovs_base=`pwd`
> > > -    trap '. "$ovs_base/cleanup"' 0
> > > +    trap ovs_on_exit 0
> > >      : > cleanup
> > >      ovs_setenv
> > >  }
> > >
> > > +# Catch testsuite error condition and cleanup test environment by
> > tearing down
> > > +# all interfaces and processes spawned.
> > > +# User has an option to leave the test environment in error state so
> > that system
> > > +# can be poked around to get more information. User can enable this
> > option by setting
> > > +# environment variable OVS_PAUSE_TEST=1. User needs to press CTRL-D to
> > resume the
> > > +# cleanup operation.
> > > +ovs_pause() {
> > > +    $as_echo "====================================================="
> > > +    $as_echo "Set following environment variable to use various ovs
> > utilities"
> > > +    $as_echo "export OVS_RUNDIR=$ovs_base"
> > > +    $as_echo "Press ENTER to continue: "
> > > +    read
> > > +}
> > > +
> > > +ovs_on_exit () {
> > > +    if [ ! -z "${OVS_PAUSE_TEST}" ] && [ -z $at_verbose ]; then
> > > +        trap '' INT
> > > +        ovs_pause
> > > +    fi
> > > +    . "$ovs_base/cleanup"
> > > +}
> > > +
> > >  # With no parameter or an empty parameter, sets the OVS_*DIR
> > >  # environment variables to point to $ovs_base, the base directory in
> > >  # which the test is running.
> >
0-day Robot July 15, 2019, 8:24 p.m. UTC | #6
Bleep bloop.  Greetings Vasu Dasari, I am a robot and I have tried out your patch.
Thanks for your contribution.

I encountered some error that I wasn't expecting.  See the details below.


checkpatch:
WARNING: Line has trailing whitespace
#58 FILE: Documentation/topics/testing.rst:78:
state, follow the instructions given out by issuing following command to use 

WARNING: Line has trailing whitespace
#91 FILE: tests/ovs-macros.at:54:
    read 

Lines checked: 107, Warnings: 2, Errors: 0


Please check this out.  If you feel there has been an error, please email aconole@bytheb.org

Thanks,
0-day Robot
Vasu Dasari July 15, 2019, 9:16 p.m. UTC | #7
Done. Patch is at:

[ovs-dev] [PATCH v2] ovs-macros: An option to suspend test execution on
error <https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360720.html>

<https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360720.html>

*Vasu Dasari*


On Mon, Jul 15, 2019 at 4:12 PM Ben Pfaff <blp@ovn.org> wrote:

> I'm not Aaron but that does seems reasonable to me.
>
> On Mon, Jul 15, 2019 at 04:01:26PM -0400, Vasu Dasari wrote:
> > Aaron,
> >
> > I see that documentation for topics/testing is mostly one liners and
> > precise. And hence I chose one liner as well. If I have to put all your
> > recommended test, I think it is better to have a subsection, "Debugging
> > unit tests". What do you think?
> >
> > -Vasu
> >
> > *Vasu Dasari*
> >
> >
> > On Mon, Jul 15, 2019 at 3:48 PM Aaron Conole <aconole@redhat.com> wrote:
> >
> > > Hi Vasu,
> > >
> > > Vasu Dasari <vdasari@gmail.com> writes:
> > >
> > > > Origins for this patch are captured at
> > > >
> https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html
> > > .
> > > >
> > > > Summarizing here, when a test fails, it would be good to pause test
> > > execution
> > > > and let the developer poke around the system to see current status of
> > > system.
> > > >
> > > > As part of this patch, made a small tweaks to ovs-macros.at, so that
> > > when test
> > > > suite fails, ovs_on_exit() function will be called. And in this
> > > function, a check
> > > > is made to see if an environment variable to OVS_PAUSE_TEST is set.
> If
> > > it is
> > > > set, then test suite is paused and will continue to wait for user
> input
> > > > Ctrl-D. Meanwhile user can poke around the system to see why test
> case
> > > has
> > > > failed. Once done with investigation, user can press ctrl-d to
> cleanup
> > > the
> > > > test suite.
> > > >
> > >
> > > From this point
> > >    vvvvvvvvvvvvvvvvvv
> > >
> > > > For example, to re-run test case 139:
> > > >
> > > > export OVS_PAUSE_TEST=1
> > > > cd tests/system-userspace-testsuite.dir/139
> > > > sudo -E ./run
> > > >
> > > > When error occurs, above command would display something like this:
> > > > =====================================================
> > > > Set environment variable to use various ovs utilities
> > > > export
> > >
> OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> > > > Press ctrl-d to continue:
> > > >
> > > > =====================================================
> > > > And from another window, one can execute ovs-xxx commands like:
> > > > export
> > >
> OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/system-userspace-testsuite.dir/139
> > > > $ ovs-ofctl dump-ports br0
> > > > .
> > > > .
> > > >
> > > > To be able to pause while performing `make check`, one can do:
> > > > $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'
> > > >
> > >
> > > To here ^^^^^^^^^^
> > >
> > > I would propose to put this information in the documentation.  A new
> > > developer may not actually trawl through the commits, but this
> > > information would help such a developer (or even a not-so-new
> developer,
> > > like me).
> > >
> > > > Signed-off-by: Vasu Dasari <vdasari@gmail.com>
> > > > Reviewed-by: Ben Pfaff <blp@ovn.org>
> > > > ---
> > > > v0 -> v1:
> > > > Discussion at:
> > > https://mail.openvswitch.org/pipermail/ovs-dev/2019-July/360589.html
> > > > 1. Incorporated review commanets from Ben.
> > > > 2. Changed topics/testing.rst to document this trick
> > > > 3. Made this trick work via "make check" as well, by issuing command:
> > > "OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v'"
> > > >
> > > > ---
> > > >  Documentation/topics/testing.rst |  6 ++++++
> > > >  tests/ovs-macros.at              | 24 +++++++++++++++++++++++-
> > > >  2 files changed, 29 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/Documentation/topics/testing.rst
> > > b/Documentation/topics/testing.rst
> > > > index a8892e1c1..058b3986f 100644
> > > > --- a/Documentation/topics/testing.rst
> > > > +++ b/Documentation/topics/testing.rst
> > > > @@ -74,6 +74,12 @@ To run tests matching a keyword, e.g. ``ovsdb``,
> run::
> > > >
> > > >      $ make check TESTSUITEFLAGS='-k ovsdb'
> > > >
> > > > +To pause at a failing test (e.g. ``ovsdb``) and inspect the system
> in
> > > failed
> > > > +state, follow the instructions given out by issuing following
> command
> > > to use
> > > > +various ovs-* utilities::
> > > > +
> > > > +    $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v -k ovsdb'
> > > > +
> > > >  To see a complete list of test options, run::
> > > >
> > > >      $ make check TESTSUITEFLAGS=--help
> > > > diff --git a/tests/ovs-macros.at b/tests/ovs-macros.at
> > > > index 10593429d..e58b0a1a0 100644
> > > > --- a/tests/ovs-macros.at
> > > > +++ b/tests/ovs-macros.at
> > > > @@ -35,11 +35,33 @@ m4_divert_push([PREPARE_TESTS])
> > > >  # directory.
> > > >  ovs_init() {
> > > >      ovs_base=`pwd`
> > > > -    trap '. "$ovs_base/cleanup"' 0
> > > > +    trap ovs_on_exit 0
> > > >      : > cleanup
> > > >      ovs_setenv
> > > >  }
> > > >
> > > > +# Catch testsuite error condition and cleanup test environment by
> > > tearing down
> > > > +# all interfaces and processes spawned.
> > > > +# User has an option to leave the test environment in error state so
> > > that system
> > > > +# can be poked around to get more information. User can enable this
> > > option by setting
> > > > +# environment variable OVS_PAUSE_TEST=1. User needs to press CTRL-D
> to
> > > resume the
> > > > +# cleanup operation.
> > > > +ovs_pause() {
> > > > +    $as_echo "====================================================="
> > > > +    $as_echo "Set following environment variable to use various ovs
> > > utilities"
> > > > +    $as_echo "export OVS_RUNDIR=$ovs_base"
> > > > +    $as_echo "Press ENTER to continue: "
> > > > +    read
> > > > +}
> > > > +
> > > > +ovs_on_exit () {
> > > > +    if [ ! -z "${OVS_PAUSE_TEST}" ] && [ -z $at_verbose ]; then
> > > > +        trap '' INT
> > > > +        ovs_pause
> > > > +    fi
> > > > +    . "$ovs_base/cleanup"
> > > > +}
> > > > +
> > > >  # With no parameter or an empty parameter, sets the OVS_*DIR
> > > >  # environment variables to point to $ovs_base, the base directory in
> > > >  # which the test is running.
> > >
>
Aaron Conole July 16, 2019, 1:01 p.m. UTC | #8
Vasu Dasari <vdasari@gmail.com> writes:

> Aaron,
>
> I see that documentation for topics/testing is mostly one liners and precise. And hence I chose one liner as well.
> If I have to put all your recommended test, I think it is better to have a subsection, "Debugging unit tests".
> What do you think?

It's a good idea.

> Ben,
>
> Will do. Also, note that I am using at_verbose variable if we are in verbose mode or not. If user has not run the
> test suite in verbose mode, none of the instructions to set the OVS_RUNDIR variable, etc will appear on screen,
> and hence user will not have a clue why script is pausing. So, I am also qualifying the check for at_verbose
> variable. I hope it is ok to use the at_verbose variable.
>
> -Vasu
>
> Vasu Dasari
>
> On Mon, Jul 15, 2019 at 3:59 PM Ben Pfaff <blp@ovn.org> wrote:
>
>  On Mon, Jul 15, 2019 at 03:22:42PM -0400, Vasu Dasari wrote:
>  > Origins for this patch are captured at
>  > https://mail.openvswitch.org/pipermail/ovs-discuss/2019-June/048923.html.
>
>  ...
>
>  Aaron provided helpful comments.  In addition:
>
>  > +ovs_pause() {
>  > +    $as_echo
>  "====================================================="
>  > +    $as_echo "Set following environment variable to use various ovs utilities"
>  > +    $as_echo "export OVS_RUNDIR=$ovs_base"
>  > +    $as_echo "Press ENTER to continue: "
>  > +    read 
>  > +}
>
>  The above uses $as_echo, which is what the AS_ECHO([...]) macro expands
>  into.  It is better to use AS_ECHO itself, since in theory the
>  implementation could change in a later version of Autoconf.
>
>  Thanks,
>
>  Ben.

Patch
diff mbox series

diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst
index a8892e1c1..058b3986f 100644
--- a/Documentation/topics/testing.rst
+++ b/Documentation/topics/testing.rst
@@ -74,6 +74,12 @@  To run tests matching a keyword, e.g. ``ovsdb``, run::
 
     $ make check TESTSUITEFLAGS='-k ovsdb'
 
+To pause at a failing test (e.g. ``ovsdb``) and inspect the system in failed
+state, follow the instructions given out by issuing following command to use 
+various ovs-* utilities::
+
+    $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v -k ovsdb'
+
 To see a complete list of test options, run::
 
     $ make check TESTSUITEFLAGS=--help
diff --git a/tests/ovs-macros.at b/tests/ovs-macros.at
index 10593429d..e58b0a1a0 100644
--- a/tests/ovs-macros.at
+++ b/tests/ovs-macros.at
@@ -35,11 +35,33 @@  m4_divert_push([PREPARE_TESTS])
 # directory.
 ovs_init() {
     ovs_base=`pwd`
-    trap '. "$ovs_base/cleanup"' 0
+    trap ovs_on_exit 0
     : > cleanup
     ovs_setenv
 }
 
+# Catch testsuite error condition and cleanup test environment by tearing down
+# all interfaces and processes spawned.
+# User has an option to leave the test environment in error state so that system
+# can be poked around to get more information. User can enable this option by setting
+# environment variable OVS_PAUSE_TEST=1. User needs to press CTRL-D to resume the
+# cleanup operation.
+ovs_pause() {
+    $as_echo "====================================================="
+    $as_echo "Set following environment variable to use various ovs utilities"
+    $as_echo "export OVS_RUNDIR=$ovs_base"
+    $as_echo "Press ENTER to continue: "
+    read 
+}
+
+ovs_on_exit () {
+    if [ ! -z "${OVS_PAUSE_TEST}" ] && [ -z $at_verbose ]; then
+        trap '' INT
+        ovs_pause
+    fi
+    . "$ovs_base/cleanup"
+}
+
 # With no parameter or an empty parameter, sets the OVS_*DIR
 # environment variables to point to $ovs_base, the base directory in
 # which the test is running.