| Message ID | 20240104121001.1155491-1-pvorel@suse.cz |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [v2,1/1] net: tst_netload_compare(): Ignore performance failures | expand |
Hi, Reviewed-by: Martin Doucha <mdoucha@suse.cz> On 04. 01. 24 13:10, Petr Vorel wrote: > Performance failures in tests which use tst_netload_compare() (tests in > runtest/net.features) can hide a real error (e.g. test fails due missing > required kernel module). Best solution would be to have feature tests > (likely written in C API) and performance tests (the current ones). > > But until it happens, allow to ignore performance failure with > environment variable LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1. > > Signed-off-by: Petr Vorel <pvorel@suse.cz> > --- > changes v1->v2: > * Ignoring performance requires setting LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1 > (previously it was ignored by default, Martin), therefore rename the variable. > * Variable uses LTP_ prefix (user defined), thus no need to whitelist it > in tst_net.sh. > > Martin also pointed out that message can be printed multiple times, but I > don't think this is a problem (it's better that tester sees it). > > testcases/lib/tst_net.sh | 19 +++++++++++++++---- > testcases/network/README.md | 4 ++++ > 2 files changed, 19 insertions(+), 4 deletions(-) > > diff --git a/testcases/lib/tst_net.sh b/testcases/lib/tst_net.sh > index 2ed0e787f..6cf7f2fcb 100644 > --- a/testcases/lib/tst_net.sh > +++ b/testcases/lib/tst_net.sh > @@ -1,7 +1,7 @@ > #!/bin/sh > # SPDX-License-Identifier: GPL-2.0-or-later > # Copyright (c) 2014-2017 Oracle and/or its affiliates. All Rights Reserved. > -# Copyright (c) 2016-2023 Petr Vorel <pvorel@suse.cz> > +# Copyright (c) 2016-2024 Petr Vorel <pvorel@suse.cz> > # Author: Alexey Kodanev <alexey.kodanev@oracle.com> > > [ -n "$TST_LIB_NET_LOADED" ] && return 0 > @@ -863,22 +863,33 @@ tst_netload() > # TIME: time that is compared to the base one > # THRESHOD_LOW: lower limit for TFAIL > # THRESHOD_HIGH: upper limit for TWARN > +# > +# Slow performance can be ignored with setting environment variable > +# LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1 > tst_netload_compare() > { > local base_time=$1 > local new_time=$2 > local threshold_low=$3 > local threshold_hi=$4 > + local ttype='TFAIL' > + local msg res > > if [ -z "$base_time" -o -z "$new_time" -o -z "$threshold_low" ]; then > tst_brk_ TBROK "tst_netload_compare: invalid argument(s)" > fi > > - local res=$(((base_time - new_time) * 100 / base_time)) > - local msg="performance result is ${res}%" > + res=$(((base_time - new_time) * 100 / base_time)) > + msg="performance result is ${res}%" > > if [ "$res" -lt "$threshold_low" ]; then > - tst_res_ TFAIL "$msg < threshold ${threshold_low}%" > + if [ "$LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE" = 1 ]; then > + ttype='TINFO'; > + tst_res_ TINFO "WARNING: slow performance is not treated as error due LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1" > + else > + tst_res_ TINFO "Following slow performance can be ignored with LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1" > + fi > + tst_res_ $ttype "$msg < threshold ${threshold_low}%" > return > fi > > diff --git a/testcases/network/README.md b/testcases/network/README.md > index a0a1d3d95..e1b1296d6 100644 > --- a/testcases/network/README.md > +++ b/testcases/network/README.md > @@ -84,6 +84,10 @@ Where > Default values for all LTP network parameters are set in `testcases/lib/tst_net.sh`. > Network stress parameters are documented in `testcases/network/stress/README`. > > +Tests which use `tst_netload_compare()` test also performance. They can fail on > +overloaded SUT. To ignore performance failure and test only the network functionality, > +set `LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1` environment variable. > + > ## Debugging > Both single and two host configurations support debugging via > `TST_NET_RHOST_RUN_DEBUG=1` environment variable.
diff --git a/testcases/lib/tst_net.sh b/testcases/lib/tst_net.sh index 2ed0e787f..6cf7f2fcb 100644 --- a/testcases/lib/tst_net.sh +++ b/testcases/lib/tst_net.sh @@ -1,7 +1,7 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-or-later # Copyright (c) 2014-2017 Oracle and/or its affiliates. All Rights Reserved. -# Copyright (c) 2016-2023 Petr Vorel <pvorel@suse.cz> +# Copyright (c) 2016-2024 Petr Vorel <pvorel@suse.cz> # Author: Alexey Kodanev <alexey.kodanev@oracle.com> [ -n "$TST_LIB_NET_LOADED" ] && return 0 @@ -863,22 +863,33 @@ tst_netload() # TIME: time that is compared to the base one # THRESHOD_LOW: lower limit for TFAIL # THRESHOD_HIGH: upper limit for TWARN +# +# Slow performance can be ignored with setting environment variable +# LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1 tst_netload_compare() { local base_time=$1 local new_time=$2 local threshold_low=$3 local threshold_hi=$4 + local ttype='TFAIL' + local msg res if [ -z "$base_time" -o -z "$new_time" -o -z "$threshold_low" ]; then tst_brk_ TBROK "tst_netload_compare: invalid argument(s)" fi - local res=$(((base_time - new_time) * 100 / base_time)) - local msg="performance result is ${res}%" + res=$(((base_time - new_time) * 100 / base_time)) + msg="performance result is ${res}%" if [ "$res" -lt "$threshold_low" ]; then - tst_res_ TFAIL "$msg < threshold ${threshold_low}%" + if [ "$LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE" = 1 ]; then + ttype='TINFO'; + tst_res_ TINFO "WARNING: slow performance is not treated as error due LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1" + else + tst_res_ TINFO "Following slow performance can be ignored with LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1" + fi + tst_res_ $ttype "$msg < threshold ${threshold_low}%" return fi diff --git a/testcases/network/README.md b/testcases/network/README.md index a0a1d3d95..e1b1296d6 100644 --- a/testcases/network/README.md +++ b/testcases/network/README.md @@ -84,6 +84,10 @@ Where Default values for all LTP network parameters are set in `testcases/lib/tst_net.sh`. Network stress parameters are documented in `testcases/network/stress/README`. +Tests which use `tst_netload_compare()` test also performance. They can fail on +overloaded SUT. To ignore performance failure and test only the network functionality, +set `LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1` environment variable. + ## Debugging Both single and two host configurations support debugging via `TST_NET_RHOST_RUN_DEBUG=1` environment variable.
Performance failures in tests which use tst_netload_compare() (tests in runtest/net.features) can hide a real error (e.g. test fails due missing required kernel module). Best solution would be to have feature tests (likely written in C API) and performance tests (the current ones). But until it happens, allow to ignore performance failure with environment variable LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1. Signed-off-by: Petr Vorel <pvorel@suse.cz> --- changes v1->v2: * Ignoring performance requires setting LTP_NET_FEATURES_IGNORE_PERFORMANCE_FAILURE=1 (previously it was ignored by default, Martin), therefore rename the variable. * Variable uses LTP_ prefix (user defined), thus no need to whitelist it in tst_net.sh. Martin also pointed out that message can be printed multiple times, but I don't think this is a problem (it's better that tester sees it). testcases/lib/tst_net.sh | 19 +++++++++++++++---- testcases/network/README.md | 4 ++++ 2 files changed, 19 insertions(+), 4 deletions(-)