| Message ID | 20211127121609.26837-1-saginakash@gmail.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [RFC] ltp-pan: Report failure if testcases failed to run | expand |
Ping On Sat, Nov 27, 2021, 14:16 Sagi Nakash <saginakash@gmail.com> wrote: > In case a test fails to run in run_child() for some reason (signaled via > SIGTERM/SIGTOP, execve() failure, etc.), runltp still reports success: > "INFO: ltp-pan reported all tests PASS". > > Failed tests are reported via check_pids(), by checking their exit > status, yet it doesn't catch tests which weren't able to run at all. > > exit_stat holds the exit status code: 0 in case of success in all tests, > otherwise, the number of failed tests. > > Fix this by increasing exit_stat also when a test fails to run. > > Signed-off-by: Sagi Nakash <saginakash@gmail.com> > --- > pan/ltp-pan.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c > index 0bdb51477..eff9a867d 100644 > --- a/pan/ltp-pan.c > +++ b/pan/ltp-pan.c > @@ -551,6 +551,8 @@ int main(int argc, char **argv) > ++num_active; > if ((cpid != -1 || sequential) && starts > 0) > --starts; > + if (cpid == -1 && track_exit_stats) > + exit_stat++; > > if (sequential) > if (++c >= coll->cnt) > -- > 2.25.1 > >
Ping On Fri, Dec 3, 2021, 18:41 Sagi Nakash <saginakash@gmail.com> wrote: > Ping > > On Sat, Nov 27, 2021, 14:16 Sagi Nakash <saginakash@gmail.com> wrote: > >> In case a test fails to run in run_child() for some reason (signaled via >> SIGTERM/SIGTOP, execve() failure, etc.), runltp still reports success: >> "INFO: ltp-pan reported all tests PASS". >> >> Failed tests are reported via check_pids(), by checking their exit >> status, yet it doesn't catch tests which weren't able to run at all. >> >> exit_stat holds the exit status code: 0 in case of success in all tests, >> otherwise, the number of failed tests. >> >> Fix this by increasing exit_stat also when a test fails to run. >> >> Signed-off-by: Sagi Nakash <saginakash@gmail.com> >> --- >> pan/ltp-pan.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c >> index 0bdb51477..eff9a867d 100644 >> --- a/pan/ltp-pan.c >> +++ b/pan/ltp-pan.c >> @@ -551,6 +551,8 @@ int main(int argc, char **argv) >> ++num_active; >> if ((cpid != -1 || sequential) && starts > 0) >> --starts; >> + if (cpid == -1 && track_exit_stats) >> + exit_stat++; >> >> if (sequential) >> if (++c >= coll->cnt) >> -- >> 2.25.1 >> >>
Hi, Reviewed-by: Petr Vorel <pvorel@suse.cz> @all: Do we want to include this one in the release? Kind regards, Petr > In case a test fails to run in run_child() for some reason (signaled via > SIGTERM/SIGTOP, execve() failure, etc.), runltp still reports success: > "INFO: ltp-pan reported all tests PASS". > Failed tests are reported via check_pids(), by checking their exit > status, yet it doesn't catch tests which weren't able to run at all. > exit_stat holds the exit status code: 0 in case of success in all tests, > otherwise, the number of failed tests. > Fix this by increasing exit_stat also when a test fails to run. > Signed-off-by: Sagi Nakash <saginakash@gmail.com> > --- > pan/ltp-pan.c | 2 ++ > 1 file changed, 2 insertions(+) > diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c > index 0bdb51477..eff9a867d 100644 > --- a/pan/ltp-pan.c > +++ b/pan/ltp-pan.c > @@ -551,6 +551,8 @@ int main(int argc, char **argv) > ++num_active; > if ((cpid != -1 || sequential) && starts > 0) > --starts; > + if (cpid == -1 && track_exit_stats) > + exit_stat++; > if (sequential) > if (++c >= coll->cnt)
Hi! > Reviewed-by: Petr Vorel <pvorel@suse.cz> > > @all: Do we want to include this one in the release? I do not think that this is complete enough. If we wanted to fix this we whould have to change the code to write a record to the failcmdfile as well. Also increasing the variable on each failure seems wrong as well, it may as well overflow on large enough runtest file...
diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c index 0bdb51477..eff9a867d 100644 --- a/pan/ltp-pan.c +++ b/pan/ltp-pan.c @@ -551,6 +551,8 @@ int main(int argc, char **argv) ++num_active; if ((cpid != -1 || sequential) && starts > 0) --starts; + if (cpid == -1 && track_exit_stats) + exit_stat++; if (sequential) if (++c >= coll->cnt)
In case a test fails to run in run_child() for some reason (signaled via SIGTERM/SIGTOP, execve() failure, etc.), runltp still reports success: "INFO: ltp-pan reported all tests PASS". Failed tests are reported via check_pids(), by checking their exit status, yet it doesn't catch tests which weren't able to run at all. exit_stat holds the exit status code: 0 in case of success in all tests, otherwise, the number of failed tests. Fix this by increasing exit_stat also when a test fails to run. Signed-off-by: Sagi Nakash <saginakash@gmail.com> --- pan/ltp-pan.c | 2 ++ 1 file changed, 2 insertions(+)