[PULL,10/19] tests/boot_linux_console: increase timeout

From: Cleber Rosa <crosa@redhat.com>

When running on very low powered environments, some tests may time out
causing false negatives.  As a conservative change, and for
considering that human time (investigating false negatives) is worth
more than some extra machine cycles (and time), let's increase the
overall timeout.

CC: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20190312171824.5134-11-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/acceptance/boot_linux_console.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

On Thu, May 02, 2019 at 09:41:21PM -0300, Eduardo Habkost wrote:
> From: Cleber Rosa <crosa@redhat.com>
> 
> When running on very low powered environments, some tests may time out
> causing false negatives.  As a conservative change, and for
> considering that human time (investigating false negatives) is worth
> more than some extra machine cycles (and time), let's increase the
> overall timeout.

Tests can also timeout due to slow downloads of test kernels.
Any chance to run the downloads without timeout?

cheers,
  Gerd

On Wed, May 08, 2019 at 12:28:24PM +0200, Gerd Hoffmann wrote:
> On Thu, May 02, 2019 at 09:41:21PM -0300, Eduardo Habkost wrote:
> > From: Cleber Rosa <crosa@redhat.com>
> > 
> > When running on very low powered environments, some tests may time out
> > causing false negatives.  As a conservative change, and for
> > considering that human time (investigating false negatives) is worth
> > more than some extra machine cycles (and time), let's increase the
> > overall timeout.
> 
> Tests can also timeout due to slow downloads of test kernels.
> Any chance to run the downloads without timeout?

I acknowledge this is an issue, and have thought about two possible
ways to solve it:

 1) Downloading/caching/checking all the test assets in a job "pre-tests"
    plugin.  This would involve/require:

   - https://trello.com/c/WPd4FrIy/1479-add-support-to-specify-assets-in-test-docstring
   - https://trello.com/c/CKP7YS6G/1481-on-cache-check-for-asset-fetcher

 2) Report the test phase (say, setUp()) to the test runner, which
    would allow the runner to:

   - Not take that time into account for the test (similar to
     https://github.com/avocado-framework/avocado/pull/3076)
   - Retry the setUp a number of times (if the test opts in or doesn't
     opt out)

I'm very much interested in your opinion so we can evolve the idea into
implementation.

Thanks!
- Cleber

> 
> cheers,
>   Gerd
>

Hi,

> > Tests can also timeout due to slow downloads of test kernels.
> > Any chance to run the downloads without timeout?
> 
> I acknowledge this is an issue, and have thought about two possible
> ways to solve it:
> 
>  1) Downloading/caching/checking all the test assets in a job "pre-tests"
>     plugin.

>  2) Report the test phase (say, setUp()) to the test runner, which
>     would allow the runner to:

> I'm very much interested in your opinion so we can evolve the idea into
> implementation.

(1) is the better approach I think.  That way you can give better
feedback to the user, along the lines of "download 2 of 5 in progress".
Also it allows for better management of the assets, you can have tools
to list them / fetch them / copy them from one to another test machine
/ find & cleanup obsolete assets (i.e. Fedora 28 kernel after switching
tests to Fedora 29).

(2) is probably still useful, to report phases of longer running
tests and maybe have separate timeouts for each phase.  But for
assets I think approach (1) is better than a "download asset"
phase.

cheers,
  Gerd

On Thu, May 09, 2019 at 06:40:40AM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > > Tests can also timeout due to slow downloads of test kernels.
> > > Any chance to run the downloads without timeout?
> > 
> > I acknowledge this is an issue, and have thought about two possible
> > ways to solve it:
> > 
> >  1) Downloading/caching/checking all the test assets in a job "pre-tests"
> >     plugin.
> 
> >  2) Report the test phase (say, setUp()) to the test runner, which
> >     would allow the runner to:
> 
> > I'm very much interested in your opinion so we can evolve the idea into
> > implementation.
> 
> (1) is the better approach I think.  That way you can give better
> feedback to the user, along the lines of "download 2 of 5 in progress".
> Also it allows for better management of the assets, you can have tools
> to list them / fetch them / copy them from one to another test machine
> / find & cleanup obsolete assets (i.e. Fedora 28 kernel after switching
> tests to Fedora 29).
> 
> (2) is probably still useful, to report phases of longer running
> tests and maybe have separate timeouts for each phase.  But for
> assets I think approach (1) is better than a "download asset"
> phase.
>

I also think that approach #1 is simpler and saner, but thinking about
where we're going with the test runner development, I started to have
doubts about it.  The reason is that we're adding parallel and multi
environment (process, machine, container) execution capabilities to the
runner. Dogfood version is here:

  https://github.com/avocado-framework/avocado/pull/3111

By doing all download/caching before any single test starts, we may be
wasting a lot of CPU time that could be used for running tests, making
the overall job execution much longer.

And with regards to test phases, the runner could distinguish between
setup and test phases, possibly re-running the setup phase a number of
times until sucessfull completion, improving even further the
reliability of the *test* results (excluding the setup phase).

Anyway, this is only brainstorm at this point, so I'll let you know
when this moves forward.

> cheers,
>   Gerd
> 
> 

FIY: one of the new runner planned features, is based on a question
use case you mention, about how to run tests built into a minimal
image, such as an initrd.

Regards,
- Cleber.

Hi,

> I also think that approach #1 is simpler and saner, but thinking about
> where we're going with the test runner development, I started to have
> doubts about it.  The reason is that we're adding parallel and multi
> environment (process, machine, container) execution capabilities to the
> runner. Dogfood version is here:
> 
>   https://github.com/avocado-framework/avocado/pull/3111
> 
> By doing all download/caching before any single test starts, we may be
> wasting a lot of CPU time that could be used for running tests, making
> the overall job execution much longer.

Well, if the internet link is the bottleneck running downloads in
parallel wouldn't speed up things ...

Also given that avocado caches the downloads the common case will
probably be to not download anything.

> FIY: one of the new runner planned features, is based on a question
> use case you mention, about how to run tests built into a minimal
> image, such as an initrd.

FYI: Here is what I did: https://git.kraxel.org/cgit/drminfo/tree/tests

cheers,
  Gerd