| Message ID | 1511817671-4025-1-git-send-email-trini@konsulko.com |
|---|---|
| State | Deferred |
| Delegated to: | Tom Rini |
| Headers | show |
| Series | [U-Boot] test/run: Exit on error from tests | expand |
Hi Tom, On 27 November 2017 at 14:21, Tom Rini <trini@konsulko.com> wrote: > Currently, result isn't saved outside of the scope of run_test, so we > always exit on error here. An easier way to deal with having test/run > show failures is to exit on failure. > > Cc: Simon Glass <sjg@chromium.org> > Fixes: 2f52018c3cbb ("test/run: Report and return failure") > Signed-off-by: Tom Rini <trini@konsulko.com> > --- > Automation is only as good as the humans behind it. If you forget, like > I did, that you finally fixed Jenkins to email you every time, that lack > of emails about my non-travis jobs failing wasn't "no problems", it was > "You forgot to migrate all the Docker stuff to the new build server". > So, I'm going to apply this ASAP as I would not have let in the > u-boot-dm PR, had my setup been functioning as expected. > --- > test/run | 18 ++++-------------- > 1 file changed, 4 insertions(+), 14 deletions(-) I'm OK with this if it's what you want. This means that we will not run all the tests if one of the earlier one fails. Did you see this one? http://patchwork.ozlabs.org/patch/841453/ Regards, Simon
On Mon, Nov 27, 2017 at 02:45:14PM -0700, Simon Glass wrote: > Hi Tom, > > On 27 November 2017 at 14:21, Tom Rini <trini@konsulko.com> wrote: > > Currently, result isn't saved outside of the scope of run_test, so we > > always exit on error here. An easier way to deal with having test/run > > show failures is to exit on failure. > > > > Cc: Simon Glass <sjg@chromium.org> > > Fixes: 2f52018c3cbb ("test/run: Report and return failure") > > Signed-off-by: Tom Rini <trini@konsulko.com> > > --- > > Automation is only as good as the humans behind it. If you forget, like > > I did, that you finally fixed Jenkins to email you every time, that lack > > of emails about my non-travis jobs failing wasn't "no problems", it was > > "You forgot to migrate all the Docker stuff to the new build server". > > So, I'm going to apply this ASAP as I would not have let in the > > u-boot-dm PR, had my setup been functioning as expected. > > --- > > test/run | 18 ++++-------------- > > 1 file changed, 4 insertions(+), 14 deletions(-) > > I'm OK with this if it's what you want. This means that we will not > run all the tests if one of the earlier one fails. > > Did you see this one? > > http://patchwork.ozlabs.org/patch/841453/ I'm playing a little catch-up and that's fine too. I've unearthed another bit of breakage to fix now (because the time PRs break my setup is the time I break runtime of my setup and assume it's working).
Hi Tom, On 27 November 2017 at 18:11, Tom Rini <trini@konsulko.com> wrote: > On Mon, Nov 27, 2017 at 02:45:14PM -0700, Simon Glass wrote: >> Hi Tom, >> >> On 27 November 2017 at 14:21, Tom Rini <trini@konsulko.com> wrote: >> > Currently, result isn't saved outside of the scope of run_test, so we >> > always exit on error here. An easier way to deal with having test/run >> > show failures is to exit on failure. >> > >> > Cc: Simon Glass <sjg@chromium.org> >> > Fixes: 2f52018c3cbb ("test/run: Report and return failure") >> > Signed-off-by: Tom Rini <trini@konsulko.com> >> > --- >> > Automation is only as good as the humans behind it. If you forget, like >> > I did, that you finally fixed Jenkins to email you every time, that lack >> > of emails about my non-travis jobs failing wasn't "no problems", it was >> > "You forgot to migrate all the Docker stuff to the new build server". >> > So, I'm going to apply this ASAP as I would not have let in the >> > u-boot-dm PR, had my setup been functioning as expected. >> > --- >> > test/run | 18 ++++-------------- >> > 1 file changed, 4 insertions(+), 14 deletions(-) >> >> I'm OK with this if it's what you want. This means that we will not >> run all the tests if one of the earlier one fails. >> >> Did you see this one? >> >> http://patchwork.ozlabs.org/patch/841453/ > > I'm playing a little catch-up and that's fine too. I've unearthed > another bit of breakage to fix now (because the time PRs break my setup > is the time I break runtime of my setup and assume it's working). Yes, OK, sorry about the churn. As you can tell I'm trying to get it so all the tests we have run automatically both with 'make tests' and with travis-ci. Still a way to go but it is closer. Regards, Simon
On Mon, Nov 27, 2017 at 09:32:06PM -0700, Simon Glass wrote: > Hi Tom, > > On 27 November 2017 at 18:11, Tom Rini <trini@konsulko.com> wrote: > > On Mon, Nov 27, 2017 at 02:45:14PM -0700, Simon Glass wrote: > >> Hi Tom, > >> > >> On 27 November 2017 at 14:21, Tom Rini <trini@konsulko.com> wrote: > >> > Currently, result isn't saved outside of the scope of run_test, so we > >> > always exit on error here. An easier way to deal with having test/run > >> > show failures is to exit on failure. > >> > > >> > Cc: Simon Glass <sjg@chromium.org> > >> > Fixes: 2f52018c3cbb ("test/run: Report and return failure") > >> > Signed-off-by: Tom Rini <trini@konsulko.com> > >> > --- > >> > Automation is only as good as the humans behind it. If you forget, like > >> > I did, that you finally fixed Jenkins to email you every time, that lack > >> > of emails about my non-travis jobs failing wasn't "no problems", it was > >> > "You forgot to migrate all the Docker stuff to the new build server". > >> > So, I'm going to apply this ASAP as I would not have let in the > >> > u-boot-dm PR, had my setup been functioning as expected. > >> > --- > >> > test/run | 18 ++++-------------- > >> > 1 file changed, 4 insertions(+), 14 deletions(-) > >> > >> I'm OK with this if it's what you want. This means that we will not > >> run all the tests if one of the earlier one fails. > >> > >> Did you see this one? > >> > >> http://patchwork.ozlabs.org/patch/841453/ > > > > I'm playing a little catch-up and that's fine too. I've unearthed > > another bit of breakage to fix now (because the time PRs break my setup > > is the time I break runtime of my setup and assume it's working). > > Yes, OK, sorry about the churn. As you can tell I'm trying to get it > so all the tests we have run automatically both with 'make tests' and > with travis-ci. Still a way to go but it is closer. I'm looking forward to it all being in there :) Thanks!
diff --git a/test/run b/test/run index caee4f83f2c1..47dfed7bdca7 100755 --- a/test/run +++ b/test/run @@ -1,24 +1,14 @@ #!/bin/bash -run_test() { - $@ - [ $? -ne 0 ] && result=$((result+1)) - echo "result $result" -} +# Exit on error +set +e # Run all tests that the standard sandbox build can support -run_test ./test/py/test.py --bd sandbox --build +./test/py/test.py --bd sandbox --build # Run tests which require sandbox_spl -run_test ./test/py/test.py --bd sandbox_spl --build -k \ +./test/py/test.py --bd sandbox_spl --build -k \ test/py/tests/test_ofplatdata.py # Run tests for the flat DT version of sandbox ./test/py/test.py --bd sandbox_flattree --build - -if [ $result == 0 ]; then - echo "Tests passed!" -else - echo "Tests FAILED" - exit 1 -fi
Currently, result isn't saved outside of the scope of run_test, so we always exit on error here. An easier way to deal with having test/run show failures is to exit on failure. Cc: Simon Glass <sjg@chromium.org> Fixes: 2f52018c3cbb ("test/run: Report and return failure") Signed-off-by: Tom Rini <trini@konsulko.com> --- Automation is only as good as the humans behind it. If you forget, like I did, that you finally fixed Jenkins to email you every time, that lack of emails about my non-travis jobs failing wasn't "no problems", it was "You forgot to migrate all the Docker stuff to the new build server". So, I'm going to apply this ASAP as I would not have let in the u-boot-dm PR, had my setup been functioning as expected. --- test/run | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)