Message ID | 20210203124447.2458527-1-sjg@chromium.org |
---|---|
Headers | show |
Series | test: Refactor tests to have a single test runner | expand |
On Wed, Feb 03, 2021 at 05:44:06AM -0700, Simon Glass wrote: > At present U-Boot has two broad sets of tests in the C code: driver model > tests which do a lot of pre-/post-init and command tests which do not. > > This separation makes it slightly harder to write a test, since there are > two different test-state structures and different rules for running the > two different test types. At present these rules are determined by where > the test is (actually its prefix). > > All unit tests can be run from the command line with the 'ut' command. > Since SPL does not have commands, it currently calls the test runner > directly and offers no control of which tests are run. > > This seems like a good time to refactor the tests into a unified test > runner, allowing U-Boot proper and SPL to use the same path, perhaps with > some different conditions along the way. > > This series sets up a unified runner called ut_run_list(), which runs a > set of tests from a linker_list. Driver model tests are distinguished by > a new UT_TESTF_DM flag so that the necessary init and cleanup can still > be done. > > The runner is modified to support running SPL tests that are not solely > for driver model. An example test for FIT loading is added as a > demonstration. > > In addition, some documentation is added to explain how to write tests. > > This series is available at u-boot-dm/test-working > > Changes in v3: > - Add new patch to re-enable test_ofplatdata > - Reword the SPL tests section for clarity > - Use test_set_state() throughout test-main.c instead of direct assignment > - Update the pytest collector as well > > Changes in v2: > - Use correct rst format for 'Ad-hoc tests' section > - Expand docs on how each type of test is marked > - Put the docs in tests_sandbox since it is more related to sandbox > - Put in a mention of tests_sandbox in the main testing docs > - Add a note that SPL tests can in fact be run individualy > - Document how to run all C tests with 'ut all' > - Fix 'get list' typo > - Fix conditions so non-DM SPL tests are actually run > - Allow for prefix to be NULL, to match function comment > - Add new patches to cover running an SPL test There was a small problem applying one or two of these patches that I fixed up. But then when testing locally (Ubuntu 18.04, but virtualenv setup and populated with test/py/requirements.txt) like 200 tests failed to run for Sandbox. Please check this and that CI runs.
On Wed, Mar 03, 2021 at 02:37:51PM -0500, Tom Rini wrote: > On Wed, Feb 03, 2021 at 05:44:06AM -0700, Simon Glass wrote: > > > At present U-Boot has two broad sets of tests in the C code: driver model > > tests which do a lot of pre-/post-init and command tests which do not. > > > > This separation makes it slightly harder to write a test, since there are > > two different test-state structures and different rules for running the > > two different test types. At present these rules are determined by where > > the test is (actually its prefix). > > > > All unit tests can be run from the command line with the 'ut' command. > > Since SPL does not have commands, it currently calls the test runner > > directly and offers no control of which tests are run. > > > > This seems like a good time to refactor the tests into a unified test > > runner, allowing U-Boot proper and SPL to use the same path, perhaps with > > some different conditions along the way. > > > > This series sets up a unified runner called ut_run_list(), which runs a > > set of tests from a linker_list. Driver model tests are distinguished by > > a new UT_TESTF_DM flag so that the necessary init and cleanup can still > > be done. > > > > The runner is modified to support running SPL tests that are not solely > > for driver model. An example test for FIT loading is added as a > > demonstration. > > > > In addition, some documentation is added to explain how to write tests. > > > > This series is available at u-boot-dm/test-working > > > > Changes in v3: > > - Add new patch to re-enable test_ofplatdata > > - Reword the SPL tests section for clarity > > - Use test_set_state() throughout test-main.c instead of direct assignment > > - Update the pytest collector as well > > > > Changes in v2: > > - Use correct rst format for 'Ad-hoc tests' section > > - Expand docs on how each type of test is marked > > - Put the docs in tests_sandbox since it is more related to sandbox > > - Put in a mention of tests_sandbox in the main testing docs > > - Add a note that SPL tests can in fact be run individualy > > - Document how to run all C tests with 'ut all' > > - Fix 'get list' typo > > - Fix conditions so non-DM SPL tests are actually run > > - Allow for prefix to be NULL, to match function comment > > - Add new patches to cover running an SPL test > > There was a small problem applying one or two of these patches that I > fixed up. But then when testing locally (Ubuntu 18.04, but virtualenv > setup and populated with test/py/requirements.txt) like 200 tests failed > to run for Sandbox. Please check this and that CI runs. Er, something has gone odd for me locally, please disregard.
On Wed, Mar 03, 2021 at 03:18:15PM -0500, Tom Rini wrote: > On Wed, Mar 03, 2021 at 02:37:51PM -0500, Tom Rini wrote: > > On Wed, Feb 03, 2021 at 05:44:06AM -0700, Simon Glass wrote: > > > > > At present U-Boot has two broad sets of tests in the C code: driver model > > > tests which do a lot of pre-/post-init and command tests which do not. > > > > > > This separation makes it slightly harder to write a test, since there are > > > two different test-state structures and different rules for running the > > > two different test types. At present these rules are determined by where > > > the test is (actually its prefix). > > > > > > All unit tests can be run from the command line with the 'ut' command. > > > Since SPL does not have commands, it currently calls the test runner > > > directly and offers no control of which tests are run. > > > > > > This seems like a good time to refactor the tests into a unified test > > > runner, allowing U-Boot proper and SPL to use the same path, perhaps with > > > some different conditions along the way. > > > > > > This series sets up a unified runner called ut_run_list(), which runs a > > > set of tests from a linker_list. Driver model tests are distinguished by > > > a new UT_TESTF_DM flag so that the necessary init and cleanup can still > > > be done. > > > > > > The runner is modified to support running SPL tests that are not solely > > > for driver model. An example test for FIT loading is added as a > > > demonstration. > > > > > > In addition, some documentation is added to explain how to write tests. > > > > > > This series is available at u-boot-dm/test-working > > > > > > Changes in v3: > > > - Add new patch to re-enable test_ofplatdata > > > - Reword the SPL tests section for clarity > > > - Use test_set_state() throughout test-main.c instead of direct assignment > > > - Update the pytest collector as well > > > > > > Changes in v2: > > > - Use correct rst format for 'Ad-hoc tests' section > > > - Expand docs on how each type of test is marked > > > - Put the docs in tests_sandbox since it is more related to sandbox > > > - Put in a mention of tests_sandbox in the main testing docs > > > - Add a note that SPL tests can in fact be run individualy > > > - Document how to run all C tests with 'ut all' > > > - Fix 'get list' typo > > > - Fix conditions so non-DM SPL tests are actually run > > > - Allow for prefix to be NULL, to match function comment > > > - Add new patches to cover running an SPL test > > > > There was a small problem applying one or two of these patches that I > > fixed up. But then when testing locally (Ubuntu 18.04, but virtualenv > > setup and populated with test/py/requirements.txt) like 200 tests failed > > to run for Sandbox. Please check this and that CI runs. > > Er, something has gone odd for me locally, please disregard. Two problems! One, this series IS breaking tests for me. Fresh venv, ~260 tests fail to run. Two, I didn't notice a fail-to-build in my build+pytest script, so the binary from this really really failed with the current pytest scripts :)
Hi Tom, On Wed, 3 Mar 2021 at 15:37, Tom Rini <trini@konsulko.com> wrote: > > On Wed, Mar 03, 2021 at 03:18:15PM -0500, Tom Rini wrote: > > On Wed, Mar 03, 2021 at 02:37:51PM -0500, Tom Rini wrote: > > > On Wed, Feb 03, 2021 at 05:44:06AM -0700, Simon Glass wrote: > > > > > > > At present U-Boot has two broad sets of tests in the C code: driver model > > > > tests which do a lot of pre-/post-init and command tests which do not. > > > > > > > > This separation makes it slightly harder to write a test, since there are > > > > two different test-state structures and different rules for running the > > > > two different test types. At present these rules are determined by where > > > > the test is (actually its prefix). > > > > > > > > All unit tests can be run from the command line with the 'ut' command. > > > > Since SPL does not have commands, it currently calls the test runner > > > > directly and offers no control of which tests are run. > > > > > > > > This seems like a good time to refactor the tests into a unified test > > > > runner, allowing U-Boot proper and SPL to use the same path, perhaps with > > > > some different conditions along the way. > > > > > > > > This series sets up a unified runner called ut_run_list(), which runs a > > > > set of tests from a linker_list. Driver model tests are distinguished by > > > > a new UT_TESTF_DM flag so that the necessary init and cleanup can still > > > > be done. > > > > > > > > The runner is modified to support running SPL tests that are not solely > > > > for driver model. An example test for FIT loading is added as a > > > > demonstration. > > > > > > > > In addition, some documentation is added to explain how to write tests. > > > > > > > > This series is available at u-boot-dm/test-working > > > > > > > > Changes in v3: > > > > - Add new patch to re-enable test_ofplatdata > > > > - Reword the SPL tests section for clarity > > > > - Use test_set_state() throughout test-main.c instead of direct assignment > > > > - Update the pytest collector as well > > > > > > > > Changes in v2: > > > > - Use correct rst format for 'Ad-hoc tests' section > > > > - Expand docs on how each type of test is marked > > > > - Put the docs in tests_sandbox since it is more related to sandbox > > > > - Put in a mention of tests_sandbox in the main testing docs > > > > - Add a note that SPL tests can in fact be run individualy > > > > - Document how to run all C tests with 'ut all' > > > > - Fix 'get list' typo > > > > - Fix conditions so non-DM SPL tests are actually run > > > > - Allow for prefix to be NULL, to match function comment > > > > - Add new patches to cover running an SPL test > > > > > > There was a small problem applying one or two of these patches that I > > > fixed up. But then when testing locally (Ubuntu 18.04, but virtualenv > > > setup and populated with test/py/requirements.txt) like 200 tests failed > > > to run for Sandbox. Please check this and that CI runs. > > > > Er, something has gone odd for me locally, please disregard. > > Two problems! One, this series IS breaking tests for me. Fresh venv, > ~260 tests fail to run. Two, I didn't notice a fail-to-build in my > build+pytest script, so the binary from this really really failed with > the current pytest scripts :) Sorry about that...it needs a fix which I had already brought into dm/master when I was intending for this series to go to the current release, but have not sent a pull request for. They are in dm/master now too and I'll send a pull request for master (that you should also apply to -next). http://patchwork.ozlabs.org/project/uboot/patch/20210204042944.2636812-1-sjg@chromium.org/ Basically there is a crash in sandbox_spl in a rather obscure case with of-platdata. Finding that was one of the fruits of adding the SPL tests. Regards, Simon