Message ID | 20221021091531.2989489-1-oguz.ozhan@mind.be |
---|---|
State | Changes Requested |
Headers | show |
Series | [1/3] support/testing: replace nose2 with pytest - Dockerfile | expand |
Oguz, All, On 2022-10-21 11:15 +0200, Oguz Ozhan spake thusly: > From: Oguz Ozhan <oguz.ozhan@mind.com> > > - From web page of nose2: > (https://docs.nose2.io/en/latest/) > nose2 vs pytest: > - pytest is an excellent test framework and we encourage users to consider it for new projects. > - It has a bigger team of maintainers and a larger community of users. This clearly states "for new projects", but we are not new! ;-) Why do we need to switch to pytest, and what does that bring us? (see below too) > - pytest is more robust and has more ability compared to nose2 You provide robustness and more abilities to advocate for switching to pytest, however, you do not explain what is currently broken that would be fixed by the pytest robustness, or what new features from pytest we could leverage, and how. Note that I am not opposed to switching, but we need a good rationale for switching. This commit should also come after we actually did the switch, i.e. we should first convert support/testing/ over to pytest, then modify the dockerfile accordingly, which allows to write a better coomit log. Also, please organise your commit logs as (first line is the title): support/docker: install pytest instead of nose2 Now that the tuntime test infra has switched to using pytest instead of nose2, install the appropriate packages in the Dockerfile for our reference build environment. Note that pytest-parallel is not available i Debian Bullseye, so we install it (and its depenencies) from Pypi instead. Signed-off-by: You See more below... > Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be> > --- > support/docker/Dockerfile | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile > index f54c31b54a..fd1527b1f9 100644 > --- a/support/docker/Dockerfile > +++ b/support/docker/Dockerfile > @@ -38,8 +38,8 @@ RUN apt-get install -y --no-install-recommends \ > mercurial \ > openssh-server \ > python3 \ > + python3-pip \ > python3-flake8 \ > - python3-nose2 \ > python3-pexpect \ > python3-pytest \ > qemu-system-arm \ > @@ -53,6 +53,9 @@ RUN apt-get install -y --no-install-recommends \ > apt-get -y autoremove && \ > apt-get -y clean > > +# To be able to run tests in parallel > +RUN pip install pytest-parallel Please, pin the version so that the image is reproducible. Also, if pytest-parallel has dependencies, explicitly install those at pinned versions too. Also note in the commit log that pytest-parallel is not available in bullseye, so we need to get it from Pypi instead (as in my suggestion). Regards, Yann E. MORIN. > # To be able to generate a toolchain with locales, enable one UTF-8 locale > RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \ > /usr/sbin/locale-gen > -- > 2.34.1 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot
Hello, I am not against to switch to pytest. But I didn't understand clearly yet the use case that motivated this series, in which nose2 is not good and pytest is better. Could you elaborate? Also, there are issues to be addressed before merging this series: - the detection of a failed test on GitLab CI is broken, even tests failing return exit code 0, so GitLab infra mark them as PASS, see [1]; - run-tests -s is broken (does not show the build and runtime console); - support to ask a single runtime test to run on GitLab CI is broken due to a regexp that needs to be adapted; - the manual needs to be updated. There are also some changes in behavior that would be nice to be highlighted: - someone willing to find a test in the output of run-tests -l can do: nose2: run-tests -l 2>&1 | grep pattern pytest: run-tests -l | grep pattern - total number of tests: nose2: run-tests -l prints the number of tests at the end "Ran 481 tests" pytest: run-tests -l does not - the name of tests change (list) nose2: test_run (tests.toolchain.test_external_bootlin.\ TestExternalToolchainBootlinPowerpc64lepower8GlibcBleedingEdge) ... ok pytest: support/testing/tests/toolchain/test_external_bootlin.py::\ TestExternalToolchainBootlinPowerpc64lepower8GlibcBleedingEdge::test_run - the name of tests change (local run) nose2: support/testing/run-tests tests.init.test_busybox \ tests.package.test_php_lua.TestPhpLuaLuajit pytest: support/testing/run-tests support/testing/tests/init/test_busybox.py \ support/testing/tests/package/test_php_lua.py::TestPhpLuaLua - the name of tests change (gitlab test suite) nose2: git push gitlab HEAD:foo-tests.init.test_busybox see [2] pytest: git push gitlab HEAD:foo-support/testing/tests/init/test_busybox.py see [3] - the name of tests change (gitlab test case) nose2: git push gitlab HEAD:foo-tests.package.test_php_lua.TestPhpLuaLuajit see [4] pytest: git push gitlab \ HEAD:foo-support/testing/tests/package/test_php_lua.py--TestPhpLuaLuajit see [5] NOTICE: the use of '--' is just the first solution that came to mind. - the name of tests are longer, so in the GitLab CI the display is a bit worse: nose2: tests.init.test_busybox.TestInitSystemBusyboxRo tests.init.test_busybox.TestInitSystemBusybox... tests.init.test_busybox.TestInitSystemBusybox... tests.init.test_busybox.TestInitSystemBusybox... tests.package.test_php_lua.TestPhpLuaLua tests.package.test_php_lua.TestPhpLuaLuajit pytest: support/testing/tests/init/test_busybox.py::Tes... support/testing/tests/init/test_busybox.py::Tes... support/testing/tests/init/test_busybox.py::Tes... support/testing/tests/init/test_busybox.py::Tes... support/testing/tests/package/test_php_lua.p... support/testing/tests/package/test_php_lua.p... - bash completion for the name of a test suite nose2: no completion support/testing/run-tests tests.init.test_busybox pytest: since it uses the path as the begin of test name one can type 'support/testing/run-tests support/testing' and use bash completion for test files names: support/testing/run-tests support/testing/tests/init/test_busybox.py - cache of results nose2: no cache pytest: it creates a .pytest_cache/ with a cache of the results, so we could later expose another parameter in run-tests that passes '-lf' to pytest, which will re-run only the tests that failed on last run - verbosity when collecting tests: nose2: no output pytest: prints a summary 'collected 1 item' 'pytest-parallel: 1 worker (process), 1 test per worker (thread)' - verbosity while tests are running: nose2: show states for each test (Starting, Building) pytest: shows nothing before each test ends - verbosity when tests fail: nose2: verbose pytest: more verbose than nose2 - verbosity when tests pass: nose2: show states for each test (Starting, Building) pytest: show only a '.' for each test [1] https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/3248620433 [2] https://gitlab.com/RicardoMartincoski/buildroot/-/pipelines/680878149 [3] https://gitlab.com/RicardoMartincoski/buildroot/-/pipelines/680879324 [4] https://gitlab.com/RicardoMartincoski/buildroot/-/pipelines/680879308 [5] https://gitlab.com/RicardoMartincoski/buildroot/-/pipelines/680878335 Regards, Ricardo
diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile index f54c31b54a..fd1527b1f9 100644 --- a/support/docker/Dockerfile +++ b/support/docker/Dockerfile @@ -38,8 +38,8 @@ RUN apt-get install -y --no-install-recommends \ mercurial \ openssh-server \ python3 \ + python3-pip \ python3-flake8 \ - python3-nose2 \ python3-pexpect \ python3-pytest \ qemu-system-arm \ @@ -53,6 +53,9 @@ RUN apt-get install -y --no-install-recommends \ apt-get -y autoremove && \ apt-get -y clean +# To be able to run tests in parallel +RUN pip install pytest-parallel + # To be able to generate a toolchain with locales, enable one UTF-8 locale RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \ /usr/sbin/locale-gen