| Message ID | 20200330162411.9678-1-romain.naour@smile.fr |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | [1/2] support/testing: add openssh runtime test | expand |
On Mon, 30 Mar 2020 18:24:10 +0200 Romain Naour <romain.naour@smile.fr> wrote: > +class TestOpenSshGlibc(TestOpensshBase): > + config = \ > + TestOpensshBase.opensshconfig + \ > + """ > + BR2_x86_64=y > + BR2_x86_core2=y Why use an x86-64 system? It forces you to build your own kernel? For this glibc configuration, you could build your own toolchain, but probably stick to using a prebuilt kernel. > + BR2_TOOLCHAIN_BUILDROOT_GLIBC=y What about using the ARM ARM toolchain, and boot using the pre-built vexpress kernel ? This will significantly speed-up the build of the system for this runtime test. Thanks! Thomas
Hi Thomas, Le 31/03/2020 à 23:32, Thomas Petazzoni a écrit : > On Mon, 30 Mar 2020 18:24:10 +0200 > Romain Naour <romain.naour@smile.fr> wrote: > >> +class TestOpenSshGlibc(TestOpensshBase): >> + config = \ >> + TestOpensshBase.opensshconfig + \ >> + """ >> + BR2_x86_64=y >> + BR2_x86_core2=y > > Why use an x86-64 system? It forces you to build your own kernel? For > this glibc configuration, you could build your own toolchain, but > probably stick to using a prebuilt kernel. I reused the configuration for Docker, that's why I used x86_64. But the testsuite basically test only with uClibc-ng with an old toolchain by default. It would be interesting to allow to select a custom toolchain. Also, I noticed that the build continue even if the defconfig provided contain an option that is dropped by make olddefconfig. For example, mesa3d: https://git.buildroot.net/buildroot/commit/?id=faec5c583ec59e968fdb547aea93ab6ed001eb45 The symbol BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_SWRAST was missing since only one swrast driver can be enabled. The test should stop is one of the provided symbol is missing (like autobuild-run do). > >> + BR2_TOOLCHAIN_BUILDROOT_GLIBC=y > > What about using the ARM ARM toolchain, and boot using the pre-built > vexpress kernel ? This will significantly speed-up the build of the > system for this runtime test. Because I made this test to trigger the openssh issue with glibc 2.31 (local patch applied to do the version bump). That's why It's important for the testsuite to allow testing with several toolchain (I believe). Best regards, Romain > > Thanks! > > Thomas >
Hello, On Thu, 2 Apr 2020 13:48:15 +0200 Romain Naour <romain.naour@smile.fr> wrote: > > Why use an x86-64 system? It forces you to build your own kernel? For > > this glibc configuration, you could build your own toolchain, but > > probably stick to using a prebuilt kernel. > > I reused the configuration for Docker, that's why I used x86_64. OK, but then you have to build your own kernel, which is no really needed. > But the testsuite basically test only with uClibc-ng with an old toolchain by > default. It would be interesting to allow to select a custom toolchain. You can already select whatever toolchain you want, since you provide for each test case the toolchain. However, we could certainly update the default toolchain used in infra.basetest.BASIC_TOOLCHAIN_CONFIG > Also, I noticed that the build continue even if the defconfig provided contain > an option that is dropped by make olddefconfig. > > For example, mesa3d: > https://git.buildroot.net/buildroot/commit/?id=faec5c583ec59e968fdb547aea93ab6ed001eb45 > > The symbol BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_SWRAST was missing since only one > swrast driver can be enabled. > > The test should stop is one of the provided symbol is missing (like > autobuild-run do). Sure, that would make sense. > >> + BR2_TOOLCHAIN_BUILDROOT_GLIBC=y > > > > What about using the ARM ARM toolchain, and boot using the pre-built > > vexpress kernel ? This will significantly speed-up the build of the > > system for this runtime test. > > Because I made this test to trigger the openssh issue with glibc 2.31 (local > patch applied to do the version bump). Ah, OK, indeed. > That's why It's important for the testsuite to allow testing with several > toolchain (I believe). It already allows to do so: you did it! Best regards, Thomas
Hello Thomas, Le 02/04/2020 à 14:16, Thomas Petazzoni a écrit : > Hello, > > On Thu, 2 Apr 2020 13:48:15 +0200 > Romain Naour <romain.naour@smile.fr> wrote: > >>> Why use an x86-64 system? It forces you to build your own kernel? For >>> this glibc configuration, you could build your own toolchain, but >>> probably stick to using a prebuilt kernel. >> >> I reused the configuration for Docker, that's why I used x86_64. > > OK, but then you have to build your own kernel, which is no really > needed. Ok > >> But the testsuite basically test only with uClibc-ng with an old toolchain by >> default. It would be interesting to allow to select a custom toolchain. > > You can already select whatever toolchain you want, since you provide > for each test case the toolchain. I mean, I would be interesting to override or provide another toolchain. For example, a package test should be pass whatever the toolchain being used. For the moment most of the tests are done with BASIC_TOOLCHAIN_CONFIG. > > However, we could certainly update the default toolchain used in > infra.basetest.BASIC_TOOLCHAIN_CONFIG What about using br-arm-full-2020.02 ? > >> Also, I noticed that the build continue even if the defconfig provided contain >> an option that is dropped by make olddefconfig. >> >> For example, mesa3d: >> https://git.buildroot.net/buildroot/commit/?id=faec5c583ec59e968fdb547aea93ab6ed001eb45 >> >> The symbol BR2_PACKAGE_MESA3D_GALLIUM_DRIVER_SWRAST was missing since only one >> swrast driver can be enabled. >> >> The test should stop is one of the provided symbol is missing (like >> autobuild-run do). > > Sure, that would make sense. > >>>> + BR2_TOOLCHAIN_BUILDROOT_GLIBC=y >>> >>> What about using the ARM ARM toolchain, and boot using the pre-built >>> vexpress kernel ? This will significantly speed-up the build of the >>> system for this runtime test. >> >> Because I made this test to trigger the openssh issue with glibc 2.31 (local >> patch applied to do the version bump). > > Ah, OK, indeed. > >> That's why It's important for the testsuite to allow testing with several >> toolchain (I believe). > > It already allows to do so: you did it! See my explanation above. Best regards, Romain > > Best regards, > > Thomas >
On Thu, 2 Apr 2020 21:38:42 +0200 Romain Naour <romain.naour@smile.fr> wrote: > >> But the testsuite basically test only with uClibc-ng with an old toolchain by > >> default. It would be interesting to allow to select a custom toolchain. > > > > You can already select whatever toolchain you want, since you provide > > for each test case the toolchain. > > I mean, I would be interesting to override or provide another toolchain. For > example, a package test should be pass whatever the toolchain being used. How would you handle that? The point of runtime tests is that they should be reproducible so that we can track when they started failing. We also can't run all tests on all toolchains/architectures, that's not really realistic from a CPU time point of view, and from a test maintenance point of view. > > However, we could certainly update the default toolchain used in > > infra.basetest.BASIC_TOOLCHAIN_CONFIG > > What about using br-arm-full-2020.02 ? Sure, would be good. Could you test this, do a Gitlab CI run, and see how it goes? Thomas
Hi Thomas, Le 02/04/2020 à 21:41, Thomas Petazzoni a écrit : > On Thu, 2 Apr 2020 21:38:42 +0200 > Romain Naour <romain.naour@smile.fr> wrote: > >>>> But the testsuite basically test only with uClibc-ng with an old toolchain by >>>> default. It would be interesting to allow to select a custom toolchain. >>> >>> You can already select whatever toolchain you want, since you provide >>> for each test case the toolchain. >> >> I mean, I would be interesting to override or provide another toolchain. For >> example, a package test should be pass whatever the toolchain being used. > > How would you handle that? The point of runtime tests is that they > should be reproducible so that we can track when they started failing. > We also can't run all tests on all toolchains/architectures, that's not > really realistic from a CPU time point of view, and from a test > maintenance point of view. My point is that we don't test with different gcc/binutils version and different libc implementation. But we have to use the same architecture since the Qemu command line is specific to the emulated architecture. If a test would be available for several architecture it would require to update the qemu command line (and the kernel configuration too). At least testing with several gcc and libc implementation would be good enough? > >>> However, we could certainly update the default toolchain used in >>> infra.basetest.BASIC_TOOLCHAIN_CONFIG >> >> What about using br-arm-full-2020.02 ? > > Sure, would be good. Could you test this, do a Gitlab CI run, and see how it goes? Ok. Best regards, Romain > > Thomas >
On Thu, 2 Apr 2020 22:04:00 +0200 Romain Naour <romain.naour@smile.fr> wrote: > My point is that we don't test with different gcc/binutils version and different > libc implementation. But we have to use the same architecture since the Qemu > command line is specific to the emulated architecture. If a test would be > available for several architecture it would require to update the qemu command > line (and the kernel configuration too). > > At least testing with several gcc and libc implementation would be good enough? But would you do that for all tests ? If you take our 172 existing tests, then run all of them on 3 different libcs, 3 different gcc versions, you suddenly have 1548 test cases. And that's just for *one* architecture. Thomas
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 416ed3c0ca..0fae2e4e53 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -430,6 +430,8 @@ tests.package.test_lxc.TestLxc: { extends: .runtime_test } tests.package.test_lzlib.TestLuaLzlib: { extends: .runtime_test } tests.package.test_netdata.TestNetdata: { extends: .runtime_test } tests.package.test_openjdk.TestOpenJdk: { extends: .runtime_test } +tests.package.test_openssh.TestOpenSshGlibc: { extends: .runtime_test } +tests.package.test_openssh.TestOpenSshuClibc: { extends: .runtime_test } tests.package.test_opkg.TestOpkg: { extends: .runtime_test } tests.package.test_perl.TestPerl: { extends: .runtime_test } tests.package.test_perl_class_load.TestPerlClassLoad: { extends: .runtime_test } diff --git a/DEVELOPERS b/DEVELOPERS index f1bf5c263c..009063347b 100644 --- a/DEVELOPERS +++ b/DEVELOPERS @@ -2214,6 +2214,7 @@ F: package/waffle/ F: package/xenomai/ F: package/zziplib/ F: support/testing/tests/package/test_glxinfo.py +F: support/testing/tests/package/test_openssh.py F: toolchain/ N: Roman Gorbenkov <roman.gorbenkov@ens2m.org> diff --git a/support/testing/tests/package/test_openssh.py b/support/testing/tests/package/test_openssh.py new file mode 100644 index 0000000000..6eb58d404d --- /dev/null +++ b/support/testing/tests/package/test_openssh.py @@ -0,0 +1,78 @@ +import os + +import infra.basetest + + +class TestOpensshBase(infra.basetest.BRTest): + passwd = "testpwd" + opensshconfig = \ + """ + BR2_TARGET_GENERIC_ROOT_PASSWD="{}" + BR2_SYSTEM_DHCP="eth0" + BR2_PACKAGE_OPENSSH=y + BR2_PACKAGE_SSHPASS=y + BR2_ROOTFS_POST_BUILD_SCRIPT="{}" + # BR2_TARGET_ROOTFS_TAR is not set + """.format( + passwd, + infra.filepath("tests/package/test_openssh/post-build.sh")) + + def openssh_test(self): + self.emulator.login(self.passwd) + cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:22" + _, exit_code = self.emulator.run(cmd) + self.assertEqual(exit_code, 0) + + cmd = "sshpass -p {} ssh -oStrictHostKeyChecking=no localhost /bin/true".format(self.passwd) + _, exit_code = self.emulator.run(cmd) + self.assertEqual(exit_code, 0) + + +class TestOpenSshuClibc(TestOpensshBase): + config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ + TestOpensshBase.opensshconfig + \ + """ + BR2_TARGET_ROOTFS_CPIO=y + """ + + def test_run(self): + img = os.path.join(self.builddir, "images", "rootfs.cpio") + self.emulator.boot(arch="armv5", + kernel="builtin", + options=["-initrd", img, + "-net", "nic", + "-net", "user"]) + self.openssh_test() + + +class TestOpenSshGlibc(TestOpensshBase): + config = \ + TestOpensshBase.opensshconfig + \ + """ + BR2_x86_64=y + BR2_x86_core2=y + BR2_TOOLCHAIN_BUILDROOT_GLIBC=y + BR2_KERNEL_HEADERS_4_19=y + BR2_TOOLCHAIN_BUILDROOT_CXX=y + BR2_LINUX_KERNEL=y + BR2_LINUX_KERNEL_CUSTOM_VERSION=y + BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.19" + BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y + BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux.config" + BR2_PACKAGE_RNG_TOOLS=y + BR2_TARGET_ROOTFS_EXT2=y + """ + + def test_run(self): + kernel = os.path.join(self.builddir, "images", "bzImage") + rootfs = os.path.join(self.builddir, "images", "rootfs.ext2") + self.emulator.boot(arch="x86_64", + kernel=kernel, + kernel_cmdline=["root=/dev/vda", "console=ttyS0"], + options=["-cpu", "core2duo", + "-m", "512M", + "-device", "virtio-rng-pci", + "-drive", "file={},format=raw,if=virtio".format(rootfs), + "-net", "nic,model=virtio", + "-net", "user"]) + self.openssh_test() diff --git a/support/testing/tests/package/test_openssh/post-build.sh b/support/testing/tests/package/test_openssh/post-build.sh new file mode 100755 index 0000000000..4c876ce554 --- /dev/null +++ b/support/testing/tests/package/test_openssh/post-build.sh @@ -0,0 +1,4 @@ +#!/usr/bin/env bash + +sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/g' ${TARGET_DIR}/etc/ssh/sshd_config +sed -i 's/#PasswordAuthentication yes/PasswordAuthentication yes/g' ${TARGET_DIR}/etc/ssh/sshd_config
This new runtime test is based on test_dropbear.py. The only required change is to use "-oStrictHostKeyChecking=no" instead of "-y" to accept the new key. Since the base test infra only provide a uClibc-ng toolchain, add a second test using a glibc based internal toolchain. For example, this allow to trigger the openssh 8.1p bug with glibc 2.31 [1]. [1] https://bugs.archlinux.org/task/65386 Signed-off-by: Romain Naour <romain.naour@smile.fr> --- .gitlab-ci.yml | 2 + DEVELOPERS | 1 + support/testing/tests/package/test_openssh.py | 78 +++++++++++++++++++ .../tests/package/test_openssh/post-build.sh | 4 + 4 files changed, 85 insertions(+) create mode 100644 support/testing/tests/package/test_openssh.py create mode 100755 support/testing/tests/package/test_openssh/post-build.sh