| Message ID | 20191106102906.28118-1-po-hsu.lin@canonical.com |
|---|---|
| Headers | show |
| Series | Handle the skip return code in kernel_selftests on Bionic | expand |
On 06.11.19 11:29, Po-Hsu Lin wrote: > BugLink: https://bugs.launchpad.net/bugs/1812352 > > == Justification == > In the ubuntu_kernel_selftests, the skipped test will uses return code 4 > (KSFT_SKIP). > > However the code to handle this non-zero return code was not implemented > in the kselftest framework. And this will generate some false-positive > results as those skipped tests were treated as a failure. > > For example the raw_skew test in timers on 4.15: > 1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons > not ok 1..7 selftests: raw_skew [FAIL] > > And the test_user_copy.sh test in user (failure could be found in KVM kernels): > user: module test_user_copy is not found [SKIP] > not ok 1..1 selftests: test_user_copy.sh [FAIL] > > Xenial is affected by this issue as well. Considering the fact that this > issue in Bionic affects more tests / kernel variants, and it's way > easier to fix this in Bionic, I decided to go for Bionic first. > > == Fix == > 771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers) > a3355440 (selftests: Fix lib.mk run_tests target shell script) > 42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable) > 3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define) > > All these can be cherry-picked into Bionic. > > The crucial patch is commit 3f4435b5, the other three patches are just > some minor fixes that pave the road for it to be cherry-picked. > > == Test == > Tested on a Bionic amd64 node, the result shows positive. > > A test summary before the patch: > END GOOD ubuntu_kernel_selftests.setup > END GOOD ubuntu_kernel_selftests.breakpoints > END GOOD ubuntu_kernel_selftests.cpu-hotplug > END GOOD ubuntu_kernel_selftests.efivarfs > END GOOD ubuntu_kernel_selftests.memfd > END GOOD ubuntu_kernel_selftests.memory-hotplug > END GOOD ubuntu_kernel_selftests.mount > END GOOD ubuntu_kernel_selftests.net > END GOOD ubuntu_kernel_selftests.ptrace > END GOOD ubuntu_kernel_selftests.seccomp > END ERROR ubuntu_kernel_selftests.timers > END GOOD ubuntu_kernel_selftests.powerpc > END GOOD ubuntu_kernel_selftests.user > END GOOD ubuntu_kernel_selftests.ftrace > > A test summary after the patch: > END GOOD ubuntu_kernel_selftests.setup > END GOOD ubuntu_kernel_selftests.breakpoints > END GOOD ubuntu_kernel_selftests.cpu-hotplug > END GOOD ubuntu_kernel_selftests.efivarfs > END GOOD ubuntu_kernel_selftests.memfd > END GOOD ubuntu_kernel_selftests.memory-hotplug > END GOOD ubuntu_kernel_selftests.mount > END GOOD ubuntu_kernel_selftests.net > END GOOD ubuntu_kernel_selftests.ptrace > END GOOD ubuntu_kernel_selftests.seccomp > END GOOD ubuntu_kernel_selftests.timers > END GOOD ubuntu_kernel_selftests.powerpc > END GOOD ubuntu_kernel_selftests.user > END GOOD ubuntu_kernel_selftests.ftrac > > No tests were skipped unintentionally. And the raw_skew test in timers > is now marked as [SKIPPED] and no longer causing failure. > > A complete test report could be found in the bug report attachment. > > == Regression Potential == > Low, this patch set just added support to handle the skip return code > and changes are limited to the kselftest framework itself, no actual > impact to normal users. > > Mathieu Desnoyers (1): > selftests: Fix lib.mk run_tests target shell script > > Shuah Khan (1): > selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers > > Shuah Khan (Samsung OSG) (2): > selftests: lib.mk: cleanup RUN_TESTS define and make it readable > selftests: lib.mk: add SKIP handling to RUN_TESTS define > > tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------ > 1 file changed, 20 insertions(+), 6 deletions(-) > Acked-by: Stefan Bader <stefan.bader@canonical.com>
On Wed, Nov 06, 2019 at 06:29:02PM +0800, Po-Hsu Lin wrote: > BugLink: https://bugs.launchpad.net/bugs/1812352 > > == Justification == > In the ubuntu_kernel_selftests, the skipped test will uses return code 4 > (KSFT_SKIP). > > However the code to handle this non-zero return code was not implemented > in the kselftest framework. And this will generate some false-positive > results as those skipped tests were treated as a failure. > > For example the raw_skew test in timers on 4.15: > 1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons > not ok 1..7 selftests: raw_skew [FAIL] > > And the test_user_copy.sh test in user (failure could be found in KVM kernels): > user: module test_user_copy is not found [SKIP] > not ok 1..1 selftests: test_user_copy.sh [FAIL] > > Xenial is affected by this issue as well. Considering the fact that this > issue in Bionic affects more tests / kernel variants, and it's way > easier to fix this in Bionic, I decided to go for Bionic first. > > == Fix == > 771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers) > a3355440 (selftests: Fix lib.mk run_tests target shell script) > 42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable) > 3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define) > > All these can be cherry-picked into Bionic. > > The crucial patch is commit 3f4435b5, the other three patches are just > some minor fixes that pave the road for it to be cherry-picked. > > == Test == > Tested on a Bionic amd64 node, the result shows positive. > > A test summary before the patch: > END GOOD ubuntu_kernel_selftests.setup > END GOOD ubuntu_kernel_selftests.breakpoints > END GOOD ubuntu_kernel_selftests.cpu-hotplug > END GOOD ubuntu_kernel_selftests.efivarfs > END GOOD ubuntu_kernel_selftests.memfd > END GOOD ubuntu_kernel_selftests.memory-hotplug > END GOOD ubuntu_kernel_selftests.mount > END GOOD ubuntu_kernel_selftests.net > END GOOD ubuntu_kernel_selftests.ptrace > END GOOD ubuntu_kernel_selftests.seccomp > END ERROR ubuntu_kernel_selftests.timers > END GOOD ubuntu_kernel_selftests.powerpc > END GOOD ubuntu_kernel_selftests.user > END GOOD ubuntu_kernel_selftests.ftrace > > A test summary after the patch: > END GOOD ubuntu_kernel_selftests.setup > END GOOD ubuntu_kernel_selftests.breakpoints > END GOOD ubuntu_kernel_selftests.cpu-hotplug > END GOOD ubuntu_kernel_selftests.efivarfs > END GOOD ubuntu_kernel_selftests.memfd > END GOOD ubuntu_kernel_selftests.memory-hotplug > END GOOD ubuntu_kernel_selftests.mount > END GOOD ubuntu_kernel_selftests.net > END GOOD ubuntu_kernel_selftests.ptrace > END GOOD ubuntu_kernel_selftests.seccomp > END GOOD ubuntu_kernel_selftests.timers > END GOOD ubuntu_kernel_selftests.powerpc > END GOOD ubuntu_kernel_selftests.user > END GOOD ubuntu_kernel_selftests.ftrac > > No tests were skipped unintentionally. And the raw_skew test in timers > is now marked as [SKIPPED] and no longer causing failure. > > A complete test report could be found in the bug report attachment. > > == Regression Potential == > Low, this patch set just added support to handle the skip return code > and changes are limited to the kselftest framework itself, no actual > impact to normal users. > > Mathieu Desnoyers (1): > selftests: Fix lib.mk run_tests target shell script > > Shuah Khan (1): > selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers > > Shuah Khan (Samsung OSG) (2): > selftests: lib.mk: cleanup RUN_TESTS define and make it readable > selftests: lib.mk: add SKIP handling to RUN_TESTS define > > tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------ > 1 file changed, 20 insertions(+), 6 deletions(-) Makes sense to me and low-impact, considering that it's only affecting the kselftest framework, therefore: Acked-by: Andrea Righi <andrea.righi@canonical.com>
On 2019-11-06 18:29:02 , Po-Hsu Lin wrote: > BugLink: https://bugs.launchpad.net/bugs/1812352 > > == Justification == > In the ubuntu_kernel_selftests, the skipped test will uses return code 4 > (KSFT_SKIP). > > However the code to handle this non-zero return code was not implemented > in the kselftest framework. And this will generate some false-positive > results as those skipped tests were treated as a failure. > > For example the raw_skew test in timers on 4.15: > 1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons > not ok 1..7 selftests: raw_skew [FAIL] > > And the test_user_copy.sh test in user (failure could be found in KVM kernels): > user: module test_user_copy is not found [SKIP] > not ok 1..1 selftests: test_user_copy.sh [FAIL] > > Xenial is affected by this issue as well. Considering the fact that this > issue in Bionic affects more tests / kernel variants, and it's way > easier to fix this in Bionic, I decided to go for Bionic first. > > == Fix == > 771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers) > a3355440 (selftests: Fix lib.mk run_tests target shell script) > 42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable) > 3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define) > > All these can be cherry-picked into Bionic. > > The crucial patch is commit 3f4435b5, the other three patches are just > some minor fixes that pave the road for it to be cherry-picked. > > == Test == > Tested on a Bionic amd64 node, the result shows positive. > > A test summary before the patch: > END GOOD ubuntu_kernel_selftests.setup > END GOOD ubuntu_kernel_selftests.breakpoints > END GOOD ubuntu_kernel_selftests.cpu-hotplug > END GOOD ubuntu_kernel_selftests.efivarfs > END GOOD ubuntu_kernel_selftests.memfd > END GOOD ubuntu_kernel_selftests.memory-hotplug > END GOOD ubuntu_kernel_selftests.mount > END GOOD ubuntu_kernel_selftests.net > END GOOD ubuntu_kernel_selftests.ptrace > END GOOD ubuntu_kernel_selftests.seccomp > END ERROR ubuntu_kernel_selftests.timers > END GOOD ubuntu_kernel_selftests.powerpc > END GOOD ubuntu_kernel_selftests.user > END GOOD ubuntu_kernel_selftests.ftrace > > A test summary after the patch: > END GOOD ubuntu_kernel_selftests.setup > END GOOD ubuntu_kernel_selftests.breakpoints > END GOOD ubuntu_kernel_selftests.cpu-hotplug > END GOOD ubuntu_kernel_selftests.efivarfs > END GOOD ubuntu_kernel_selftests.memfd > END GOOD ubuntu_kernel_selftests.memory-hotplug > END GOOD ubuntu_kernel_selftests.mount > END GOOD ubuntu_kernel_selftests.net > END GOOD ubuntu_kernel_selftests.ptrace > END GOOD ubuntu_kernel_selftests.seccomp > END GOOD ubuntu_kernel_selftests.timers > END GOOD ubuntu_kernel_selftests.powerpc > END GOOD ubuntu_kernel_selftests.user > END GOOD ubuntu_kernel_selftests.ftrac > > No tests were skipped unintentionally. And the raw_skew test in timers > is now marked as [SKIPPED] and no longer causing failure. > > A complete test report could be found in the bug report attachment. > > == Regression Potential == > Low, this patch set just added support to handle the skip return code > and changes are limited to the kselftest framework itself, no actual > impact to normal users. > > Mathieu Desnoyers (1): > selftests: Fix lib.mk run_tests target shell script > > Shuah Khan (1): > selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers > > Shuah Khan (Samsung OSG) (2): > selftests: lib.mk: cleanup RUN_TESTS define and make it readable > selftests: lib.mk: add SKIP handling to RUN_TESTS define > > tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------ > 1 file changed, 20 insertions(+), 6 deletions(-) > > -- > 2.7.4 > > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team
BugLink: https://bugs.launchpad.net/bugs/1812352 == Justification == In the ubuntu_kernel_selftests, the skipped test will uses return code 4 (KSFT_SKIP). However the code to handle this non-zero return code was not implemented in the kselftest framework. And this will generate some false-positive results as those skipped tests were treated as a failure. For example the raw_skew test in timers on 4.15: 1..0 # Skipped: The clock was adjusted externally. Shutdown NTPd or other time sync daemons not ok 1..7 selftests: raw_skew [FAIL] And the test_user_copy.sh test in user (failure could be found in KVM kernels): user: module test_user_copy is not found [SKIP] not ok 1..1 selftests: test_user_copy.sh [FAIL] Xenial is affected by this issue as well. Considering the fact that this issue in Bionic affects more tests / kernel variants, and it's way easier to fix this in Bionic, I decided to go for Bionic first. == Fix == 771cbc3b (selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers) a3355440 (selftests: Fix lib.mk run_tests target shell script) 42b44c34 (selftests: lib.mk: cleanup RUN_TESTS define and make it readable) 3f4435b5 (selftests: lib.mk: add SKIP handling to RUN_TESTS define) All these can be cherry-picked into Bionic. The crucial patch is commit 3f4435b5, the other three patches are just some minor fixes that pave the road for it to be cherry-picked. == Test == Tested on a Bionic amd64 node, the result shows positive. A test summary before the patch: END GOOD ubuntu_kernel_selftests.setup END GOOD ubuntu_kernel_selftests.breakpoints END GOOD ubuntu_kernel_selftests.cpu-hotplug END GOOD ubuntu_kernel_selftests.efivarfs END GOOD ubuntu_kernel_selftests.memfd END GOOD ubuntu_kernel_selftests.memory-hotplug END GOOD ubuntu_kernel_selftests.mount END GOOD ubuntu_kernel_selftests.net END GOOD ubuntu_kernel_selftests.ptrace END GOOD ubuntu_kernel_selftests.seccomp END ERROR ubuntu_kernel_selftests.timers END GOOD ubuntu_kernel_selftests.powerpc END GOOD ubuntu_kernel_selftests.user END GOOD ubuntu_kernel_selftests.ftrace A test summary after the patch: END GOOD ubuntu_kernel_selftests.setup END GOOD ubuntu_kernel_selftests.breakpoints END GOOD ubuntu_kernel_selftests.cpu-hotplug END GOOD ubuntu_kernel_selftests.efivarfs END GOOD ubuntu_kernel_selftests.memfd END GOOD ubuntu_kernel_selftests.memory-hotplug END GOOD ubuntu_kernel_selftests.mount END GOOD ubuntu_kernel_selftests.net END GOOD ubuntu_kernel_selftests.ptrace END GOOD ubuntu_kernel_selftests.seccomp END GOOD ubuntu_kernel_selftests.timers END GOOD ubuntu_kernel_selftests.powerpc END GOOD ubuntu_kernel_selftests.user END GOOD ubuntu_kernel_selftests.ftrac No tests were skipped unintentionally. And the raw_skew test in timers is now marked as [SKIPPED] and no longer causing failure. A complete test report could be found in the bug report attachment. == Regression Potential == Low, this patch set just added support to handle the skip return code and changes are limited to the kselftest framework itself, no actual impact to normal users. Mathieu Desnoyers (1): selftests: Fix lib.mk run_tests target shell script Shuah Khan (1): selftests: lib.mk set KSFT_TAP_LEVEL to prevent nested TAP headers Shuah Khan (Samsung OSG) (2): selftests: lib.mk: cleanup RUN_TESTS define and make it readable selftests: lib.mk: add SKIP handling to RUN_TESTS define tools/testing/selftests/lib.mk | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)