| Message ID | 20190307153222.29255-1-xnox@ubuntu.com |
|---|---|
| Headers | show |
| Series | autopkgtest skippable fixes | expand |
--- debian/tests/rebuild | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 debian/tests/rebuild diff --git a/debian/tests/rebuild b/debian/tests/rebuild old mode 100644 new mode 100755
On Thu, Mar 07, 2019 at 03:32:17PM +0000, Dimitri John Ledkov wrote: > This is a set of patches to unbreak autopkgtests, as designed and > agreed in Malta kernel room mini-round-table between xnox, apw, > sforshee, bjf. > > The general notio is that we don't skip tests enough, when they are > over-triggered, and thus we keep on attempting to run them, when we > should be bailing out. I think this looks sane. There are a few details that I'm not informed enough about to provide useful review, so I'd still like apw to have a look. One thing, we do require your signed-off-by for kernel patches. I presume you are okay with us adding that when we apply the patches? Aside from that: Acked-by: Seth Forshee <seth.forshee@canonical.com> Thanks, Seth > > It is hard for me to build a full abi bump kernel, with a matching > bumped meta, and signed variants, so I monkey patched these tests into > various trees, and tried to run them with appropriate envrionment > variable as seen in britney results. > > Here are new things: > > $ cd linux-azure-4.18.0/ && ./debian/tests/ubuntu-regression-suite > ubuntu-regression-suite is pointless, if one cannot boot the kernel > $ echo $? > 77 > > $ cd linux-5.0.0/ && ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/ubuntu-regression-suite > ubuntu-regression-suite is not requested, as there is no linux-meta trigger > $ echo $? > 77 > > I also validated that the current expected tests do run correctly, e.g.: > > $ ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/rebuild > dpkg-buildpackage: info: source package linux > dpkg-buildpackage: info: source version 5.0.0-7.8 > dpkg-buildpackage: info: source distribution disco > ... > (build attempted) > > $ ADT_TEST_TRIGGERS=linux-meta/5.0.0.7.8 ./debian/tests/ubuntu-regression-suite > Source Package Version: 5.0.0-7.8 > Running Kernel Version: 4.19.0-13.14 > ERROR: running version does not match source package > ... > (well, i did fake the trigger, but yeah, the right kernel is pulled > in, in this case, and the test will be run) > > Please review these patches and apply them against all kernel > flavours, in all releases. > > Dimitri John Ledkov (5): > Set +x on rebuild testcase. > Skip rebuild test, for regression-suite deps. > Make ubuntu-regression-suite skippable on unbootable kernels. > make rebuild use skippable error codes when skipping. > Only run regression-suite, if requested to. > > debian/tests/control | 4 ++-- > debian/tests/rebuild | 4 ++-- > debian/tests/ubuntu-regression-suite | 26 ++++++++++++++++++++++++++ > 3 files changed, 30 insertions(+), 4 deletions(-) > mode change 100644 => 100755 debian/tests/rebuild > > -- > 2.20.1 > > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team
On 3/27/19 3:50 PM, seth.forshee at canonical.com (Seth Forshee) wrote: > On Thu, Mar 07, 2019 at 03:32:17PM +0000, Dimitri John Ledkov wrote: >> This is a set of patches to unbreak autopkgtests, as designed and >> agreed in Malta kernel room mini-round-table between xnox, apw, >> sforshee, bjf. >> >> The general notio is that we don't skip tests enough, when they are >> over-triggered, and thus we keep on attempting to run them, when we >> should be bailing out. > > I think this looks sane. There are a few details that I'm not informed > enough about to provide useful review, so I'd still like apw to have a > look. > For example, at the moment in disco gcc-7 & gcc-8 has been stuck for 6 days now in disco-proposed. They trigger linux tests and run both the rebuild test (good and passes) and ubuntu-regression-testsuite (shouldn't run and fails due to lack of matching meta trigger). With these patches applies, the ubuntu-regression-testsuite will be correctly skipped / not attempted to be executed, as for these triggers we only care about the rebuild test. Note this is currently blocking velocity of updating the toolchain in Disco. And there is _no_ correct syntax to override this in hints. As we do want to see the results of the rebuild test, and block toolchain migration if rebuild test fails. (It currently passes, but flagged up as false negative). This has costed my engineering time to type this email, and a release team member to investigate this and flag it up. Yet, it could have sailed through automatically with the patches I have proposed. http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#gcc-7 http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#gcc-8 > One thing, we do require your signed-off-by for kernel patches. I > presume you are okay with us adding that when we apply the patches? Yes please, and sorry for not including those. > Aside from that: > > Acked-by: Seth Forshee <seth.forshee at canonical.com> > > Thanks, > Seth > >> >> It is hard for me to build a full abi bump kernel, with a matching >> bumped meta, and signed variants, so I monkey patched these tests into >> various trees, and tried to run them with appropriate envrionment >> variable as seen in britney results. >> >> Here are new things: >> >> $ cd linux-azure-4.18.0/ && ./debian/tests/ubuntu-regression-suite >> ubuntu-regression-suite is pointless, if one cannot boot the kernel >> $ echo $? >> 77 >> >> $ cd linux-5.0.0/ && ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/ubuntu-regression-suite >> ubuntu-regression-suite is not requested, as there is no linux-meta trigger >> $ echo $? >> 77 >> >> I also validated that the current expected tests do run correctly, e.g.: >> >> $ ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/rebuild >> dpkg-buildpackage: info: source package linux >> dpkg-buildpackage: info: source version 5.0.0-7.8 >> dpkg-buildpackage: info: source distribution disco >> ... >> (build attempted) >> >> $ ADT_TEST_TRIGGERS=linux-meta/5.0.0.7.8 ./debian/tests/ubuntu-regression-suite >> Source Package Version: 5.0.0-7.8 >> Running Kernel Version: 4.19.0-13.14 >> ERROR: running version does not match source package >> ... >> (well, i did fake the trigger, but yeah, the right kernel is pulled >> in, in this case, and the test will be run) >> >> Please review these patches and apply them against all kernel >> flavours, in all releases. >> >> Dimitri John Ledkov (5): >> Set +x on rebuild testcase. >> Skip rebuild test, for regression-suite deps. >> Make ubuntu-regression-suite skippable on unbootable kernels. >> make rebuild use skippable error codes when skipping. >> Only run regression-suite, if requested to. >> >> debian/tests/control | 4 ++-- >> debian/tests/rebuild | 4 ++-- >> debian/tests/ubuntu-regression-suite | 26 ++++++++++++++++++++++++++ >> 3 files changed, 30 insertions(+), 4 deletions(-) >> mode change 100644 => 100755 debian/tests/rebuild >> >> -- >> 2.20.1 >> >> >> -- >> kernel-team mailing list >> kernel-team at lists.ubuntu.com >> https://lists.ubuntu.com/mailman/listinfo/kernel-team > >
On 3/27/19 3:50 PM, seth.forshee at canonical.com (Seth Forshee) wrote: > On Thu, Mar 07, 2019 at 03:32:17PM +0000, Dimitri John Ledkov wrote: >> This is a set of patches to unbreak autopkgtests, as designed and >> agreed in Malta kernel room mini-round-table between xnox, apw, >> sforshee, bjf. >> >> The general notio is that we don't skip tests enough, when they are >> over-triggered, and thus we keep on attempting to run them, when we >> should be bailing out. > > I think this looks sane. There are a few details that I'm not informed > enough about to provide useful review, so I'd still like apw to have a > look. > For example, at the moment in disco gcc-7 & gcc-8 has been stuck for 6 days now in disco-proposed. They trigger linux tests and run both the rebuild test (good and passes) and ubuntu-regression-testsuite (shouldn't run and fails due to lack of matching meta trigger). With these patches applies, the ubuntu-regression-testsuite will be correctly skipped / not attempted to be executed, as for these triggers we only care about the rebuild test. Note this is currently blocking velocity of updating the toolchain in Disco. And there is _no_ correct syntax to override this in hints. As we do want to see the results of the rebuild test, and block toolchain migration if rebuild test fails. (It currently passes, but flagged up as false negative). This has costed my engineering time to type this email, and a release team member to investigate this and flag it up. Yet, it could have sailed through automatically with the patches I have proposed. http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#gcc-7 http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#gcc-8 > One thing, we do require your signed-off-by for kernel patches. I > presume you are okay with us adding that when we apply the patches? Yes please, and sorry for not including those. > Aside from that: > > Acked-by: Seth Forshee <seth.forshee at canonical.com> > > Thanks, > Seth > >> >> It is hard for me to build a full abi bump kernel, with a matching >> bumped meta, and signed variants, so I monkey patched these tests into >> various trees, and tried to run them with appropriate envrionment >> variable as seen in britney results. >> >> Here are new things: >> >> $ cd linux-azure-4.18.0/ && ./debian/tests/ubuntu-regression-suite >> ubuntu-regression-suite is pointless, if one cannot boot the kernel >> $ echo $? >> 77 >> >> $ cd linux-5.0.0/ && ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/ubuntu-regression-suite >> ubuntu-regression-suite is not requested, as there is no linux-meta trigger >> $ echo $? >> 77 >> >> I also validated that the current expected tests do run correctly, e.g.: >> >> $ ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/rebuild >> dpkg-buildpackage: info: source package linux >> dpkg-buildpackage: info: source version 5.0.0-7.8 >> dpkg-buildpackage: info: source distribution disco >> ... >> (build attempted) >> >> $ ADT_TEST_TRIGGERS=linux-meta/5.0.0.7.8 ./debian/tests/ubuntu-regression-suite >> Source Package Version: 5.0.0-7.8 >> Running Kernel Version: 4.19.0-13.14 >> ERROR: running version does not match source package >> ... >> (well, i did fake the trigger, but yeah, the right kernel is pulled >> in, in this case, and the test will be run) >> >> Please review these patches and apply them against all kernel >> flavours, in all releases. >> >> Dimitri John Ledkov (5): >> Set +x on rebuild testcase. >> Skip rebuild test, for regression-suite deps. >> Make ubuntu-regression-suite skippable on unbootable kernels. >> make rebuild use skippable error codes when skipping. >> Only run regression-suite, if requested to. >> >> debian/tests/control | 4 ++-- >> debian/tests/rebuild | 4 ++-- >> debian/tests/ubuntu-regression-suite | 26 ++++++++++++++++++++++++++ >> 3 files changed, 30 insertions(+), 4 deletions(-) >> mode change 100644 => 100755 debian/tests/rebuild >> >> -- >> 2.20.1 >> >> >> -- >> kernel-team mailing list >> kernel-team at lists.ubuntu.com >> https://lists.ubuntu.com/mailman/listinfo/kernel-team > >
On Thu, Mar 07, 2019 at 03:32:17PM +0000, Dimitri John Ledkov wrote: > This is a set of patches to unbreak autopkgtests, as designed and > agreed in Malta kernel room mini-round-table between xnox, apw, > sforshee, bjf. > > The general notio is that we don't skip tests enough, when they are > over-triggered, and thus we keep on attempting to run them, when we > should be bailing out. > > It is hard for me to build a full abi bump kernel, with a matching > bumped meta, and signed variants, so I monkey patched these tests into > various trees, and tried to run them with appropriate envrionment > variable as seen in britney results. > > Here are new things: > > $ cd linux-azure-4.18.0/ && ./debian/tests/ubuntu-regression-suite > ubuntu-regression-suite is pointless, if one cannot boot the kernel > $ echo $? > 77 > > $ cd linux-5.0.0/ && ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/ubuntu-regression-suite > ubuntu-regression-suite is not requested, as there is no linux-meta trigger > $ echo $? > 77 > > I also validated that the current expected tests do run correctly, e.g.: > > $ ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/rebuild > dpkg-buildpackage: info: source package linux > dpkg-buildpackage: info: source version 5.0.0-7.8 > dpkg-buildpackage: info: source distribution disco > ... > (build attempted) > > $ ADT_TEST_TRIGGERS=linux-meta/5.0.0.7.8 ./debian/tests/ubuntu-regression-suite > Source Package Version: 5.0.0-7.8 > Running Kernel Version: 4.19.0-13.14 > ERROR: running version does not match source package > ... > (well, i did fake the trigger, but yeah, the right kernel is pulled > in, in this case, and the test will be run) > > Please review these patches and apply them against all kernel > flavours, in all releases. Applied to disco with your sign-off added, and a bug link: BugLink: https://bugs.launchpad.net/bugs/1823056 We'll give this a spin in disco to make sure everything works as expected, then look at SRU to older releases. Thank!
On 3/7/19 4:32 PM, Dimitri John Ledkov wrote: > This is a set of patches to unbreak autopkgtests, as designed and > agreed in Malta kernel room mini-round-table between xnox, apw, > sforshee, bjf. > > The general notio is that we don't skip tests enough, when they are > over-triggered, and thus we keep on attempting to run them, when we > should be bailing out. > > It is hard for me to build a full abi bump kernel, with a matching > bumped meta, and signed variants, so I monkey patched these tests into > various trees, and tried to run them with appropriate envrionment > variable as seen in britney results. > > Here are new things: > > $ cd linux-azure-4.18.0/ && ./debian/tests/ubuntu-regression-suite > ubuntu-regression-suite is pointless, if one cannot boot the kernel > $ echo $? > 77 > > $ cd linux-5.0.0/ && ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/ubuntu-regression-suite > ubuntu-regression-suite is not requested, as there is no linux-meta trigger > $ echo $? > 77 > > I also validated that the current expected tests do run correctly, e.g.: > > $ ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/rebuild > dpkg-buildpackage: info: source package linux > dpkg-buildpackage: info: source version 5.0.0-7.8 > dpkg-buildpackage: info: source distribution disco > ... > (build attempted) > > $ ADT_TEST_TRIGGERS=linux-meta/5.0.0.7.8 ./debian/tests/ubuntu-regression-suite > Source Package Version: 5.0.0-7.8 > Running Kernel Version: 4.19.0-13.14 > ERROR: running version does not match source package > ... > (well, i did fake the trigger, but yeah, the right kernel is pulled > in, in this case, and the test will be run) > > Please review these patches and apply them against all kernel > flavours, in all releases. > > Dimitri John Ledkov (5): > Set +x on rebuild testcase. > Skip rebuild test, for regression-suite deps. > Make ubuntu-regression-suite skippable on unbootable kernels. > make rebuild use skippable error codes when skipping. > Only run regression-suite, if requested to. > > debian/tests/control | 4 ++-- > debian/tests/rebuild | 4 ++-- > debian/tests/ubuntu-regression-suite | 26 ++++++++++++++++++++++++++ > 3 files changed, 30 insertions(+), 4 deletions(-) > mode change 100644 => 100755 debian/tests/rebuild > Applied to bionic and cosmic with Dimitri's sign-off, adding a subject prefix "UBUNTU: [Debian]" and a bug link: BugLink: https://bugs.launchpad.net/bugs/1823056 Not applied to Xenial yet as it has already been cranked for the current cycle, we can apply them later for the next one. Thanks, Kleber
On 3/7/19 4:32 PM, Dimitri John Ledkov wrote: > This is a set of patches to unbreak autopkgtests, as designed and > agreed in Malta kernel room mini-round-table between xnox, apw, > sforshee, bjf. > > The general notio is that we don't skip tests enough, when they are > over-triggered, and thus we keep on attempting to run them, when we > should be bailing out. > > It is hard for me to build a full abi bump kernel, with a matching > bumped meta, and signed variants, so I monkey patched these tests into > various trees, and tried to run them with appropriate envrionment > variable as seen in britney results. > > Here are new things: > > $ cd linux-azure-4.18.0/ && ./debian/tests/ubuntu-regression-suite > ubuntu-regression-suite is pointless, if one cannot boot the kernel > $ echo $? > 77 > > $ cd linux-5.0.0/ && ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/ubuntu-regression-suite > ubuntu-regression-suite is not requested, as there is no linux-meta trigger > $ echo $? > 77 > > I also validated that the current expected tests do run correctly, e.g.: > > $ ADT_TEST_TRIGGERS=binutils/2.32-5ubuntu1 ./debian/tests/rebuild > dpkg-buildpackage: info: source package linux > dpkg-buildpackage: info: source version 5.0.0-7.8 > dpkg-buildpackage: info: source distribution disco > ... > (build attempted) > > $ ADT_TEST_TRIGGERS=linux-meta/5.0.0.7.8 ./debian/tests/ubuntu-regression-suite > Source Package Version: 5.0.0-7.8 > Running Kernel Version: 4.19.0-13.14 > ERROR: running version does not match source package > ... > (well, i did fake the trigger, but yeah, the right kernel is pulled > in, in this case, and the test will be run) > > Please review these patches and apply them against all kernel > flavours, in all releases. > > Dimitri John Ledkov (5): > Set +x on rebuild testcase. > Skip rebuild test, for regression-suite deps. > Make ubuntu-regression-suite skippable on unbootable kernels. > make rebuild use skippable error codes when skipping. > Only run regression-suite, if requested to. > > debian/tests/control | 4 ++-- > debian/tests/rebuild | 4 ++-- > debian/tests/ubuntu-regression-suite | 26 ++++++++++++++++++++++++++ > 3 files changed, 30 insertions(+), 4 deletions(-) > mode change 100644 => 100755 debian/tests/rebuild > Applied to xenial/master-next branch, with the same changes made as for B/C. Also fixed some context conflicts. Thanks, Kleber