| Message ID | 20221027152921.448751-1-frank.heimes@canonical.com |
|---|---|
| Headers | show |
| Series | Ubuntu 18.04 kernel 4.15.0-194 crashes on IPL (LP: 1994601) | expand |
On 10/27/22 9:29 AM, frank.heimes@canonical.com wrote: > BugLink: https://bugs.launchpad.net/bugs/1994601 > > SRU Justification: > > [ Impact ] > > * Ubuntu 18.04 / bionic installations with latest kernel 4.15.0-194 > are no longer able to IPL (boot) on IBM z14 or newer hardware. > > * This issue got introduced by upstream commit e4f74400308c > "s390/archrandom: simplify back to earlier design and initialize earlier" > that was SRUed to 18.04/bionic based on LP#1989625, > which made changes in the s390s IPL/boot area of kernel/arch/random. > > * The reason seems to be that the bad patch moves the decision about > if arch randomness is available to setup.c function setup_randomness(). > This code uses a static_key s390_arch_random_available. > But in the Canonical kernel the initialization function > for the jump labels (where the static keys are based on) > jump_label_init() is called in generic start_kernel() > wheres in the upstream kernel the init function is > called early in setup_arch(). > > * Reverting this commit from bionic master-next makes bionic systems > again bootable. > (https://launchpad.net/~fheimes/+archive/ubuntu/test/) > > [ Test Plan ] > > * An IBM z14 or LinuxONE II or newer system is needed. > > * Now install latest bionic on that system - doesn't if it's on LPAR, > z/VM or KVM. > > * After the installation (an the trigger of the post install reboot), > the system will not come up. > > * To test a patched kernel with e4f74400308c can be tested in the > following way: > > * Install 18.04 GA and prevent it from doing any kernel updates. > > * Means, install in 'island' mode > or select in d-i 'Advanced Installation' > and explicitly choose '4.15.0-50 generic' to install. > > * That allows the system to come up and to update the kernel to > a modified one. > > * Then reboot and verify if the system comes up properly. > > [ Where problems could occur ] > > * Problems could occur due to the fact that the commit > was not cleanly reversible because of minor context changes. > > * Adjustments that were needed might break other things if not > done carefully. > > * Further commits (applied after e4f74400308c) may still rely > on the bad e4f74400308c commit - or even further patches > (from upstream stable). > > * In worst case IPL / boot might get broken, > even on hardware older than z14. > > * If the revert works fine can be easily tested and was tested based on > https://launchpad.net/~fheimes/+archive/ubuntu/test/ > and the above test plan. > > [ Other Info ] > > * Ubuntu 20.04 (focal, using legacy image with virt-install) > was tested as well, but is not affected by this issue. > > Frank Heimes (1): > Revert "s390/archrandom: simplify back to earlier design and > initialize earlier" > > arch/s390/crypto/arch_random.c | 111 ++++++++++++++++++++++++++++- > arch/s390/include/asm/archrandom.h | 27 +++---- > arch/s390/kernel/setup.c | 5 -- > 3 files changed, 121 insertions(+), 22 deletions(-) > Acked-by: Tim Gardner <tim.gardner@canonical.com>
Acked-by: Luke Nowakowski-Krijger <luke.nowakowskikrijger@canonical.com> On Thu, Oct 27, 2022 at 8:30 AM <frank.heimes@canonical.com> wrote: > BugLink: https://bugs.launchpad.net/bugs/1994601 > > SRU Justification: > > [ Impact ] > > * Ubuntu 18.04 / bionic installations with latest kernel 4.15.0-194 > are no longer able to IPL (boot) on IBM z14 or newer hardware. > > * This issue got introduced by upstream commit e4f74400308c > "s390/archrandom: simplify back to earlier design and initialize > earlier" > that was SRUed to 18.04/bionic based on LP#1989625, > which made changes in the s390s IPL/boot area of kernel/arch/random. > > * The reason seems to be that the bad patch moves the decision about > if arch randomness is available to setup.c function setup_randomness(). > This code uses a static_key s390_arch_random_available. > But in the Canonical kernel the initialization function > for the jump labels (where the static keys are based on) > jump_label_init() is called in generic start_kernel() > wheres in the upstream kernel the init function is > called early in setup_arch(). > > * Reverting this commit from bionic master-next makes bionic systems > again bootable. > (https://launchpad.net/~fheimes/+archive/ubuntu/test/) > > [ Test Plan ] > > * An IBM z14 or LinuxONE II or newer system is needed. > > * Now install latest bionic on that system - doesn't if it's on LPAR, > z/VM or KVM. > > * After the installation (an the trigger of the post install reboot), > the system will not come up. > > * To test a patched kernel with e4f74400308c can be tested in the > following way: > > * Install 18.04 GA and prevent it from doing any kernel updates. > > * Means, install in 'island' mode > or select in d-i 'Advanced Installation' > and explicitly choose '4.15.0-50 generic' to install. > > * That allows the system to come up and to update the kernel to > a modified one. > > * Then reboot and verify if the system comes up properly. > > [ Where problems could occur ] > > * Problems could occur due to the fact that the commit > was not cleanly reversible because of minor context changes. > > * Adjustments that were needed might break other things if not > done carefully. > > * Further commits (applied after e4f74400308c) may still rely > on the bad e4f74400308c commit - or even further patches > (from upstream stable). > > * In worst case IPL / boot might get broken, > even on hardware older than z14. > > * If the revert works fine can be easily tested and was tested based on > https://launchpad.net/~fheimes/+archive/ubuntu/test/ > and the above test plan. > > [ Other Info ] > > * Ubuntu 20.04 (focal, using legacy image with virt-install) > was tested as well, but is not affected by this issue. > > Frank Heimes (1): > Revert "s390/archrandom: simplify back to earlier design and > initialize earlier" > > arch/s390/crypto/arch_random.c | 111 ++++++++++++++++++++++++++++- > arch/s390/include/asm/archrandom.h | 27 +++---- > arch/s390/kernel/setup.c | 5 -- > 3 files changed, 121 insertions(+), 22 deletions(-) > > -- > 2.25.1 > > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team >
BugLink: https://bugs.launchpad.net/bugs/1994601 SRU Justification: [ Impact ] * Ubuntu 18.04 / bionic installations with latest kernel 4.15.0-194 are no longer able to IPL (boot) on IBM z14 or newer hardware. * This issue got introduced by upstream commit e4f74400308c "s390/archrandom: simplify back to earlier design and initialize earlier" that was SRUed to 18.04/bionic based on LP#1989625, which made changes in the s390s IPL/boot area of kernel/arch/random. * The reason seems to be that the bad patch moves the decision about if arch randomness is available to setup.c function setup_randomness(). This code uses a static_key s390_arch_random_available. But in the Canonical kernel the initialization function for the jump labels (where the static keys are based on) jump_label_init() is called in generic start_kernel() wheres in the upstream kernel the init function is called early in setup_arch(). * Reverting this commit from bionic master-next makes bionic systems again bootable. (https://launchpad.net/~fheimes/+archive/ubuntu/test/) [ Test Plan ] * An IBM z14 or LinuxONE II or newer system is needed. * Now install latest bionic on that system - doesn't if it's on LPAR, z/VM or KVM. * After the installation (an the trigger of the post install reboot), the system will not come up. * To test a patched kernel with e4f74400308c can be tested in the following way: * Install 18.04 GA and prevent it from doing any kernel updates. * Means, install in 'island' mode or select in d-i 'Advanced Installation' and explicitly choose '4.15.0-50 generic' to install. * That allows the system to come up and to update the kernel to a modified one. * Then reboot and verify if the system comes up properly. [ Where problems could occur ] * Problems could occur due to the fact that the commit was not cleanly reversible because of minor context changes. * Adjustments that were needed might break other things if not done carefully. * Further commits (applied after e4f74400308c) may still rely on the bad e4f74400308c commit - or even further patches (from upstream stable). * In worst case IPL / boot might get broken, even on hardware older than z14. * If the revert works fine can be easily tested and was tested based on https://launchpad.net/~fheimes/+archive/ubuntu/test/ and the above test plan. [ Other Info ] * Ubuntu 20.04 (focal, using legacy image with virt-install) was tested as well, but is not affected by this issue. Frank Heimes (1): Revert "s390/archrandom: simplify back to earlier design and initialize earlier" arch/s390/crypto/arch_random.c | 111 ++++++++++++++++++++++++++++- arch/s390/include/asm/archrandom.h | 27 +++---- arch/s390/kernel/setup.c | 5 -- 3 files changed, 121 insertions(+), 22 deletions(-)