| Message ID | cover.1426793116.git.geoff@infradead.org |
|---|---|
| State | New |
| Headers | show |
On Thu, Mar 19, 2015 at 08:35:27PM +0000, Geoff Levand wrote: > Hi All, Hi Geoff, As a heads-up, I'm not going to have the chanec to review this until after the FW summit next week. [...] > KVM > --- > > PROBLEM: KVM acquires hypervisor resources on startup, but does not free those > resources on shutdown, so the first stage kernel cannot be shutdown properly > when using kexec. > > WORK-AROUND: Build the first stage kernel with CONFIG_KVM=n, or apply KVM bug > fix patches from [1]. > > FIX: Takahiro Akashi has preliminary patches to fix the KVM shutdown problem. I > have those in my master branch at [1]. KVM + kexec works properly with that > branch. Patches neeeded: > > arm64: kvm: add a cpu tear-down function > arm64: kexec: fix kvm issue > arm64/kvm: Remove !KEXEC Kconfig dependency > arm64/kexec: Enable kexec in the arm64 defconfig These (or some descendents thereof) are going to be a prerequisite for kexec. > /memreserve/ > ---------- > > PROBLEM: Device tree /memreserve/ entries are not available in > /proc/device-tree. For systems that have /memreserve/ entries and use > /proc/device-tree during kexec, the second stage kernel will use the reserved > regions and the system will become unstable. > > WORK-AROUND: Enable the kernel config option CONFIG_SYSFS=y to expose a binary > device tree to user space at /sys/firmware/fdt that includes /memreserve/ > entries OR pass a user specified DTB using the kexec --dtb option. > > FIX: This is expected behavior. To maximize user support, rework device tree > definitions to not use /memreserve/ entries. I'd say that requiring CONFIG_SYSFS is the real fix here. That's a kernel change that maximises compatibility with DTBs (which we don't really have control over). I hope that the tools warn somehow if they cannot see the fdt. Mark.
Hi, I rebased my series to the latest arm64 for-next/core (v4.0-rc4) and pushed it out to: git://git.kernel.org/pub/scm/linux/kernel/git/geoff/linux-kexec.git kexec-v9 -Geoff On Thu, 2015-03-19 at 20:35 +0000, Geoff Levand wrote: > Hi All, > > This series adds the core support for kexec re-boots on arm64. This v8 of the > series is mainly just a rebase to Linux-4.0-rc3, and a few very minor changes > requested for v7. > > To load a second stage kernel and execute a kexec re-boot on arm64 my patches to > kexec-tools [2], which have not yet been merged upstream, are needed. > > I have tested with the ARM VE fast model, the ARM Base model and the ARM > Foundation model with various kernel config options for both the first and > second stage kernels. Kexec on EFI systems works correctly. With the ACPI > kernel patches from [3] applied, kexec on ACPI systems seeems to work correctly. > More ACPI + kexec testing is needed. > > Patch 1 here moves the macros from proc-macros.S to asm/assembler.h so that the > dcache_line_size macro it defines can be uesd by kexec's relocate kernel > routine. > > Patches 2-4 rework the arm64 hcall mechanism to give the arm64 soft_restart() > routine the ability to switch exception levels from EL1 to EL2 for kernels that > were entered in EL2. > > Patches 5-6 add the actual kexec support. > > Please consider all patches for inclusion. > > [1] https://git.kernel.org/cgit/linux/kernel/git/geoff/linux-kexec.git > [2] https://git.kernel.org/cgit/linux/kernel/git/geoff/kexec-tools.git > [3] http://git.linaro.org/leg/acpi/acpi.git #acpi-topic-juno-fvp > > Several things are known to have problems on kexec re-boot: > > spin-table > ---------- > > PROBLEM: The spin-table enable method does not implement all the methods needed > for CPU hot-plug, so the first stage kernel cannot be shutdown properly. > > WORK-AROUND: Upgrade to system firmware that provides PSCI enable method > support, OR build the first stage kernel with CONFIG_SMP=n, OR pass 'maxcpus=1' > on the first stage kernel command line. > > FIX: Upgrade system firmware to provide PSCI enable method support or add > missing spin-table support to the kernel. > > KVM > --- > > PROBLEM: KVM acquires hypervisor resources on startup, but does not free those > resources on shutdown, so the first stage kernel cannot be shutdown properly > when using kexec. > > WORK-AROUND: Build the first stage kernel with CONFIG_KVM=n, or apply KVM bug > fix patches from [1]. > > FIX: Takahiro Akashi has preliminary patches to fix the KVM shutdown problem. I > have those in my master branch at [1]. KVM + kexec works properly with that > branch. Patches neeeded: > > arm64: kvm: add a cpu tear-down function > arm64: kexec: fix kvm issue > arm64/kvm: Remove !KEXEC Kconfig dependency > arm64/kexec: Enable kexec in the arm64 defconfig > > /memreserve/ > ---------- > > PROBLEM: Device tree /memreserve/ entries are not available in > /proc/device-tree. For systems that have /memreserve/ entries and use > /proc/device-tree during kexec, the second stage kernel will use the reserved > regions and the system will become unstable. > > WORK-AROUND: Enable the kernel config option CONFIG_SYSFS=y to expose a binary > device tree to user space at /sys/firmware/fdt that includes /memreserve/ > entries OR pass a user specified DTB using the kexec --dtb option. > > FIX: This is expected behavior. To maximize user support, rework device tree > definitions to not use /memreserve/ entries. > > -Geoff
On 19/03/15 20:35, Geoff Levand wrote: > Add three new files, kexec.h, machine_kexec.c and relocate_kernel.S to the > arm64 architecture that add support for the kexec re-boot mechanism > (CONFIG_KEXEC) on arm64 platforms. > > With the addition of arm64 kexec support shutdown code paths through the kernel > are executed that previously were not. To avoid system instability do to > problems in the current arm64 KVM kernel implementation add a Kconfig dependency > on !KEXEC to the arm64 KVM menu item. > > Signed-off-by: Geoff Levand <geoff@infradead.org> > --- > arch/arm64/Kconfig | 9 +++ > arch/arm64/include/asm/kexec.h | 48 ++++++++++++ > arch/arm64/kernel/Makefile | 1 + > arch/arm64/kernel/machine_kexec.c | 125 ++++++++++++++++++++++++++++++ > arch/arm64/kernel/relocate_kernel.S | 149 ++++++++++++++++++++++++++++++++++++ > arch/arm64/kvm/Kconfig | 1 + > include/uapi/linux/kexec.h | 1 + > 7 files changed, 334 insertions(+) > create mode 100644 arch/arm64/include/asm/kexec.h > create mode 100644 arch/arm64/kernel/machine_kexec.c > create mode 100644 arch/arm64/kernel/relocate_kernel.S > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index 1b8e973..5a606d1 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -528,6 +528,15 @@ config SECCOMP > and the task is only allowed to execute a few safe syscalls > defined by each seccomp mode. > > +config KEXEC > + depends on (!SMP || PM_SLEEP_SMP) > + bool "kexec system call" > + ---help--- > + kexec is a system call that implements the ability to shutdown your > + current kernel, and to start another kernel. It is like a reboot > + but it is independent of the system firmware. And like a reboot > + you can start any kernel with it, not just Linux. > + > config XEN_DOM0 > def_bool y > depends on XEN > diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h > new file mode 100644 > index 0000000..3530ff5 > --- /dev/null > +++ b/arch/arm64/include/asm/kexec.h > @@ -0,0 +1,48 @@ > +/* > + * kexec for arm64 > + * > + * Copyright (C) Linaro. > + * Copyright (C) Futurewei Technologies. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + */ > + > +#if !defined(_ARM64_KEXEC_H) > +#define _ARM64_KEXEC_H > + > +/* Maximum physical address we can use pages from */ > + > +#define KEXEC_SOURCE_MEMORY_LIMIT (-1UL) > + > +/* Maximum address we can reach in physical address mode */ > + > +#define KEXEC_DESTINATION_MEMORY_LIMIT (-1UL) > + > +/* Maximum address we can use for the control code buffer */ > + > +#define KEXEC_CONTROL_MEMORY_LIMIT (-1UL) > + > +#define KEXEC_CONTROL_PAGE_SIZE 4096 > + > +#define KEXEC_ARCH KEXEC_ARCH_ARM64 > + > +#if !defined(__ASSEMBLY__) > + > +/** > + * crash_setup_regs() - save registers for the panic kernel > + * > + * @newregs: registers are saved here > + * @oldregs: registers to be saved (may be %NULL) > + */ > + > +static inline void crash_setup_regs(struct pt_regs *newregs, > + struct pt_regs *oldregs) > +{ > + /* Empty routine needed to avoid build errors. */ > +} > + > +#endif /* !defined(__ASSEMBLY__) */ > + > +#endif > diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile > index 5ee07ee..da9a7ee 100644 > --- a/arch/arm64/kernel/Makefile > +++ b/arch/arm64/kernel/Makefile > @@ -35,6 +35,7 @@ arm64-obj-$(CONFIG_KGDB) += kgdb.o > arm64-obj-$(CONFIG_EFI) += efi.o efi-stub.o efi-entry.o > arm64-obj-$(CONFIG_PCI) += pci.o > arm64-obj-$(CONFIG_ARMV8_DEPRECATED) += armv8_deprecated.o > +arm64-obj-$(CONFIG_KEXEC) += machine_kexec.o relocate_kernel.o > > obj-y += $(arm64-obj-y) vdso/ > obj-m += $(arm64-obj-m) > diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c > new file mode 100644 > index 0000000..f1387d0 > --- /dev/null > +++ b/arch/arm64/kernel/machine_kexec.c > @@ -0,0 +1,125 @@ > +/* > + * kexec for arm64 > + * > + * Copyright (C) Linaro. > + * Copyright (C) Futurewei Technologies. > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + */ > + > +#include <linux/kexec.h> > +#include <linux/of_fdt.h> > +#include <linux/slab.h> > +#include <linux/uaccess.h> > + > +#include <asm/cacheflush.h> > +#include <asm/system_misc.h> > + > +/* Global variables for the relocate_kernel routine. */ > +extern const unsigned char relocate_new_kernel[]; > +extern const unsigned long relocate_new_kernel_size; > +extern unsigned long arm64_kexec_dtb_addr; > +extern unsigned long arm64_kexec_kimage_head; > +extern unsigned long arm64_kexec_kimage_start; > + > +void machine_kexec_cleanup(struct kimage *image) > +{ > + /* Empty routine needed to avoid build errors. */ > +} > + > +/** > + * machine_kexec_prepare - Prepare for a kexec reboot. > + * > + * Called from the core kexec code when a kernel image is loaded. > + */ > +int machine_kexec_prepare(struct kimage *image) > +{ > + arm64_kexec_kimage_start = image->start; > + return 0; > +} > + > +/** > + * kexec_list_flush - Helper to flush the kimage list to PoC. > + */ > +static void kexec_list_flush(unsigned long kimage_head) > +{ > + void *dest; What is the use of dest ? > + unsigned long *entry; > + > + for (entry = &kimage_head, dest = NULL; ; entry++) { > + unsigned int flag = *entry & > + (IND_DESTINATION | IND_INDIRECTION | IND_DONE | > + IND_SOURCE); You could instead do : flag = *entry & IND_FLAGS; > + void *addr = phys_to_virt(*entry & PAGE_MASK); > + > + switch (flag) { > + case IND_INDIRECTION: > + entry = (unsigned long *)addr - 1; > + __flush_dcache_area(addr, PAGE_SIZE); > + break; > + case IND_DESTINATION: > + dest = addr; > + break; > + case IND_SOURCE: > + __flush_dcache_area(addr, PAGE_SIZE); > + dest += PAGE_SIZE; > + break; > + case IND_DONE: > + return; > + default: > + BUG(); > + } > + } > +} Thanks Suzuki
Hi Suzuki, On Tue, 2015-04-07 at 17:38 +0100, Suzuki K. Poulose wrote: > On 19/03/15 20:35, Geoff Levand wrote: > > +static void kexec_list_flush(unsigned long kimage_head) > > +{ > > + void *dest; > > What is the use of dest ? dest is left over from when I had a generic list walk routine. I'll remove it. > > + unsigned long *entry; > > + > > + for (entry = &kimage_head, dest = NULL; ; entry++) { > > + unsigned int flag = *entry & > > + (IND_DESTINATION | IND_INDIRECTION | IND_DONE | > > + IND_SOURCE); > You could instead do : > > flag = *entry & IND_FLAGS; Yes, now that the patch to define IND_FLAGS has been merged. -Geoff
On 19/03/15 20:35, Geoff Levand wrote: > Hi All, > > This series adds the core support for kexec re-boots on arm64. This v8 of the > series is mainly just a rebase to Linux-4.0-rc3, and a few very minor changes > requested for v7. > > To load a second stage kernel and execute a kexec re-boot on arm64 my patches to > kexec-tools [2], which have not yet been merged upstream, are needed. > > I have tested with the ARM VE fast model, the ARM Base model and the ARM > Foundation model with various kernel config options for both the first and > second stage kernels. Kexec on EFI systems works correctly. With the ACPI > kernel patches from [3] applied, kexec on ACPI systems seeems to work correctly. > More ACPI + kexec testing is needed. > > Patch 1 here moves the macros from proc-macros.S to asm/assembler.h so that the > dcache_line_size macro it defines can be uesd by kexec's relocate kernel > routine. > > Patches 2-4 rework the arm64 hcall mechanism to give the arm64 soft_restart() > routine the ability to switch exception levels from EL1 to EL2 for kernels that > were entered in EL2. > > Patches 5-6 add the actual kexec support. > > Please consider all patches for inclusion. > > [1] https://git.kernel.org/cgit/linux/kernel/git/geoff/linux-kexec.git > [2] https://git.kernel.org/cgit/linux/kernel/git/geoff/kexec-tools.git Btw, I get the following build failure for kexec-tools master branch, with Linaro tool chain (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09 ) $ ./configure --host=aarch64-linux-gnu [...] $ make -j4 kexec/arch/arm64/kexec-arm64.c: In function ‘machine_verify_elf_rel’: kexec/arch/arm64/kexec-arm64.c:970:29: error: ‘EM_AARCH64’ undeclared (first use in this function) return (ehdr->e_machine == EM_AARCH64); ^ kexec/arch/arm64/kexec-arm64.c:970:29: note: each undeclared identifier is reported only once for each function it appears in kexec/arch/arm64/kexec-arm64.c:971:1: warning: control reaches end of non-void function [-Wreturn-type] } ^ make: *** [kexec/arch/arm64/kexec-arm64.o] Error 1 make: *** Waiting for unfinished jobs.... ---- You may need to add the definition of EM_AARCH64 to include/elf.h and include that instead of linux/elf.h, like the other archs. Cheers Suzuki
Hi, On Wed, 2015-04-08 at 12:16 +0100, Suzuki K. Poulose wrote: > Btw, I get the following build failure for kexec-tools master branch, > with Linaro tool chain (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro > GCC 4.9-2014.09 ) > > $ ./configure --host=aarch64-linux-gnu > [...] > $ make -j4 > kexec/arch/arm64/kexec-arm64.c: In function ‘machine_verify_elf_rel’: > kexec/arch/arm64/kexec-arm64.c:970:29: error: ‘EM_AARCH64’ undeclared I guess that toolchain is using some ancient headers. As a workaround you can use configure CPPFLAGS='-DEM_AARCH64=183'. -Geoff
Hi Geoff, I suppose there would be a next revision of kexec patches. When is the that expected? Thanks a lot for working on this. ~Pratyush
Hi Pratyush, On Fri, 2015-07-03 at 09:09 +0530, Pratyush Anand wrote: > I suppose there would be a next revision of kexec patches. When is the > that expected? As I just replied to Marc's mail, I'll work on preparing a rebased kexec patch set the next chance I get. I'm hoping I can start on it his week. -Geoff
On Thu, Mar 19, 2015 at 3:35 PM, Geoff Levand <geoff@infradead.org> wrote: > The existing arm64 hcall implementations are limited in that they only allow > for two distinct hcalls; with the x0 register either zero or not zero. Also, > the API of the hyp-stub exception vector routines and the KVM exception vector > routines differ; hyp-stub uses a non-zero value in x0 to implement > __hyp_set_vectors, whereas KVM uses it to implement kvm_call_hyp. > > To allow for additional hcalls to be defined and to make the arm64 hcall API > more consistent across exception vector routines, change the hcall > implementations to use the 16 bit immediate value of the HVC instruction to > specify the hcall type. > > Define three new preprocessor macros HVC_CALL_HYP, HVC_GET_VECTORS, and > HVC_SET_VECTORS to be used as hcall type specifiers and convert the > existing __hyp_get_vectors(), __hyp_set_vectors() and kvm_call_hyp() routines > to use these new macros when executing an HVC call. Also, change the > corresponding hyp-stub and KVM el1_sync exception vector routines to use these > new macros. > > Signed-off-by: Geoff Levand <geoff@infradead.org> What is the status of this patch? I see it's not in 4.2. I ask because this patch might be breaking our 4.2-based kernel, and I'm just wondering if it's still applicable.
On Wed, 2015-09-23 at 14:21 -0500, Timur Tabi wrote: > What is the status of this patch? I see it's not in 4.2. I ask > because this patch might be breaking our 4.2-based kernel, and I'm > just wondering if it's still applicable. I have it in my linux-kexec repo, which currently based on 4.3-rc2. I plan to submit it again in the next week or so after testing. -Geoff
Hi All, This series adds the core support for kexec re-boots on arm64. This v8 of the series is mainly just a rebase to Linux-4.0-rc3, and a few very minor changes requested for v7. To load a second stage kernel and execute a kexec re-boot on arm64 my patches to kexec-tools [2], which have not yet been merged upstream, are needed. I have tested with the ARM VE fast model, the ARM Base model and the ARM Foundation model with various kernel config options for both the first and second stage kernels. Kexec on EFI systems works correctly. With the ACPI kernel patches from [3] applied, kexec on ACPI systems seeems to work correctly. More ACPI + kexec testing is needed. Patch 1 here moves the macros from proc-macros.S to asm/assembler.h so that the dcache_line_size macro it defines can be uesd by kexec's relocate kernel routine. Patches 2-4 rework the arm64 hcall mechanism to give the arm64 soft_restart() routine the ability to switch exception levels from EL1 to EL2 for kernels that were entered in EL2. Patches 5-6 add the actual kexec support. Please consider all patches for inclusion. [1] https://git.kernel.org/cgit/linux/kernel/git/geoff/linux-kexec.git [2] https://git.kernel.org/cgit/linux/kernel/git/geoff/kexec-tools.git [3] http://git.linaro.org/leg/acpi/acpi.git #acpi-topic-juno-fvp Several things are known to have problems on kexec re-boot: spin-table ---------- PROBLEM: The spin-table enable method does not implement all the methods needed for CPU hot-plug, so the first stage kernel cannot be shutdown properly. WORK-AROUND: Upgrade to system firmware that provides PSCI enable method support, OR build the first stage kernel with CONFIG_SMP=n, OR pass 'maxcpus=1' on the first stage kernel command line. FIX: Upgrade system firmware to provide PSCI enable method support or add missing spin-table support to the kernel. KVM --- PROBLEM: KVM acquires hypervisor resources on startup, but does not free those resources on shutdown, so the first stage kernel cannot be shutdown properly when using kexec. WORK-AROUND: Build the first stage kernel with CONFIG_KVM=n, or apply KVM bug fix patches from [1]. FIX: Takahiro Akashi has preliminary patches to fix the KVM shutdown problem. I have those in my master branch at [1]. KVM + kexec works properly with that branch. Patches neeeded: arm64: kvm: add a cpu tear-down function arm64: kexec: fix kvm issue arm64/kvm: Remove !KEXEC Kconfig dependency arm64/kexec: Enable kexec in the arm64 defconfig /memreserve/ ---------- PROBLEM: Device tree /memreserve/ entries are not available in /proc/device-tree. For systems that have /memreserve/ entries and use /proc/device-tree during kexec, the second stage kernel will use the reserved regions and the system will become unstable. WORK-AROUND: Enable the kernel config option CONFIG_SYSFS=y to expose a binary device tree to user space at /sys/firmware/fdt that includes /memreserve/ entries OR pass a user specified DTB using the kexec --dtb option. FIX: This is expected behavior. To maximize user support, rework device tree definitions to not use /memreserve/ entries. -Geoff The following changes since commit 06e5801b8cb3fc057d88cb4dc03c0b64b2744cda: Linux 4.0-rc4 (2015-03-15 17:38:20 -0700) are available in the git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/geoff/linux-kexec.git kexec-v8 for you to fetch changes up to 9d94104463ae6e3472526fb69e8111201c4a1fa7: arm64/kexec: Add pr_devel output (2015-03-19 12:21:43 -0700) ---------------------------------------------------------------- Geoff Levand (6): arm64: Fold proc-macros.S into assembler.h arm64: Convert hcalls to use HVC immediate value arm64: Add new hcall HVC_CALL_FUNC arm64: Add EL2 switch to soft_restart arm64/kexec: Add core kexec support arm64/kexec: Add pr_devel output arch/arm64/Kconfig | 9 ++ arch/arm64/include/asm/assembler.h | 37 ++++++- arch/arm64/include/asm/kexec.h | 48 +++++++++ arch/arm64/include/asm/proc-fns.h | 4 +- arch/arm64/include/asm/virt.h | 40 ++++++++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/hyp-stub.S | 43 +++++--- arch/arm64/kernel/machine_kexec.c | 189 ++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 6 +- arch/arm64/kernel/relocate_kernel.S | 149 ++++++++++++++++++++++++++++ arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp.S | 16 +-- arch/arm64/mm/cache.S | 2 - arch/arm64/mm/proc-macros.S | 54 ----------- arch/arm64/mm/proc.S | 49 +++++++--- include/uapi/linux/kexec.h | 1 + 16 files changed, 557 insertions(+), 92 deletions(-) create mode 100644 arch/arm64/include/asm/kexec.h create mode 100644 arch/arm64/kernel/machine_kexec.c create mode 100644 arch/arm64/kernel/relocate_kernel.S delete mode 100644 arch/arm64/mm/proc-macros.S