From patchwork Fri Dec 6 23:09:56 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kamal Mostafa X-Patchwork-Id: 298676 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) by ozlabs.org (Postfix) with ESMTP id 333A22C00DE for ; Sun, 8 Dec 2013 01:02:33 +1100 (EST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.76) (envelope-from ) id 1VpISf-0003dV-IN; Sat, 07 Dec 2013 14:02:25 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtp (Exim 4.76) (envelope-from ) id 1Vp4d9-0005Ps-V3 for kernel-team@lists.ubuntu.com; Fri, 06 Dec 2013 23:16:19 +0000 Received: from c-67-160-231-162.hsd1.ca.comcast.net ([67.160.231.162] helo=fourier) by youngberry.canonical.com with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1Vp4Zb-0004d1-Vl; Fri, 06 Dec 2013 23:12:40 +0000 Received: from kamal by fourier with local (Exim 4.80) (envelope-from ) id 1Vp4ZZ-0000qN-7G; Fri, 06 Dec 2013 15:12:37 -0800 From: Kamal Mostafa To: linux-kernel@vger.kernel.org, stable@vger.kernel.org, kernel-team@lists.ubuntu.com Subject: [PATCH 3.8 072/152] x86/ioapic/kcrash: Prevent crash_kexec() from deadlocking on ioapic_lock Date: Fri, 6 Dec 2013 15:09:56 -0800 Message-Id: <1386371476-2477-73-git-send-email-kamal@canonical.com> X-Mailer: git-send-email 1.8.3.2 In-Reply-To: <1386371476-2477-1-git-send-email-kamal@canonical.com> References: <1386371476-2477-1-git-send-email-kamal@canonical.com> X-Extended-Stable: 3.8 X-Mailman-Approved-At: Sat, 07 Dec 2013 14:02:19 +0000 Cc: Andi Kleen , Gleb Natapov , Konrad Rzeszutek Wilk , Kamal Mostafa , Yoshihiro YUNOMAE , Marcelo Tosatti , Ingo Molnar , Masami Hiramatsu , Sebastian Andrzej Siewior , Zhang Yanfei , "Eric W. Biederman" , yrl.pp-manager.tt@hitachi.com, Hidehiro Kawai , Seiji Aguchi , Joerg Roedel X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.14 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: kernel-team-bounces@lists.ubuntu.com 3.8.13.14 -stable review patch. If anyone has any objections, please let me know. ------------------ From: Yoshihiro YUNOMAE commit 17405453f4ad0220721a29978692081be6392b8f upstream. Prevent crash_kexec() from deadlocking on ioapic_lock. When crash_kexec() is executed on a CPU, the CPU will take ioapic_lock in disable_IO_APIC(). So if the cpu gets an NMI while locking ioapic_lock, a deadlock will happen. In this patch, ioapic_lock is zapped/initialized before disable_IO_APIC(). You can reproduce this deadlock the following way: 1. Add mdelay(1000) after raw_spin_lock_irqsave() in native_ioapic_set_affinity()@arch/x86/kernel/apic/io_apic.c Although the deadlock can occur without this modification, it will increase the potential of the deadlock problem. 2. Build and install the kernel 3. Set up the OS which will run panic() and kexec when NMI is injected # echo "kernel.unknown_nmi_panic=1" >> /etc/sysctl.conf # vim /etc/default/grub add "nmi_watchdog=0 crashkernel=256M" in GRUB_CMDLINE_LINUX line # grub2-mkconfig 4. Reboot the OS 5. Run following command for each vcpu on the guest # while true; do echo > /proc/irq//smp_affinitity; done; By running this command, cpus will get ioapic_lock for setting affinity. 6. Inject NMI (push a dump button or execute 'virsh inject-nmi ' if you use VM). After injecting NMI, panic() is called in an nmi-handler context. Then, kexec will normally run in panic(), but the operation will be stopped by deadlock on ioapic_lock in crash_kexec()->machine_crash_shutdown()-> native_machine_crash_shutdown()->disable_IO_APIC()->clear_IO_APIC()-> clear_IO_APIC_pin()->ioapic_read_entry(). Signed-off-by: Yoshihiro YUNOMAE Cc: Andi Kleen Cc: Gleb Natapov Cc: Konrad Rzeszutek Wilk Cc: Joerg Roedel Cc: Marcelo Tosatti Cc: Hidehiro Kawai Cc: Sebastian Andrzej Siewior Cc: Zhang Yanfei Cc: Eric W. Biederman Cc: yrl.pp-manager.tt@hitachi.com Cc: Masami Hiramatsu Cc: Seiji Aguchi Link: http://lkml.kernel.org/r/20130820070107.28245.83806.stgit@yunodevel Signed-off-by: Ingo Molnar [ kamal: 3.8 stable prereq for 522e664 x86/apic: Disable I/O APIC before shutdown of the local APIC ] Signed-off-by: Kamal Mostafa --- arch/x86/include/asm/apic.h | 2 ++ arch/x86/kernel/apic/io_apic.c | 5 +++++ arch/x86/kernel/crash.c | 4 +++- 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h index 3388034..cb8bc81 100644 --- a/arch/x86/include/asm/apic.h +++ b/arch/x86/include/asm/apic.h @@ -688,4 +688,6 @@ extern int default_check_phys_apicid_present(int phys_apicid); #endif /* CONFIG_X86_LOCAL_APIC */ +extern void ioapic_zap_locks(void); + #endif /* _ASM_X86_APIC_H */ diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c index b739d39..c605173 100644 --- a/arch/x86/kernel/apic/io_apic.c +++ b/arch/x86/kernel/apic/io_apic.c @@ -1513,6 +1513,11 @@ static void __init setup_timer_IRQ0_pin(unsigned int ioapic_idx, ioapic_write_entry(ioapic_idx, pin, entry); } +void ioapic_zap_locks(void) +{ + raw_spin_lock_init(&ioapic_lock); +} + __apicdebuginit(void) print_IO_APIC(int ioapic_idx) { int i; diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c index 74467fe..e0e0841 100644 --- a/arch/x86/kernel/crash.c +++ b/arch/x86/kernel/crash.c @@ -128,7 +128,9 @@ void native_machine_crash_shutdown(struct pt_regs *regs) cpu_emergency_svm_disable(); lapic_shutdown(); -#if defined(CONFIG_X86_IO_APIC) +#ifdef CONFIG_X86_IO_APIC + /* Prevent crash_kexec() from deadlocking on ioapic_lock. */ + ioapic_zap_locks(); disable_IO_APIC(); #endif #ifdef CONFIG_HPET_TIMER