From patchwork Sat Dec 2 13:48:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Balbir Singh X-Patchwork-Id: 843885 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3ypsvs4PZLz9s71 for ; Sun, 3 Dec 2017 00:50:01 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Avo65wVE"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3ypsvs34YdzDx66 for ; Sun, 3 Dec 2017 00:50:01 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Avo65wVE"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::241; helo=mail-pg0-x241.google.com; envelope-from=bsingharora@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Avo65wVE"; dkim-atps=neutral Received: from mail-pg0-x241.google.com (mail-pg0-x241.google.com [IPv6:2607:f8b0:400e:c05::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3ypssy05ndzDwNm for ; Sun, 3 Dec 2017 00:48:21 +1100 (AEDT) Received: by mail-pg0-x241.google.com with SMTP id o2so5684166pgc.8 for ; Sat, 02 Dec 2017 05:48:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=2pYqZsohWNrU+Yd8GVTKcySkaUCkJZFiCjmLlsic7KM=; b=Avo65wVE6p8x4vTFBUI3SgJHjUu3U5621shYRXng6XGx3QnDhpP4XAqqyHBofqqRri 6zMHswni/ls/Koo8SoemY143GN+FYbJA57MMwPnKT64mZ5m1zQ9P38x+/5W8GyA0CLUO aWDQQJDzXGqmeccVSvgvGkZs6Gtq9ScOq+iNq0bhyl4KaT3o7baSOn8ZK/bXKvDz+Bma dQ3Axnxs2mfeIxwBnwIuwVZf3ImQT0/JY5vIUM06y3XNId0ccNUBPighFrxe8bSeFC2f P9m3VVqZVwxw3ZKNk0ggYJHxkniNAb03E0h+ahOeGrIp/1hEy4bd2MXVt1CCwkqRvuaf LcaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=2pYqZsohWNrU+Yd8GVTKcySkaUCkJZFiCjmLlsic7KM=; b=eZeVNO1qKCKrlvy7+kDBLywNRq49fUNd9OOLuoN9lXVHCkcrluLXIiwggxHyUWULTF w+hIz9OF0AfreLPv6qYv4VJSmDLF+4AP/tIsihfkDMwEbKIJ5ypVgvwy2BjoIiSFskIu Do97SX2h8YvVpkJMt2qWAkBTGl9anfX8W+qPMZ5kFav6mtTi991ezgZKUskcq8dTBwkU +07fuhXNF9znzefExjEKYkhy0RlZZGH2BgOZAi5UwcQ06TRiOeLsjpILr3xM6iw75Oli BsLW7HPNxnATUr7b3at7D4whj7Bj4+EHokMNo/tIet5wM1L8DrRjIKTnAdB+FMoXiGJs 1Yxw== X-Gm-Message-State: AJaThX5vatne6cliMQtQtFkft3iuLPM92GA+b+DDjv8iC8ljCQZQJUkK t+F86zi2iMN/QxHlo4fzeyktjXAp X-Google-Smtp-Source: AGs4zMbxZEOtIhnMorklW8UjQI6ph3si61UjwL/jwNjdJm4JeCws0+U0BTbdAoQwpkm8u2mGi4Hxog== X-Received: by 10.101.71.134 with SMTP id e6mr9173702pgs.141.1512222497985; Sat, 02 Dec 2017 05:48:17 -0800 (PST) Received: from localhost.au.ibm.com (14-202-194-140.static.tpgi.com.au. [14.202.194.140]) by smtp.googlemail.com with ESMTPSA id n22sm16010376pgd.47.2017.12.02.05.48.15 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 02 Dec 2017 05:48:17 -0800 (PST) From: Balbir Singh To: linuxppc-dev@lists.ozlabs.org Subject: [rfc] powernv/kdump: Fix cases where the kdump kernel can get HMI's Date: Sun, 3 Dec 2017 00:48:08 +1100 Message-Id: <20171202134808.27480-1-bsingharora@gmail.com> X-Mailer: git-send-email 2.13.6 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: npiggin@gmail.com Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Certain HMI's such as malfunction error propagate through all threads/core on the system. If a thread was offline prior to us crashing the system and jumping to the kdump kernel, bad things happen when it wakes up due to an HMI in the kdump kernel. There are several possible ways to solve this problem 1. Put the offline cores in a state such that they are not woken up for machine check and HMI errors. This does not work, since we might need to wake up offline threads occasionally to handle TB errors 2. Ignore HMI errors, setup HMEER to mask HMI errors, but this still leads the window open for any MCEs and masking them for the duration of the dump might be a concern 3. Wake up offline CPUs, as in send them to crash_ipi_callback (not wake them up as in mark them online as seen by the scheduler). kexec does a wake_online_cpus() call, this patch does something similar, but instead sends an IPI and forces them to crash_ipi_callback Care is taken to enable this only for powenv platforms via crash_wake_offline (a global value set at setup time). The crash code sends out IPI's to all CPU's which then move to crash_ipi_callback and kexec_smp_wait(). We don't grab the pt_regs for offline CPU's. Signed-off-by: Balbir Singh --- arch/powerpc/include/asm/kexec.h | 2 ++ arch/powerpc/kernel/crash.c | 18 +++++++++++++----- arch/powerpc/kernel/smp.c | 11 ++++++++--- arch/powerpc/platforms/powernv/smp.c | 23 +++++++++++++++++++---- 4 files changed, 42 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/include/asm/kexec.h b/arch/powerpc/include/asm/kexec.h index 4419d435639a..9dcbfa6bbb91 100644 --- a/arch/powerpc/include/asm/kexec.h +++ b/arch/powerpc/include/asm/kexec.h @@ -73,6 +73,8 @@ extern void kexec_smp_wait(void); /* get and clear naca physid, wait for master to copy new code to 0 */ extern int crashing_cpu; extern void crash_send_ipi(void (*crash_ipi_callback)(struct pt_regs *)); +extern void crash_ipi_callback(struct pt_regs *); +extern int crash_wake_offline; struct kimage; struct pt_regs; diff --git a/arch/powerpc/kernel/crash.c b/arch/powerpc/kernel/crash.c index cbabb5adccd9..7e2ddfa9213e 100644 --- a/arch/powerpc/kernel/crash.c +++ b/arch/powerpc/kernel/crash.c @@ -44,6 +44,14 @@ #define REAL_MODE_TIMEOUT 10000 static int time_to_dump; +/* + * crash_wake_offline should be set to 1 by platforms that intend to wake + * up offline cpus prior to jumping to a kdump kernel. Currently powernv + * sets it to 1, since we want to avoid things from happening when an + * offline CPU wakes up due to something like an HMI (malfunction error), + * which propagates to all threads. + */ +int crash_wake_offline; #define CRASH_HANDLER_MAX 3 /* List of shutdown handles */ @@ -63,17 +71,14 @@ static int handle_fault(struct pt_regs *regs) #ifdef CONFIG_SMP static atomic_t cpus_in_crash; -static void crash_ipi_callback(struct pt_regs *regs) +void crash_ipi_callback(struct pt_regs *regs) { static cpumask_t cpus_state_saved = CPU_MASK_NONE; int cpu = smp_processor_id(); - if (!cpu_online(cpu)) - return; - hard_irq_disable(); - if (!cpumask_test_cpu(cpu, &cpus_state_saved)) { + if (cpu_online(cpu) && !cpumask_test_cpu(cpu, &cpus_state_saved)) { crash_save_cpu(regs, cpu); cpumask_set_cpu(cpu, &cpus_state_saved); } @@ -109,6 +114,9 @@ static void crash_kexec_prepare_cpus(int cpu) printk(KERN_EMERG "Sending IPI to other CPUs\n"); + if (crash_wake_offline) + ncpus = num_present_cpus() - 1; + crash_send_ipi(crash_ipi_callback); smp_wmb(); diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index e0a4c1f82e25..f485db54c2f9 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -429,10 +429,12 @@ static void do_smp_send_nmi_ipi(int cpu) } else { int c; - for_each_online_cpu(c) { + for_each_present_cpu(c) { if (c == raw_smp_processor_id()) continue; - do_message_pass(c, PPC_MSG_NMI_IPI); + if (cpu_online(c) || + (kdump_in_progress() && crash_wake_offline)) + do_message_pass(c, PPC_MSG_NMI_IPI); } } } @@ -485,7 +487,10 @@ int smp_send_nmi_ipi(int cpu, void (*fn)(struct pt_regs *), u64 delay_us) if (cpu < 0) { /* ALL_OTHERS */ - cpumask_copy(&nmi_ipi_pending_mask, cpu_online_mask); + if (kdump_in_progress() && crash_wake_offline) + cpumask_copy(&nmi_ipi_pending_mask, cpu_present_mask); + else + cpumask_copy(&nmi_ipi_pending_mask, cpu_online_mask); cpumask_clear_cpu(me, &nmi_ipi_pending_mask); } else { /* cpumask starts clear */ diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c index ba030669eca1..a8d8f6aaeb11 100644 --- a/arch/powerpc/platforms/powernv/smp.c +++ b/arch/powerpc/platforms/powernv/smp.c @@ -37,6 +37,7 @@ #include #include #include +#include #include "powernv.h" @@ -187,6 +188,14 @@ static void pnv_smp_cpu_kill_self(void) WARN_ON(lazy_irq_pending()); /* + * For kdump kernels, we process the ipi and jump to + * crash_ipi_callback. For more details see the description + * at crash_wake_offline + */ + if (kdump_in_progress()) + crash_ipi_callback(NULL); + + /* * If the SRR1 value indicates that we woke up due to * an external interrupt, then clear the interrupt. * We clear the interrupt before checking for the @@ -324,14 +333,17 @@ static int pnv_cause_nmi_ipi(int cpu) * exactly what semantics Linux wants or the firmware should * provide. */ - for_each_online_cpu(c) { + for_each_present_cpu(c) { if (c == smp_processor_id()) continue; - rc = opal_signal_system_reset( + if (cpu_online(c) || + (kdump_in_progress() && crash_wake_offline)) { + rc = opal_signal_system_reset( get_hard_smp_processor_id(c)); - if (rc != OPAL_SUCCESS) - success = false; + if (rc != OPAL_SUCCESS) + success = false; + } } if (success) return 1; @@ -371,5 +383,8 @@ void __init pnv_smp_init(void) #ifdef CONFIG_HOTPLUG_CPU ppc_md.cpu_die = pnv_smp_cpu_kill_self; +#ifdef CONFIG_KEXEC_CORE + crash_wake_offline = 1; +#endif #endif }