[{"id":1724094,"web_url":"http://patchwork.ozlabs.org/comment/1724094/","msgid":"<5ca83edd-35a7-68ef-73c8-2f0f0c82a335@linux.vnet.ibm.com>","date":"2017-07-20T07:14:52","subject":"Re: [PATCH v2 2/3] powerpc/powernv: machine check use kernel crash\n\tpath","submitter":{"id":1436,"url":"http://patchwork.ozlabs.org/api/people/1436/","name":"Mahesh J Salgaonkar","email":"mahesh@linux.vnet.ibm.com"},"content":"On 07/19/2017 12:29 PM, Nicholas Piggin wrote:\n> There are quite a few machine check exceptions that can be caused by\n> kernel bugs. To make debugging easier, use the kernel crash path in\n> cases of synchronous machine checks that occur in kernel mode, if that\n> would not result in the machine going straight to panic or crash dump.\n> \n> There is a downside here that die()ing the process in kernel mode can\n> still leave the system unstable. panic_on_oops will always force the\n> system to fail-stop, so systems where that behaviour is important will\n> still do the right thing.\n> \n> As a test, when triggering an i-side 0111b error (ifetch from foreign\n> address) in kernel mode process context on POWER9, the kernel currently\n> dies quickly like this:\n> \n> Severe Machine check interrupt [Not recovered]\n> NIP [ffff000000000000]: 0xffff000000000000\n> Initiator: CPU\n> Error type: Real address [Instruction fetch (foreign)]\n> [ 127.426651616,0] OPAL: Reboot requested due to Platform error.\n> Effective[ 127.426693712,3] OPAL: Reboot requested due to Platform error. address: ffff000000000000\n> opal: Reboot type 1 not supported\n> Kernel panic - not syncing: PowerNV Unrecovered Machine Check\n> CPU: 56 PID: 4425 Comm: syscall Tainted: G M 4.12.0-rc1-13857-ga4700a261072-dirty #35\n> Call Trace:\n> [ 128.017988928,4] IPMI: BUG: Dropping ESEL on the floor due to buggy/mising code in OPAL for this BMCRebooting in 10 seconds..\n> Trying to free IRQ 496 from IRQ context!\n> \n> \n> After this patch, the process is killed and the kernel continues with\n> this message, which gives enough information to identify the offending\n> branch (i.e., with CFAR):\n> \n> Severe Machine check interrupt [Not recovered]\n> NIP [ffff000000000000]: 0xffff000000000000\n> Initiator: CPU\n> Error type: Real address [Instruction fetch (foreign)]\n> Effective address: ffff000000000000\n> Oops: Machine check, sig: 7 [#1]\n> SMP NR_CPUS=2048\n> NUMA\n> PowerNV\n> Modules linked in: iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp tun bridge stp llc kvm_hv kvm iptable_filter binfmt_misc vmx_crypto ip_tables x_tables autofs4 crc32c_vpmsum\n> CPU: 22 PID: 4436 Comm: syscall Tainted: G M 4.12.0-rc1-13857-ga4700a261072-dirty #36\n> task: c000000932300000 task.stack: c000000932380000\n> NIP: ffff000000000000 LR: 00000000217706a4 CTR: ffff000000000000\n> REGS: c00000000fc8fd80 TRAP: 0200 Tainted: G M (4.12.0-rc1-13857-ga4700a261072-dirty)\n> MSR: 90000000001c1003 \n> CR: 24000484 XER: 20000000\n> CFAR: c000000000004c80 DAR: 0000000021770a90 DSISR: 0a000000 SOFTE: 1\n> GPR00: 0000000000001ebe 00007fffce4818b0 0000000021797f00 0000000000000000\n> GPR04: 00007fff8007ac24 0000000044000484 0000000000004000 00007fff801405e8\n> GPR08: 900000000280f033 0000000024000484 0000000000000000 0000000000000030\n> GPR12: 9000000000001003 00007fff801bc370 0000000000000000 0000000000000000\n> GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000\n> GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000\n> GPR24: 0000000000000000 0000000000000000 0000000000000000 0000000000000000\n> GPR28: 00007fff801b0000 0000000000000000 00000000217707a0 00007fffce481918\n> NIP [ffff000000000000] 0xffff000000000000\n> LR [00000000217706a4] 0x217706a4\n> Call Trace:\n> Instruction dump:\n> XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX\n> XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX\n> ---[ end trace 32ae1dabb4f8dae6 ]---\n> \n> Signed-off-by: Nicholas Piggin \n\nReviewed-by: Mahesh Salgaonkar \n\nThanks,\n-Mahesh.\n\n> ---\n> arch/powerpc/include/asm/bug.h | 1 +\n> arch/powerpc/include/asm/fadump.h | 2 ++\n> arch/powerpc/kernel/fadump.c | 9 ++++++++-\n> arch/powerpc/kernel/traps.c | 22 ++++++++++++++++++++++\n> arch/powerpc/platforms/powernv/opal.c | 32 ++++++++++++++++++++++++++------\n> 5 files changed, 59 insertions(+), 7 deletions(-)\n> \n> diff --git a/arch/powerpc/include/asm/bug.h b/arch/powerpc/include/asm/bug.h\n> index 0151af6c2a50..9a918b3ca5ee 100644\n> --- a/arch/powerpc/include/asm/bug.h\n> +++ b/arch/powerpc/include/asm/bug.h\n> @@ -133,6 +133,7 @@ extern int do_page_fault(struct pt_regs *, unsigned long, unsigned long);\n> extern void bad_page_fault(struct pt_regs *, unsigned long, int);\n> extern void _exception(int, struct pt_regs *, int, unsigned long);\n> extern void die(const char *, struct pt_regs *, long);\n> +extern bool die_will_crash(void);\n> \n> #endif /* !__ASSEMBLY__ */\n> \n> diff --git a/arch/powerpc/include/asm/fadump.h b/arch/powerpc/include/asm/fadump.h\n> index ce88bbe1d809..5a23010af600 100644\n> --- a/arch/powerpc/include/asm/fadump.h\n> +++ b/arch/powerpc/include/asm/fadump.h\n> @@ -209,11 +209,13 @@ extern int early_init_dt_scan_fw_dump(unsigned long node,\n> extern int fadump_reserve_mem(void);\n> extern int setup_fadump(void);\n> extern int is_fadump_active(void);\n> +extern int should_fadump_crash(void);\n> extern void crash_fadump(struct pt_regs *, const char *);\n> extern void fadump_cleanup(void);\n> \n> #else\t/* CONFIG_FA_DUMP */\n> static inline int is_fadump_active(void) { return 0; }\n> +static inline int should_fadump_crash(void) { return 0; }\n> static inline void crash_fadump(struct pt_regs *regs, const char *str) { }\n> #endif\n> #endif\n> diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c\n> index da8830e49696..8a3058f5943b 100644\n> --- a/arch/powerpc/kernel/fadump.c\n> +++ b/arch/powerpc/kernel/fadump.c\n> @@ -125,6 +125,13 @@ int is_fadump_boot_memory_area(u64 addr, ulong size)\n> \treturn (addr + size) > RMA_START && addr <= fw_dump.boot_memory_size;\n> }\n> \n> +int should_fadump_crash(void)\n> +{\n> +\tif (!fw_dump.dump_registered || !fw_dump.fadumphdr_addr)\n> +\t\treturn 0;\n> +\treturn 1;\n> +}\n> +\n> int is_fadump_active(void)\n> {\n> \treturn fw_dump.dump_active;\n> @@ -518,7 +525,7 @@ void crash_fadump(struct pt_regs *regs, const char *str)\n> \tstruct fadump_crash_info_header *fdh = NULL;\n> \tint old_cpu, this_cpu;\n> \n> -\tif (!fw_dump.dump_registered || !fw_dump.fadumphdr_addr)\n> +\tif (!should_fadump_crash())\n> \t\treturn;\n> \n> \t/*\n> diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c\n> index 574e949f8db9..2849c4f50324 100644\n> --- a/arch/powerpc/kernel/traps.c\n> +++ b/arch/powerpc/kernel/traps.c\n> @@ -114,6 +114,28 @@ static void pmac_backlight_unblank(void)\n> static inline void pmac_backlight_unblank(void) { }\n> #endif\n> \n> +/*\n> + * If oops/die is expected to crash the machine, return true here.\n> + *\n> + * This should not be expected to be 100% accurate, there may be\n> + * notifiers registered or other unexpected conditions that may bring\n> + * down the kernel. Or if the current process in the kernel is holding\n> + * locks or has other critical state, the kernel may become effectively\n> + * unusable anyway.\n> + */\n> +bool die_will_crash(void)\n> +{\n> +\tif (should_fadump_crash())\n> +\t\treturn true;\n> +\tif (kexec_should_crash(current))\n> +\t\treturn true;\n> +\tif (in_interrupt() || panic_on_oops ||\n> +\t\t\t!current->pid || is_global_init(current))\n> +\t\treturn true;\n> +\n> +\treturn false;\n> +}\n> +\n> static arch_spinlock_t die_lock = __ARCH_SPIN_LOCK_UNLOCKED;\n> static int die_owner = -1;\n> static unsigned int die_nest_count;\n> diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c\n> index 96436d129684..140350aacca5 100644\n> --- a/arch/powerpc/platforms/powernv/opal.c\n> +++ b/arch/powerpc/platforms/powernv/opal.c\n> @@ -34,6 +34,7 @@\n> #include \n> #include \n> #include \n> +#include \n> \n> #include \"powernv.h\"\n> \n> @@ -426,17 +427,36 @@ static int opal_recover_mce(struct pt_regs *regs,\n> \t\t/* Fatal machine check */\n> \t\tpr_err(\"Machine check interrupt is fatal\\n\");\n> \t\trecovered = 0;\n> -\t} else if ((evt->severity == MCE_SEV_ERROR_SYNC) &&\n> -\t\t\t(user_mode(regs) && !is_global_init(current))) {\n> +\t}\n> +\n> +\tif (!recovered && evt->severity == MCE_SEV_ERROR_SYNC) {\n> \t\t/*\n> -\t\t * For now, kill the task if we have received exception when\n> -\t\t * in userspace.\n> +\t\t * Try to kill processes if we get a synchronous machine check\n> +\t\t * (e.g., one caused by execution of this instruction). This\n> +\t\t * will devolve into a panic if we try to kill init or are in\n> +\t\t * an interrupt etc.\n> \t\t *\n> \t\t * TODO: Queue up this address for hwpoisioning later.\n> +\t\t * TODO: This is not quite right for d-side machine\n> +\t\t * checks ->nip is not necessarily the important\n> +\t\t * address.\n> \t\t */\n> -\t\t_exception(SIGBUS, regs, BUS_MCEERR_AR, regs->nip);\n> -\t\trecovered = 1;\n> +\t\tif ((user_mode(regs))) {\n> +\t\t\t_exception(SIGBUS, regs, BUS_MCEERR_AR, regs->nip);\n> +\t\t\trecovered = 1;\n> +\t\t} else if (die_will_crash()) {\n> +\t\t\t/*\n> +\t\t\t * die() would kill the kernel, so better to go via\n> +\t\t\t * the platform reboot code that will log the\n> +\t\t\t * machine check.\n> +\t\t\t */\n> +\t\t\trecovered = 0;\n> +\t\t} else {\n> +\t\t\tdie(\"Machine check\", regs, SIGBUS);\n> +\t\t\trecovered = 1;\n> +\t\t}\n> \t}\n> +\n> \treturn recovered;\n> }\n>","headers":{"Return-Path":"","X-Original-To":["patchwork-incoming@ozlabs.org","linuxppc-dev@lists.ozlabs.org"],"Delivered-To":["patchwork-incoming@ozlabs.org","linuxppc-dev@lists.ozlabs.org"],"Received":["from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68])\n\t(using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xClYr4Xp0z9t1G\n\tfor ;\n\tThu, 20 Jul 2017 17:16:16 +1000 (AEST)","from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3])\n\tby lists.ozlabs.org (Postfix) with ESMTP id 3xClYr3ZyCzDrG0\n\tfor ;\n\tThu, 20 Jul 2017 17:16:16 +1000 (AEST)","from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com\n\t[148.163.158.5])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256\n\tbits)) (No client certificate requested)\n\tby lists.ozlabs.org (Postfix) with ESMTPS id 3xClXR6RlFzDrDR\n\tfor ;\n\tThu, 20 Jul 2017 17:15:03 +1000 (AEST)","from pps.filterd (m0098419.ppops.net [127.0.0.1])\n\tby mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id\n\tv6K7DWVx107324\n\tfor ; Thu, 20 Jul 2017 03:15:01 -0400","from e23smtp05.au.ibm.com (e23smtp05.au.ibm.com [202.81.31.147])\n\tby mx0b-001b2d01.pphosted.com with ESMTP id 2btpg4b9vs-1\n\t(version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT)\n\tfor ; Thu, 20 Jul 2017 03:15:00 -0400","from localhost\n\tby e23smtp05.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use\n\tOnly! Violators will be prosecuted\n\tfor from ; \n\tThu, 20 Jul 2017 17:14:57 +1000","from d23relay08.au.ibm.com (202.81.31.227)\n\tby e23smtp05.au.ibm.com (202.81.31.211) with IBM ESMTP SMTP Gateway:\n\tAuthorized Use Only! Violators will be prosecuted; \n\tThu, 20 Jul 2017 17:14:55 +1000","from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139])\n\tby d23relay08.au.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id\n\tv6K7Eso024379418\n\tfor ; Thu, 20 Jul 2017 17:14:54 +1000","from d23av04.au.ibm.com (localhost [127.0.0.1])\n\tby d23av04.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id\n\tv6K7EqfE032123\n\tfor ; Thu, 20 Jul 2017 17:14:53 +1000","from [9.195.46.51] ([9.195.46.51])\n\tby d23av04.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id\n\tv6K7EoV6032057; Thu, 20 Jul 2017 17:14:51 +1000"],"Subject":"Re: [PATCH v2 2/3] powerpc/powernv: machine check use kernel crash\n\tpath","To":"Nicholas Piggin , linuxppc-dev@lists.ozlabs.org","References":"<20170719065912.19183-1-npiggin@gmail.com>\n\t<20170719065912.19183-3-npiggin@gmail.com>","From":"Mahesh Jagannath Salgaonkar ","Date":"Thu, 20 Jul 2017 12:44:52 +0530","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.2.1","MIME-Version":"1.0","In-Reply-To":"<20170719065912.19183-3-npiggin@gmail.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-MW","Content-Transfer-Encoding":"7bit","X-TM-AS-MML":"disable","x-cbid":"17072007-0016-0000-0000-0000025E5FD8","X-IBM-AV-DETECTION":"SAVI=unused REMOTE=unused XFE=unused","x-cbparentid":"17072007-0017-0000-0000-000006DF317E","Message-Id":"<5ca83edd-35a7-68ef-73c8-2f0f0c82a335@linux.vnet.ibm.com>","X-Proofpoint-Virus-Version":"vendor=fsecure engine=2.50.10432:, ,\n\tdefinitions=2017-07-20_04:, , signatures=0","X-Proofpoint-Spam-Details":"rule=outbound_notspam policy=outbound score=0\n\tspamscore=0 suspectscore=2\n\tmalwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam\n\tadjust=0 reason=mlx scancount=1 engine=8.0.1-1706020000\n\tdefinitions=main-1707200117","X-BeenThere":"linuxppc-dev@lists.ozlabs.org","X-Mailman-Version":"2.1.23","Precedence":"list","List-Id":"Linux on PowerPC Developers Mail List\n\t","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Errors-To":"linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org","Sender":"\"Linuxppc-dev\"\n\t"}}]