| Message ID | b39003738a320a1e5e9bd05c668c6979140a8551.1528141814.git.joseph.salisbury@canonical.com |
|---|---|
| State | New |
| Headers | show |
| Series | powerpc/mce: Fix a bug where mce loops on memory UE. | expand |
On 06/05/18 09:21, Joseph Salisbury wrote: > From: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com> > > BugLink: http://bugs.launchpad.net/bugs/1774964 > > The current code extracts the physical address for UE errors and then > hooks it up into memory failure infrastructure. On successful > extraction of physical address it wrongly sets "handled = 1" which > means this UE error has been recovered. Since MCE handler gets return > value as handled = 1, it assumes that error has been recovered and > goes back to same NIP. This causes MCE interrupt again and again in a > loop leading to hard lockup. > > Also, initialize phys_addr to ULONG_MAX so that we don't end up > queuing undesired page to hwpoison. > > Without this patch we see: > Severe Machine check interrupt [Recovered] > NIP: [000000001002588c] PID: 7109 Comm: find > Initiator: CPU > Error type: UE [Load/Store] > Effective address: 00007fffd2755940 > Physical address: 000020181a080000 > ... > Severe Machine check interrupt [Recovered] > NIP: [000000001002588c] PID: 7109 Comm: find > Initiator: CPU > Error type: UE [Load/Store] > Effective address: 00007fffd2755940 > Physical address: 000020181a080000 > Severe Machine check interrupt [Recovered] > NIP: [000000001002588c] PID: 7109 Comm: find > Initiator: CPU > Error type: UE [Load/Store] > Effective address: 00007fffd2755940 > Physical address: 000020181a080000 > Memory failure: 0x20181a08: recovery action for dirty LRU page: Recovered > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > ... > Watchdog CPU:38 Hard LOCKUP > > After this patch we see: > > Severe Machine check interrupt [Not recovered] > NIP: [00007fffaae585f4] PID: 7168 Comm: find > Initiator: CPU > Error type: UE [Load/Store] > Effective address: 00007fffaafe28ac > Physical address: 00002017c0bd0000 > find[7168]: unhandled signal 7 at 00007fffaae585f4 nip 00007fffaae585f4 lr 00007fffaae585e0 code 4 > Memory failure: 0x2017c0bd: recovery action for dirty LRU page: Recovered > > Fixes: 01eaac2b0591 ("powerpc/mce: Hookup ierror (instruction) UE errors") > Fixes: ba41e1e1ccb9 ("powerpc/mce: Hookup derror (load/store) UE errors") > Cc: stable@vger.kernel.org # v4.15+ > Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com> > Signed-off-by: Balbir Singh <bsingharora@gmail.com> > Reviewed-by: Balbir Singh <bsingharora@gmail.com> > Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> > (cherry picked from commit 75ecfb49516c53da00c57b9efe48fa3f5504a791) > Signed-off-by: Joseph Salisbury <joseph.salisbury@canonical.com> Acked-by: Kleber Sacilotto de Souza <kleber.souza@canonical.com> > --- > arch/powerpc/kernel/mce_power.c | 7 ++----- > 1 file changed, 2 insertions(+), 5 deletions(-) > > diff --git a/arch/powerpc/kernel/mce_power.c b/arch/powerpc/kernel/mce_power.c > index 644f704..aa15142 100644 > --- a/arch/powerpc/kernel/mce_power.c > +++ b/arch/powerpc/kernel/mce_power.c > @@ -552,7 +552,6 @@ static int mce_handle_ierror(struct pt_regs *regs, > if (pfn != ULONG_MAX) { > *phys_addr = > (pfn << PAGE_SHIFT); > - handled = 1; > } > } > } > @@ -643,9 +642,7 @@ static int mce_handle_derror(struct pt_regs *regs, > * kernel/exception-64s.h > */ > if (get_paca()->in_mce < MAX_MCE_DEPTH) > - if (!mce_find_instr_ea_and_pfn(regs, addr, > - phys_addr)) > - handled = 1; > + mce_find_instr_ea_and_pfn(regs, addr, phys_addr); > } > found = 1; > } > @@ -683,7 +680,7 @@ static long mce_handle_error(struct pt_regs *regs, > const struct mce_ierror_table itable[]) > { > struct mce_error_info mce_err = { 0 }; > - uint64_t addr, phys_addr; > + uint64_t addr, phys_addr = ULONG_MAX; > uint64_t srr1 = regs->msr; > long handled; > >
On 05.06.2018 09:21, Joseph Salisbury wrote: > From: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com> > > BugLink: http://bugs.launchpad.net/bugs/1774964 > > The current code extracts the physical address for UE errors and then > hooks it up into memory failure infrastructure. On successful > extraction of physical address it wrongly sets "handled = 1" which > means this UE error has been recovered. Since MCE handler gets return > value as handled = 1, it assumes that error has been recovered and > goes back to same NIP. This causes MCE interrupt again and again in a > loop leading to hard lockup. > > Also, initialize phys_addr to ULONG_MAX so that we don't end up > queuing undesired page to hwpoison. > > Without this patch we see: > Severe Machine check interrupt [Recovered] > NIP: [000000001002588c] PID: 7109 Comm: find > Initiator: CPU > Error type: UE [Load/Store] > Effective address: 00007fffd2755940 > Physical address: 000020181a080000 > ... > Severe Machine check interrupt [Recovered] > NIP: [000000001002588c] PID: 7109 Comm: find > Initiator: CPU > Error type: UE [Load/Store] > Effective address: 00007fffd2755940 > Physical address: 000020181a080000 > Severe Machine check interrupt [Recovered] > NIP: [000000001002588c] PID: 7109 Comm: find > Initiator: CPU > Error type: UE [Load/Store] > Effective address: 00007fffd2755940 > Physical address: 000020181a080000 > Memory failure: 0x20181a08: recovery action for dirty LRU page: Recovered > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > Memory failure: 0x20181a08: already hardware poisoned > ... > Watchdog CPU:38 Hard LOCKUP > > After this patch we see: > > Severe Machine check interrupt [Not recovered] > NIP: [00007fffaae585f4] PID: 7168 Comm: find > Initiator: CPU > Error type: UE [Load/Store] > Effective address: 00007fffaafe28ac > Physical address: 00002017c0bd0000 > find[7168]: unhandled signal 7 at 00007fffaae585f4 nip 00007fffaae585f4 lr 00007fffaae585e0 code 4 > Memory failure: 0x2017c0bd: recovery action for dirty LRU page: Recovered > > Fixes: 01eaac2b0591 ("powerpc/mce: Hookup ierror (instruction) UE errors") > Fixes: ba41e1e1ccb9 ("powerpc/mce: Hookup derror (load/store) UE errors") > Cc: stable@vger.kernel.org # v4.15+ > Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com> > Signed-off-by: Balbir Singh <bsingharora@gmail.com> > Reviewed-by: Balbir Singh <bsingharora@gmail.com> > Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> > (cherry picked from commit 75ecfb49516c53da00c57b9efe48fa3f5504a791) > Signed-off-by: Joseph Salisbury <joseph.salisbury@canonical.com> Acked-by: Stefan Bader <stefan.bader@canonical.com> > --- > arch/powerpc/kernel/mce_power.c | 7 ++----- > 1 file changed, 2 insertions(+), 5 deletions(-) > > diff --git a/arch/powerpc/kernel/mce_power.c b/arch/powerpc/kernel/mce_power.c > index 644f704..aa15142 100644 > --- a/arch/powerpc/kernel/mce_power.c > +++ b/arch/powerpc/kernel/mce_power.c > @@ -552,7 +552,6 @@ static int mce_handle_ierror(struct pt_regs *regs, > if (pfn != ULONG_MAX) { > *phys_addr = > (pfn << PAGE_SHIFT); > - handled = 1; > } > } > } > @@ -643,9 +642,7 @@ static int mce_handle_derror(struct pt_regs *regs, > * kernel/exception-64s.h > */ > if (get_paca()->in_mce < MAX_MCE_DEPTH) > - if (!mce_find_instr_ea_and_pfn(regs, addr, > - phys_addr)) > - handled = 1; > + mce_find_instr_ea_and_pfn(regs, addr, phys_addr); > } > found = 1; > } > @@ -683,7 +680,7 @@ static long mce_handle_error(struct pt_regs *regs, > const struct mce_ierror_table itable[]) > { > struct mce_error_info mce_err = { 0 }; > - uint64_t addr, phys_addr; > + uint64_t addr, phys_addr = ULONG_MAX; > uint64_t srr1 = regs->msr; > long handled; > >
diff --git a/arch/powerpc/kernel/mce_power.c b/arch/powerpc/kernel/mce_power.c index 644f704..aa15142 100644 --- a/arch/powerpc/kernel/mce_power.c +++ b/arch/powerpc/kernel/mce_power.c @@ -552,7 +552,6 @@ static int mce_handle_ierror(struct pt_regs *regs, if (pfn != ULONG_MAX) { *phys_addr = (pfn << PAGE_SHIFT); - handled = 1; } } } @@ -643,9 +642,7 @@ static int mce_handle_derror(struct pt_regs *regs, * kernel/exception-64s.h */ if (get_paca()->in_mce < MAX_MCE_DEPTH) - if (!mce_find_instr_ea_and_pfn(regs, addr, - phys_addr)) - handled = 1; + mce_find_instr_ea_and_pfn(regs, addr, phys_addr); } found = 1; } @@ -683,7 +680,7 @@ static long mce_handle_error(struct pt_regs *regs, const struct mce_ierror_table itable[]) { struct mce_error_info mce_err = { 0 }; - uint64_t addr, phys_addr; + uint64_t addr, phys_addr = ULONG_MAX; uint64_t srr1 = regs->msr; long handled;