Message ID | 20200326232542.503157-1-leonardo@linux.ibm.com (mailing list archive) |
---|---|
State | Superseded, archived |
Headers | show |
Series | [v2,1/1] ppc/crash: Skip spinlocks during crash | expand |
Context | Check | Description |
---|---|---|
snowpatch_ozlabs/apply_patch | success | Successfully applied on branch powerpc/merge (c6624071c338732402e8c726df6a4074473eaa0e) |
snowpatch_ozlabs/build-ppc64le | success | Build succeeded |
snowpatch_ozlabs/build-ppc64be | success | Build succeeded |
snowpatch_ozlabs/build-ppc64e | fail | build failed! |
snowpatch_ozlabs/build-pmac32 | success | Build succeeded |
snowpatch_ozlabs/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 40 lines checked |
snowpatch_ozlabs/needsstable | success | Patch has no Fixes tags |
Hi Leonardo, Leonardo Bras <leonardo@linux.ibm.com> writes: > During a crash, there is chance that the cpus that handle the NMI IPI > are holding a spin_lock. If this spin_lock is needed by crashing_cpu it > will cause a deadlock. (rtas_lock and printk logbuf_log as of today) Please give us more detail on how those locks are causing you trouble, a stack trace would be good if you have it. > This is a problem if the system has kdump set up, given if it crashes > for any reason kdump may not be saved for crash analysis. > > Skip spinlocks after NMI IPI is sent to all other cpus. We don't want to add overhead to all spinlocks for the life of the system, just to handle this one case. There's already a flag that is set when the system is crashing, "oops_in_progress", maybe we need to use that somewhere to skip a lock or do an early return. cheers > diff --git a/arch/powerpc/include/asm/spinlock.h b/arch/powerpc/include/asm/spinlock.h > index 860228e917dc..a6381d110795 100644 > --- a/arch/powerpc/include/asm/spinlock.h > +++ b/arch/powerpc/include/asm/spinlock.h > @@ -111,6 +111,8 @@ static inline void splpar_spin_yield(arch_spinlock_t *lock) {}; > static inline void splpar_rw_yield(arch_rwlock_t *lock) {}; > #endif > > +extern bool crash_skip_spinlock __read_mostly; > + > static inline bool is_shared_processor(void) > { > #ifdef CONFIG_PPC_SPLPAR > @@ -142,6 +144,8 @@ static inline void arch_spin_lock(arch_spinlock_t *lock) > if (likely(__arch_spin_trylock(lock) == 0)) > break; > do { > + if (unlikely(crash_skip_spinlock)) > + return; > HMT_low(); > if (is_shared_processor()) > splpar_spin_yield(lock); > @@ -161,6 +165,8 @@ void arch_spin_lock_flags(arch_spinlock_t *lock, unsigned long flags) > local_save_flags(flags_dis); > local_irq_restore(flags); > do { > + if (unlikely(crash_skip_spinlock)) > + return; > HMT_low(); > if (is_shared_processor()) > splpar_spin_yield(lock); > diff --git a/arch/powerpc/kexec/crash.c b/arch/powerpc/kexec/crash.c > index d488311efab1..ae081f0f2472 100644 > --- a/arch/powerpc/kexec/crash.c > +++ b/arch/powerpc/kexec/crash.c > @@ -66,6 +66,9 @@ static int handle_fault(struct pt_regs *regs) > > #ifdef CONFIG_SMP > > +bool crash_skip_spinlock; > +EXPORT_SYMBOL(crash_skip_spinlock); > + > static atomic_t cpus_in_crash; > void crash_ipi_callback(struct pt_regs *regs) > { > @@ -129,6 +132,7 @@ static void crash_kexec_prepare_cpus(int cpu) > /* Would it be better to replace the trap vector here? */ > > if (atomic_read(&cpus_in_crash) >= ncpus) { > + crash_skip_spinlock = true; > printk(KERN_EMERG "IPI complete\n"); > return; > } > -- > 2.24.1
Hello Michael, On Fri, 2020-03-27 at 14:50 +1100, Michael Ellerman wrote: > Hi Leonardo, > > Leonardo Bras <leonardo@linux.ibm.com> writes: > > During a crash, there is chance that the cpus that handle the NMI IPI > > are holding a spin_lock. If this spin_lock is needed by crashing_cpu it > > will cause a deadlock. (rtas_lock and printk logbuf_log as of today) > > Please give us more detail on how those locks are causing you trouble, a > stack trace would be good if you have it. Sure, I have hit it in printf and rtas_call, as said before. After crash_send_ipi(), it's tested how many cpus_in_crash are there, and once they hit the total value, it's printed "IPI complete". This printk call itself already got stuck in spin_lock, for example. Here are the stack traces: #0 arch_spin_lock #1 do_raw_spin_lock #2 __raw_spin_lock #3 _raw_spin_lock #4 vprintk_emit #5 vprintk_func #7 crash_kexec_prepare_cpus #8 default_machine_crash_shutdown #9 machine_crash_shutdown #10 __crash_kexec #11 crash_kexec #12 oops_end #0 arch_spin_lock #1 lock_rtas () #2 rtas_call (token=8204, nargs=1, nret=1, outputs=0x0) #3 ics_rtas_mask_real_irq (hw_irq=4100) #4 machine_kexec_mask_interrupts #5 default_machine_crash_shutdown #6 machine_crash_shutdown #7 __crash_kexec #8 crash_kexec #9 oops_end > > This is a problem if the system has kdump set up, given if it crashes > > for any reason kdump may not be saved for crash analysis. > > > > Skip spinlocks after NMI IPI is sent to all other cpus. > > We don't want to add overhead to all spinlocks for the life of the > system, just to handle this one case. I understand. Other than this patch, I would propose doing something uglier, like forcing the said locks to unlocked state when cpus_in_crash hits it's maximum value, before printing "IPI complete". Creating similar functions that don't lock, just for this case, looks like overkill to me. Do you have any other suggestion? > There's already a flag that is set when the system is crashing, > "oops_in_progress", maybe we need to use that somewhere to skip a lock > or do an early return. I think that would not work, because oops_in_progress should be 0 here: oops_end() calls bust_spinlocks(0) before calling crash_kexec(), and bust_spinlocks(0) will decrement oops_in_progress. (just verified, it's 0 before printing "IPI complete"). Thank you the feedback, :) Best regards, Leonardo
Hi Leonardo, Thank you for the patch! Yet something to improve: [auto build test ERROR on tip/locking/core] [also build test ERROR on powerpc/next paulus-powerpc/kvm-ppc-next v5.6 next-20200401] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982] url: https://github.com/0day-ci/linux/commits/Leonardo-Bras/ppc-crash-Skip-spinlocks-during-crash/20200327-105958 base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 8bf6c677ddb9c922423ea3bf494fe7c508bfbb8c config: powerpc-randconfig-a001-20200401 (attached as .config) compiler: powerpc-linux-gcc (GCC) 9.3.0 reproduce: wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree GCC_VERSION=9.3.0 make.cross ARCH=powerpc If you fix the issue, kindly add following tag Reported-by: kbuild test robot <lkp@intel.com> All errors (new ones prefixed by >>): powerpc-linux-ld: arch/powerpc/kernel/traps.o: in function `arch_spin_lock': >> arch/powerpc/include/asm/spinlock.h:147: undefined reference to `crash_skip_spinlock' >> powerpc-linux-ld: arch/powerpc/include/asm/spinlock.h:147: undefined reference to `crash_skip_spinlock' powerpc-linux-ld: arch/powerpc/kernel/rtas.o: in function `arch_spin_lock': >> arch/powerpc/include/asm/spinlock.h:147: undefined reference to `crash_skip_spinlock' >> powerpc-linux-ld: arch/powerpc/include/asm/spinlock.h:147: undefined reference to `crash_skip_spinlock' powerpc-linux-ld: kernel/locking/lockdep.o: in function `arch_spin_lock': >> arch/powerpc/include/asm/spinlock.h:147: undefined reference to `crash_skip_spinlock' powerpc-linux-ld: kernel/locking/lockdep.o:arch/powerpc/include/asm/spinlock.h:147: more undefined references to `crash_skip_spinlock' follow >> pahole: .tmp_vmlinux.btf: No such file or directory powerpc-linux-objdump: '.tmp_vmlinux.btf': No such file powerpc-linux-objdump: '.tmp_vmlinux.btf': No such file powerpc-linux-objcopy: '.tmp_vmlinux.btf': No such file powerpc-linux-objcopy: --change-section-vma .BTF=0x0000000000000000 never used powerpc-linux-objcopy: --change-section-lma .BTF=0x0000000000000000 never used powerpc-linux-objcopy: '.btf.vmlinux.bin': No such file Failed to generate BTF for vmlinux Try to disable CONFIG_DEBUG_INFO_BTF vim +147 arch/powerpc/include/asm/spinlock.h 140 141 static inline void arch_spin_lock(arch_spinlock_t *lock) 142 { 143 while (1) { 144 if (likely(__arch_spin_trylock(lock) == 0)) 145 break; 146 do { > 147 if (unlikely(crash_skip_spinlock)) 148 return; 149 HMT_low(); 150 if (is_shared_processor()) 151 splpar_spin_yield(lock); 152 } while (unlikely(lock->slock != 0)); 153 HMT_medium(); 154 } 155 } 156 --- 0-DAY CI Kernel Test Service, Intel Corporation https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
diff --git a/arch/powerpc/include/asm/spinlock.h b/arch/powerpc/include/asm/spinlock.h index 860228e917dc..a6381d110795 100644 --- a/arch/powerpc/include/asm/spinlock.h +++ b/arch/powerpc/include/asm/spinlock.h @@ -111,6 +111,8 @@ static inline void splpar_spin_yield(arch_spinlock_t *lock) {}; static inline void splpar_rw_yield(arch_rwlock_t *lock) {}; #endif +extern bool crash_skip_spinlock __read_mostly; + static inline bool is_shared_processor(void) { #ifdef CONFIG_PPC_SPLPAR @@ -142,6 +144,8 @@ static inline void arch_spin_lock(arch_spinlock_t *lock) if (likely(__arch_spin_trylock(lock) == 0)) break; do { + if (unlikely(crash_skip_spinlock)) + return; HMT_low(); if (is_shared_processor()) splpar_spin_yield(lock); @@ -161,6 +165,8 @@ void arch_spin_lock_flags(arch_spinlock_t *lock, unsigned long flags) local_save_flags(flags_dis); local_irq_restore(flags); do { + if (unlikely(crash_skip_spinlock)) + return; HMT_low(); if (is_shared_processor()) splpar_spin_yield(lock); diff --git a/arch/powerpc/kexec/crash.c b/arch/powerpc/kexec/crash.c index d488311efab1..ae081f0f2472 100644 --- a/arch/powerpc/kexec/crash.c +++ b/arch/powerpc/kexec/crash.c @@ -66,6 +66,9 @@ static int handle_fault(struct pt_regs *regs) #ifdef CONFIG_SMP +bool crash_skip_spinlock; +EXPORT_SYMBOL(crash_skip_spinlock); + static atomic_t cpus_in_crash; void crash_ipi_callback(struct pt_regs *regs) { @@ -129,6 +132,7 @@ static void crash_kexec_prepare_cpus(int cpu) /* Would it be better to replace the trap vector here? */ if (atomic_read(&cpus_in_crash) >= ncpus) { + crash_skip_spinlock = true; printk(KERN_EMERG "IPI complete\n"); return; }
During a crash, there is chance that the cpus that handle the NMI IPI are holding a spin_lock. If this spin_lock is needed by crashing_cpu it will cause a deadlock. (rtas_lock and printk logbuf_log as of today) This is a problem if the system has kdump set up, given if it crashes for any reason kdump may not be saved for crash analysis. Skip spinlocks after NMI IPI is sent to all other cpus. Signed-off-by: Leonardo Bras <leonardo@linux.ibm.com> --- arch/powerpc/include/asm/spinlock.h | 6 ++++++ arch/powerpc/kexec/crash.c | 4 ++++ 2 files changed, 10 insertions(+)