| Message ID | 4E1543B6.9060800@linux.vnet.ibm.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Headers | show |
In message <4E1543B6.9060800@linux.vnet.ibm.com> you wrote: > Hi , > > Problem Description: > Firmware update using the update_flash -f<filename> results to soft lockup > BUG > FLASH: preparing saved firmware image for flash > FLASH: flash image is 50141296 bytes > FLASH: performing flash and reboot > FLASH: this will take several minutes. Do not power off! > BUG: soft lockup - CPU#1 stuck for 67s! [events/1:36] > > Steps to reproduce: > 1. Check the firmware information on the machine (using ASM or lsmcode) > 2. Update the system firmware with the update_flash command > update_flash -f 01FL350_039_038.img > info: Temporary side will be updated with a newer or > identical image > > Projected Flash Update Results: > Current T Image: FL350_039 > Current P Image: FL350_039 > New T Image: FL350_039 > New P Image: FL350_039 > Flash image ready...rebooting the system... > > Broadcast message from root@abc > (/dev/hvc0) at 5:25 ... > > The system is going down for reboot NOW! > [root@abc /]# Stopping rhsmcertd[ OK ] > Stopping atd: [ OK ] > Stopping cups: [ OK ] > Stopping abrt daemon: [ OK ] > Stopping sshd: [ OK ] > Shutting down postfix: [ OK ] > Stopping rtas_errd (platform error handling) daemon: [ OK ] > Stopping crond: [ OK ] > Stopping automount: [ OK ] > Stopping HAL daemon: [ OK ] > Stopping iprdump: [ OK ] > Killing mdmonitor: [ OK ]] > Stopping system message bus: [ OK ] > Stopping rpcbind: [ OK ] > Stopping auditd: [ OK ] > Shutting down interface eth0: [ OK ] > Shutting down loopback interface: [ OK ] > ip6tables: Flushing firewall rules: [ OK ] > ip6tables: Setting chains to policy ACCEPT: filter [ OK ] > ip6tables: Unloading modules: [ OK ] > iptables: Flushing firewall rules: [ OK ] > iptables: Setting chains to policy ACCEPT: filter [ OK ] > iptables: Unloading modules: [ OK ] > Sending all processes the TERM signal... [ OK ] > Sending all processes the KILL signal... [ OK ] > Saving random seed: [ OK ] > Turning off swap: [ OK ] > Turning off quotas: [ OK ] > Unmounting pipe file systems: [ OK ] > Unmounting file systems: [ OK ] > init: Re-executing /sbin/init > Please stand by while rebooting the system... > Restarting system. > FLASH: preparing saved firmware image for flash > FLASH: flash image is 50141296 bytes > FLASH: performing flash and reboot > FLASH: this will take several minutes. Do not power off! > BUG: soft lockup - CPU#1 stuck for 67s! [events/1:36] > > This is solved by the following patch Can you please explain how it fixes it? Also you need a signed off by. > > --- arch/powerpc/kernel/setup-common.c.orig 2011-07-01 22:41:12.952507971 - 0400 > +++ arch/powerpc/kernel/setup-common.c 2011-07-01 22:48:31.182507915 - 0400 > @@ -109,11 +109,12 @@ void machine_shutdown(void) > void machine_restart(char *cmd) > { > machine_shutdown(); > - if (ppc_md.restart) > - ppc_md.restart(cmd); > #ifdef CONFIG_SMP > - smp_send_stop(); > + smp_send_stop(); Random white space change here. > #endif > + if (ppc_md.restart) > + ppc_md.restart(cmd); > + > printk(KERN_EMERG "System Halted, OK to turn off power\n"); > local_irq_disable(); > while (1) ; > > Thanks > Divya > > _______________________________________________ > Linuxppc-dev mailing list > Linuxppc-dev@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/linuxppc-dev >
On Thursday 07 July 2011 11:42 AM, Michael Neuling wrote: > In message<4E1543B6.9060800@linux.vnet.ibm.com> you wrote: > >> Hi , >> >> Problem Description: >> Firmware update using the update_flash -f<filename> results to soft lockup >> BUG >> FLASH: preparing saved firmware image for flash >> FLASH: flash image is 50141296 bytes >> FLASH: performing flash and reboot >> FLASH: this will take several minutes. Do not power off! >> BUG: soft lockup - CPU#1 stuck for 67s! [events/1:36] >> >> Steps to reproduce: >> 1. Check the firmware information on the machine (using ASM or lsmcode) >> 2. Update the system firmware with the update_flash command >> update_flash -f 01FL350_039_038.img >> info: Temporary side will be updated with a newer or >> identical image >> >> Projected Flash Update Results: >> Current T Image: FL350_039 >> Current P Image: FL350_039 >> New T Image: FL350_039 >> New P Image: FL350_039 >> Flash image ready...rebooting the system... >> >> Broadcast message from root@abc >> (/dev/hvc0) at 5:25 ... >> >> The system is going down for reboot NOW! >> [root@abc /]# Stopping rhsmcertd[ OK ] >> Stopping atd: [ OK ] >> Stopping cups: [ OK ] >> Stopping abrt daemon: [ OK ] >> Stopping sshd: [ OK ] >> Shutting down postfix: [ OK ] >> Stopping rtas_errd (platform error handling) daemon: [ OK ] >> Stopping crond: [ OK ] >> Stopping automount: [ OK ] >> Stopping HAL daemon: [ OK ] >> Stopping iprdump: [ OK ] >> Killing mdmonitor: [ OK ]] >> Stopping system message bus: [ OK ] >> Stopping rpcbind: [ OK ] >> Stopping auditd: [ OK ] >> Shutting down interface eth0: [ OK ] >> Shutting down loopback interface: [ OK ] >> ip6tables: Flushing firewall rules: [ OK ] >> ip6tables: Setting chains to policy ACCEPT: filter [ OK ] >> ip6tables: Unloading modules: [ OK ] >> iptables: Flushing firewall rules: [ OK ] >> iptables: Setting chains to policy ACCEPT: filter [ OK ] >> iptables: Unloading modules: [ OK ] >> Sending all processes the TERM signal... [ OK ] >> Sending all processes the KILL signal... [ OK ] >> Saving random seed: [ OK ] >> Turning off swap: [ OK ] >> Turning off quotas: [ OK ] >> Unmounting pipe file systems: [ OK ] >> Unmounting file systems: [ OK ] >> init: Re-executing /sbin/init >> Please stand by while rebooting the system... >> Restarting system. >> FLASH: preparing saved firmware image for flash >> FLASH: flash image is 50141296 bytes >> FLASH: performing flash and reboot >> FLASH: this will take several minutes. Do not power off! >> BUG: soft lockup - CPU#1 stuck for 67s! [events/1:36] >> >> This is solved by the following patch >> > Can you please explain how it fixes it? > Here goes the explanation for the fix. The flash update is conducted with an RTAS call. The RTAS calls are serialized by lock_rtas() which uses a spin_lock. Now there is rtasd which keeps scanning for the RTAS events generated on the machine. This is performed via workqueue mechanism. The rtas_event_scan() also uses an RTAS call to scan the events, eventually taking the lock_rtas() before it issues the request. The flash update is an operation which takes long time, and hence while we are at it, anyboy else who wants to make an RTAS call will have to wait until the update is completed. Now in this case, the rtas_event_scan() is being kicked in to check for events and it waits a long time on the spin_lock, getting us a SOFT Lockup. Before the rtas firmware update starts, all other CPUs should be stopped. Which means no other CPU should be in lock_rtas(). We do not want other CPUs execute while FW update is in progress and the system will be rebooted anyway after the update. > Also you need a signed off by. > > Signed-off-by: Divya <dipraksh@linux.vnet.ibm.com <mailto:dipraksh@linux.vnet.ibm.com>> > > >> --- arch/powerpc/kernel/setup-common.c.orig 2011-07-01 22:41:12.952507971 - >> > 0400 > >> +++ arch/powerpc/kernel/setup-common.c 2011-07-01 22:48:31.182507915 - >> > 0400 > >> @@ -109,11 +109,12 @@ void machine_shutdown(void) >> void machine_restart(char *cmd) >> { >> machine_shutdown(); >> - if (ppc_md.restart) >> - ppc_md.restart(cmd); >> #ifdef CONFIG_SMP >> - smp_send_stop(); >> + smp_send_stop(); >> > Random white space change here. > > >> #endif >> + if (ppc_md.restart) >> + ppc_md.restart(cmd); >> + >> printk(KERN_EMERG "System Halted, OK to turn off power\n"); >> local_irq_disable(); >> while (1) ; >> >> Thanks >> Divya >> >> _______________________________________________ >> Linuxppc-dev mailing list >> Linuxppc-dev@lists.ozlabs.org >> https://lists.ozlabs.org/listinfo/linuxppc-dev >> >>
In message <4E155F15.1030805@linux.vnet.ibm.com> you wrote: > This is a multi-part message in MIME format. > --===============3790206687486290502== > Content-Type: multipart/alternative; > boundary="------------080309090408040507080807" > > This is a multi-part message in MIME format. > --------------080309090408040507080807 > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > Content-Transfer-Encoding: 7bit > > On Thursday 07 July 2011 11:42 AM, Michael Neuling wrote: > > In message<4E1543B6.9060800@linux.vnet.ibm.com> you wrote: > > > >> Hi , > >> > >> Problem Description: > >> Firmware update using the update_flash -f<filename> results to soft lock up > >> BUG > >> FLASH: preparing saved firmware image for flash > >> FLASH: flash image is 50141296 bytes > >> FLASH: performing flash and reboot > >> FLASH: this will take several minutes. Do not power off! > >> BUG: soft lockup - CPU#1 stuck for 67s! [events/1:36] > >> > >> Steps to reproduce: > >> 1. Check the firmware information on the machine (using ASM or lsmcode) > >> 2. Update the system firmware with the update_flash command > >> update_flash -f 01FL350_039_038.img > >> info: Temporary side will be updated with a newer or > >> identical image > >> > >> Projected Flash Update Results: > >> Current T Image: FL350_039 > >> Current P Image: FL350_039 > >> New T Image: FL350_039 > >> New P Image: FL350_039 > >> Flash image ready...rebooting the system... > >> > >> Broadcast message from root@abc > >> (/dev/hvc0) at 5:25 ... > >> > >> The system is going down for reboot NOW! > >> [root@abc /]# Stopping rhsmcertd[ OK ] > >> Stopping atd: [ OK ] > >> Stopping cups: [ OK ] > >> Stopping abrt daemon: [ OK ] > >> Stopping sshd: [ OK ] > >> Shutting down postfix: [ OK ] > >> Stopping rtas_errd (platform error handling) daemon: [ OK ] > >> Stopping crond: [ OK ] > >> Stopping automount: [ OK ] > >> Stopping HAL daemon: [ OK ] > >> Stopping iprdump: [ OK ] > >> Killing mdmonitor: [ OK ]] > >> Stopping system message bus: [ OK ] > >> Stopping rpcbind: [ OK ] > >> Stopping auditd: [ OK ] > >> Shutting down interface eth0: [ OK ] > >> Shutting down loopback interface: [ OK ] > >> ip6tables: Flushing firewall rules: [ OK ] > >> ip6tables: Setting chains to policy ACCEPT: filter [ OK ] > >> ip6tables: Unloading modules: [ OK ] > >> iptables: Flushing firewall rules: [ OK ] > >> iptables: Setting chains to policy ACCEPT: filter [ OK ] > >> iptables: Unloading modules: [ OK ] > >> Sending all processes the TERM signal... [ OK ] > >> Sending all processes the KILL signal... [ OK ] > >> Saving random seed: [ OK ] > >> Turning off swap: [ OK ] > >> Turning off quotas: [ OK ] > >> Unmounting pipe file systems: [ OK ] > >> Unmounting file systems: [ OK ] > >> init: Re-executing /sbin/init > >> Please stand by while rebooting the system... > >> Restarting system. > >> FLASH: preparing saved firmware image for flash > >> FLASH: flash image is 50141296 bytes > >> FLASH: performing flash and reboot > >> FLASH: this will take several minutes. Do not power off! > >> BUG: soft lockup - CPU#1 stuck for 67s! [events/1:36] > >> > >> This is solved by the following patch > >> > > Can you please explain how it fixes it? > > > The flash update is conducted with an RTAS call. The RTAS calls are > serialized by lock_rtas() which uses a spin_lock. > > Now there is rtasd which keeps scanning for the RTAS events generated > on the machine. This is performed via workqueue mechanism. The > rtas_event_scan() also uses an RTAS call to scan the events, > eventually taking the lock_rtas() before it issues the request. > > The flash update is an operation which takes long time, and hence > while we are at it, anyboy else who wants to make an RTAS call will > have to wait until the update is completed. Now in this case, the > rtas_event_scan() is being kicked in to check for events and it waits > a long time on the spin_lock, getting us a SOFT Lockup. What other RTAS calls are going on at this point? It worries me we are stopping a CPU that's doing RTAS calls. Your solution would seem to be papering over a more serious problem. > Before the rtas firmware update starts, all other CPUs should be > stopped. Which means no other CPU should be in lock_rtas(). We do not > want other CPUs execute while FW update is in progress and the system > will be rebooted anyway after the update. Mikey
--- arch/powerpc/kernel/setup-common.c.orig 2011-07-01 22:41:12.952507971 -0400 +++ arch/powerpc/kernel/setup-common.c 2011-07-01 22:48:31.182507915 -0400 @@ -109,11 +109,12 @@ void machine_shutdown(void) void machine_restart(char *cmd) { machine_shutdown(); - if (ppc_md.restart) - ppc_md.restart(cmd); #ifdef CONFIG_SMP - smp_send_stop(); + smp_send_stop(); #endif + if (ppc_md.restart) + ppc_md.restart(cmd); + printk(KERN_EMERG "System Halted, OK to turn off power\n"); local_irq_disable(); while (1) ;