diff mbox

Sun fire V890 UltraSparc server kernel panic, offertoassist with fixing it and future ports

Message ID 20091122.190339.232644560.davem@davemloft.net
State RFC
Delegated to: David Miller
Headers show

Commit Message

David Miller Nov. 23, 2009, 3:03 a.m. UTC 
From: Josip Rodin <joy@entuzijast.net>
Date: Sun, 22 Nov 2009 23:06:21 +0100

>> [   82.663939] Kernel unaligned access at TPC[46d24c]
>> notifier_chain_register+0x10/0x38
>> [   82.671606] Unable to handle kernel paging request in mna handler<1>
>> at virtual address e00000a00b000046

That address is not only unaligned, it's completely out of range
for valid kernel virtual addresses.

notifier_chain_register() and their ilk are pretty simply and
take constant kernel symbol addresses rather than, for example,
dynamically allocated memory or something else more easily
corrupted.  So this is even more mysterious.

Backtrace is:

>> [   82.759812] RPC: <atomic_notifier_chain_register+0x1c/0x44>
 ...
>> [   82.800022] I7: <kernel_init+0x38/0x204>
 ...
>> [   82.803917] Caller[0000000000782128]: kernel_init+0x38/0x204
>> [   82.809549] Caller[0000000000426df8]: kernel_thread+0x38/0x48
>> [   82.815268] Caller[000000000067bae0]: rest_init+0x18/0x60

And I can't see anything that registers notifiers from kernel_init().

I suspect the real caller of atomic_notifier_chain_register() is
being optimized out by tail-call optimizations so we can't see
exactly what it is.

Can you build a test kernel with something like the following
patch applied and reproduce?  Thanks.

--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Aaron Scheiner Nov. 23, 2009, 9:16 a.m. UTC | #1 
I don't know who's going to do this, but if it's me does anyone have a
guide lying around ? :P

...or some useful links.

Aaron

On Sun, 2009-11-22 at 19:03 -0800, David Miller wrote:
> From: Josip Rodin <joy@entuzijast.net>
> Date: Sun, 22 Nov 2009 23:06:21 +0100
> 
> >> [   82.663939] Kernel unaligned access at TPC[46d24c]
> >> notifier_chain_register+0x10/0x38
> >> [   82.671606] Unable to handle kernel paging request in mna handler<1>
> >> at virtual address e00000a00b000046
> 
> That address is not only unaligned, it's completely out of range
> for valid kernel virtual addresses.
> 
> notifier_chain_register() and their ilk are pretty simply and
> take constant kernel symbol addresses rather than, for example,
> dynamically allocated memory or something else more easily
> corrupted.  So this is even more mysterious.
> 
> Backtrace is:
> 
> >> [   82.759812] RPC: <atomic_notifier_chain_register+0x1c/0x44>
>  ...
> >> [   82.800022] I7: <kernel_init+0x38/0x204>
>  ...
> >> [   82.803917] Caller[0000000000782128]: kernel_init+0x38/0x204
> >> [   82.809549] Caller[0000000000426df8]: kernel_thread+0x38/0x48
> >> [   82.815268] Caller[000000000067bae0]: rest_init+0x18/0x60
> 
> And I can't see anything that registers notifiers from kernel_init().
> 
> I suspect the real caller of atomic_notifier_chain_register() is
> being optimized out by tail-call optimizations so we can't see
> exactly what it is.
> 
> Can you build a test kernel with something like the following
> patch applied and reproduce?  Thanks.
> 
> diff --git a/Makefile b/Makefile
> index aa3e13a..d953cd3 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -537,6 +537,8 @@ ifndef CONFIG_CC_STACKPROTECTOR
>  KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
>  endif
>  
> +KBUILD_CFLAGS	+= -fno-optimize-sibling-calls
> +
>  ifdef CONFIG_FRAME_POINTER
>  KBUILD_CFLAGS	+= -fno-omit-frame-pointer -fno-optimize-sibling-calls
>  else
> 
> 


--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Josip Rodin Nov. 23, 2009, 10:34 p.m. UTC | #2 
On Mon, Nov 23, 2009 at 11:16:59AM +0200, Aaron Scheiner wrote:
> I don't know who's going to do this, but if it's me does anyone have a
> guide lying around ? :P
> 
> ...or some useful links.

Yeah, the problem is, if you had just a single working Linux version on the
machine, you could try these things on your own; but since you don't, you're
depending on either cross-compiling or someone else compiling the kernel,
*AND* building it into a bootable image.

Maybe if you could plug the machine into a network with another machine
which could serve the boot image over PXE or something...
Aaron Scheiner Nov. 23, 2009, 10:52 p.m. UTC | #3 
Hey Joy

The machine is currently connected to my home LAN. I powered it up from
a remote location today via SSH -> Telnet (RSC) and executed some
commands on the Solaris installation... I think that's pretty cool :P . 

My home network has DHCP server on it (Smoothwall) and two other
machines running Ubuntu. One of the Ubuntu machines has been configured
as a TFTP server (used it to hack a Mikrotik router a while back). In
other words, I should(?) have the environment to boot this Sun machine
via the network.

As for the kernel, I found a guide today that should put me on the right
track : http://wiki.debian.org/DebianInstaller/Modify/CustomKernel .

I'll work on it a little later today :) .

On Mon, 2009-11-23 at 23:34 +0100, Josip Rodin wrote:
> On Mon, Nov 23, 2009 at 11:16:59AM +0200, Aaron Scheiner wrote:
> > I don't know who's going to do this, but if it's me does anyone have a
> > guide lying around ? :P
> > 
> > ...or some useful links.
> 
> Yeah, the problem is, if you had just a single working Linux version on the
> machine, you could try these things on your own; but since you don't, you're
> depending on either cross-compiling or someone else compiling the kernel,
> *AND* building it into a bootable image.
> 
> Maybe if you could plug the machine into a network with another machine
> which could serve the boot image over PXE or something...
> 


--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Josip Rodin Nov. 23, 2009, 10:56 p.m. UTC | #4 
On Tue, Nov 24, 2009 at 12:52:21AM +0200, Aaron Scheiner wrote:
> My home network has DHCP server on it (Smoothwall) and two other
> machines running Ubuntu. One of the Ubuntu machines has been configured
> as a TFTP server (used it to hack a Mikrotik router a while back). In
> other words, I should(?) have the environment to boot this Sun machine
> via the network.
> 
> As for the kernel, I found a guide today that should put me on the right
> track : http://wiki.debian.org/DebianInstaller/Modify/CustomKernel .
> 
> I'll work on it a little later today :) .

While we're at it, you mentioned earlier that you tried Ubuntu Dapper (6.06)
with 2.6.15 which also failed miserably - Ubuntu shipped a few more sparc
releases other than that one, so you may get lucky with another one. A quick
Google tells me that their 7.04 and 7.10 releases still had sparc, so try
and find installer images from those.
Josip Rodin Nov. 23, 2009, 11:10 p.m. UTC | #5 
On Mon, Nov 23, 2009 at 11:56:23PM +0100, Josip Rodin wrote:
> On Tue, Nov 24, 2009 at 12:52:21AM +0200, Aaron Scheiner wrote:
> > My home network has DHCP server on it (Smoothwall) and two other
> > machines running Ubuntu. One of the Ubuntu machines has been configured
> > as a TFTP server (used it to hack a Mikrotik router a while back). In
> > other words, I should(?) have the environment to boot this Sun machine
> > via the network.
> > 
> > As for the kernel, I found a guide today that should put me on the right
> > track : http://wiki.debian.org/DebianInstaller/Modify/CustomKernel .
> > 
> > I'll work on it a little later today :) .
> 
> While we're at it, you mentioned earlier that you tried Ubuntu Dapper (6.06)
> with 2.6.15 which also failed miserably - Ubuntu shipped a few more sparc
> releases other than that one, so you may get lucky with another one. A quick
> Google tells me that their 7.04 and 7.10 releases still had sparc, so try
> and find installer images from those.

Oddly enough, once I actually figured out the right place to look on the
web site, I could also easily find newer ones:
http://cdimage.ubuntu.com/ports/releases/hardy/release/ubuntu-8.04.1-server-sparc.iso
http://cdimage.ubuntu.com/ports/releases/intrepid/release/ubuntu-8.10-server-sparc.iso
http://cdimage.ubuntu.com/ports/releases/jaunty/release/ubuntu-9.04-server-sparc.iso
http://cdimage.ubuntu.com/ports/releases/karmic/release/ubuntu-9.10-server-sparc.iso
Aaron Scheiner Nov. 26, 2009, 8:26 a.m. UTC | #6 
Booting both Ubuntu 9.10 and 8.10 results in this error :
"
boot: 
Allocated 64 Megs of memory at 0x40000000 for kernel
Loaded kernel version 2.6.25
Loading initial ramdisk (6178206 bytes at 0xA1FF000000 phys, 0x40C00000
virt)...
ERROR: Last Trap: Fast Data Access MMU Miss

{2} ok 
"

Interestingly, the machine returns to the boot prom instead of just
dying altogether.

Hope that helps :)


Aaron

On Tue, 2009-11-24 at 00:10 +0100, Josip Rodin wrote:
> On Mon, Nov 23, 2009 at 11:56:23PM +0100, Josip Rodin wrote:
> > On Tue, Nov 24, 2009 at 12:52:21AM +0200, Aaron Scheiner wrote:
> > > My home network has DHCP server on it (Smoothwall) and two other
> > > machines running Ubuntu. One of the Ubuntu machines has been configured
> > > as a TFTP server (used it to hack a Mikrotik router a while back). In
> > > other words, I should(?) have the environment to boot this Sun machine
> > > via the network.
> > > 
> > > As for the kernel, I found a guide today that should put me on the right
> > > track : http://wiki.debian.org/DebianInstaller/Modify/CustomKernel .
> > > 
> > > I'll work on it a little later today :) .
> > 
> > While we're at it, you mentioned earlier that you tried Ubuntu Dapper (6.06)
> > with 2.6.15 which also failed miserably - Ubuntu shipped a few more sparc
> > releases other than that one, so you may get lucky with another one. A quick
> > Google tells me that their 7.04 and 7.10 releases still had sparc, so try
> > and find installer images from those.
> 
> Oddly enough, once I actually figured out the right place to look on the
> web site, I could also easily find newer ones:
> http://cdimage.ubuntu.com/ports/releases/hardy/release/ubuntu-8.04.1-server-sparc.iso
> http://cdimage.ubuntu.com/ports/releases/intrepid/release/ubuntu-8.10-server-sparc.iso
> http://cdimage.ubuntu.com/ports/releases/jaunty/release/ubuntu-9.04-server-sparc.iso
> http://cdimage.ubuntu.com/ports/releases/karmic/release/ubuntu-9.10-server-sparc.iso
> 


--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Aaron Scheiner Dec. 23, 2009, 5:53 p.m. UTC | #7 
I just thought I'd let you know that OpenBSD's guys have managed to get
OpenBSD running on this machine.

Aaron

On Thu, 2009-11-26 at 10:26 +0200, Aaron Scheiner wrote:
> Booting both Ubuntu 9.10 and 8.10 results in this error :
> "
> boot: 
> Allocated 64 Megs of memory at 0x40000000 for kernel
> Loaded kernel version 2.6.25
> Loading initial ramdisk (6178206 bytes at 0xA1FF000000 phys, 0x40C00000
> virt)...
> ERROR: Last Trap: Fast Data Access MMU Miss
> 
> {2} ok 
> "
> 
> Interestingly, the machine returns to the boot prom instead of just
> dying altogether.
> 
> Hope that helps :)
> 
> 
> Aaron
> 
> On Tue, 2009-11-24 at 00:10 +0100, Josip Rodin wrote:
> > On Mon, Nov 23, 2009 at 11:56:23PM +0100, Josip Rodin wrote:
> > > On Tue, Nov 24, 2009 at 12:52:21AM +0200, Aaron Scheiner wrote:
> > > > My home network has DHCP server on it (Smoothwall) and two other
> > > > machines running Ubuntu. One of the Ubuntu machines has been configured
> > > > as a TFTP server (used it to hack a Mikrotik router a while back). In
> > > > other words, I should(?) have the environment to boot this Sun machine
> > > > via the network.
> > > > 
> > > > As for the kernel, I found a guide today that should put me on the right
> > > > track : http://wiki.debian.org/DebianInstaller/Modify/CustomKernel .
> > > > 
> > > > I'll work on it a little later today :) .
> > > 
> > > While we're at it, you mentioned earlier that you tried Ubuntu Dapper (6.06)
> > > with 2.6.15 which also failed miserably - Ubuntu shipped a few more sparc
> > > releases other than that one, so you may get lucky with another one. A quick
> > > Google tells me that their 7.04 and 7.10 releases still had sparc, so try
> > > and find installer images from those.
> > 
> > Oddly enough, once I actually figured out the right place to look on the
> > web site, I could also easily find newer ones:
> > http://cdimage.ubuntu.com/ports/releases/hardy/release/ubuntu-8.04.1-server-sparc.iso
> > http://cdimage.ubuntu.com/ports/releases/intrepid/release/ubuntu-8.10-server-sparc.iso
> > http://cdimage.ubuntu.com/ports/releases/jaunty/release/ubuntu-9.04-server-sparc.iso
> > http://cdimage.ubuntu.com/ports/releases/karmic/release/ubuntu-9.10-server-sparc.iso
> > 
> 
> 
> 


--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/Makefile b/Makefile
index aa3e13a..d953cd3 100644
--- a/Makefile
+++ b/Makefile
@@ -537,6 +537,8 @@  ifndef CONFIG_CC_STACKPROTECTOR
 KBUILD_CFLAGS += $(call cc-option, -fno-stack-protector)
 endif
 
+KBUILD_CFLAGS	+= -fno-optimize-sibling-calls
+
 ifdef CONFIG_FRAME_POINTER
 KBUILD_CFLAGS	+= -fno-omit-frame-pointer -fno-optimize-sibling-calls
 else