WARNING: at mm/slub.c:3357, kernel BUG at mm/slub.c:3413

Message ID	1321866988.2552.10.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC
State	Not Applicable, archived
Delegated to:	David Miller
Headers	show Return-Path: <netdev-owner@vger.kernel.org> Message-ID: <1321866988.2552.10.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> Subject: Re: WARNING: at mm/slub.c:3357, kernel BUG at mm/slub.c:3413 From: Eric Dumazet <eric.dumazet@gmail.com> To: Markus Trippelsdorf <markus@trippelsdorf.de> Cc: "Alex,Shi" <alex.shi@intel.com>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>, Christoph Lameter <cl@linux-foundation.org>, Pekka Enberg <penberg@kernel.org>, Matt Mackall <mpm@selenic.com>, "netdev@vger.kernel.org" <netdev@vger.kernel.org> Date: Mon, 21 Nov 2011 10:16:28 +0100 In-Reply-To: <20111121082445.GD1625@x4.trippels.de> References: <20111118072519.GA1615@x4.trippels.de> <20111118075521.GB1615@x4.trippels.de> <1321605837.30341.551.camel@debian> <20111118085436.GC1615@x4.trippels.de> <20111118120201.GA1642@x4.trippels.de> <1321836285.30341.554.camel@debian> <20111121080554.GB1625@x4.trippels.de> <20111121082445.GD1625@x4.trippels.de> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Mime-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk

Eric Dumazet Nov. 21, 2011, 9:16 a.m. UTC

Le lundi 21 novembre 2011 à 09:24 +0100, Markus Trippelsdorf a écrit :
> On 2011.11.21 at 09:05 +0100, Markus Trippelsdorf wrote:
> > On 2011.11.21 at 08:44 +0800, Alex,Shi wrote:
> > > On Fri, 2011-11-18 at 20:02 +0800, Markus Trippelsdorf wrote:
> > > > On 2011.11.18 at 09:54 +0100, Markus Trippelsdorf wrote:
> > > > > On 2011.11.18 at 16:43 +0800, Alex,Shi wrote:
> > > > > > > > 
> > > > > > > > The dirty flag comes from a bunch of unrelated xfs patches from Christoph, that
> > > > > > > > I'm testing right now.
> > > > > > 
> > > > > > Where is the xfs patchset? I am wondering if it is due to slub code. 
> > > > 
> > > > I begin to wonder if this might be the result of a compiler bug. 
> > > > The kernel in question was compiled with gcc version 4.7.0 20111117. And
> > > > there was commit to the gcc repository today that looks suspicious:
> > > > http://gcc.gnu.org/viewcvs?view=revision&revision=181466
> > > > 
> > > 
> > > Tell us if it is still there and you can reproduce it.
> > 
> > Hm, just noticed the "3.2.0-rc1 panic on PowerPC" thread:
> > http://thread.gmane.org/gmane.linux.kernel/1215584
> > 
> > The backtraces look suspiciously similar to mine.
> 
> So everything points to commit 87fb4b7b533:
> "net: more accurate skb truesize"
> 
> Can you take a look Eric?


This commit was followed by a fix (for SLOB, since SLUB/SLAB were not
affected)

Check commit bc417e30f8df (net: Add back alignment for size for
__alloc_skb)

If current kernel still crash, I believe there is a problem elsewhere (a
refcounting problem) that makes underlying page being reused :

The ksize(skb->head) call in pskb_expand_head() assumes skb->head is a
valid zone, not an already freed one...

By the way, we probably can remove (almost dead) code from
pskb_expand_head() since we now place struct skb_shared_info at the end
of skb->head at skb creation.

I'll send this patch later for net-next









--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Markus Trippelsdorf Nov. 21, 2011, 1:15 p.m. UTC | #1

On 2011.11.21 at 10:16 +0100, Eric Dumazet wrote:
> Le lundi 21 novembre 2011 à 09:24 +0100, Markus Trippelsdorf a écrit :
> > On 2011.11.21 at 09:05 +0100, Markus Trippelsdorf wrote:
> > > On 2011.11.21 at 08:44 +0800, Alex,Shi wrote:
> > > > On Fri, 2011-11-18 at 20:02 +0800, Markus Trippelsdorf wrote:
> > > > > On 2011.11.18 at 09:54 +0100, Markus Trippelsdorf wrote:
> > > > > > On 2011.11.18 at 16:43 +0800, Alex,Shi wrote:
> > > > > > > > > 
> > > > > > > > > The dirty flag comes from a bunch of unrelated xfs patches from Christoph, that
> > > > > > > > > I'm testing right now.
> > > > > > > 
> > > > > > > Where is the xfs patchset? I am wondering if it is due to slub code. 
> > > > > 
> > > > > I begin to wonder if this might be the result of a compiler bug. 
> > > > > The kernel in question was compiled with gcc version 4.7.0 20111117. And
> > > > > there was commit to the gcc repository today that looks suspicious:
> > > > > http://gcc.gnu.org/viewcvs?view=revision&revision=181466
> > > > > 
> > > > 
> > > > Tell us if it is still there and you can reproduce it.
> > > 
> > > Hm, just noticed the "3.2.0-rc1 panic on PowerPC" thread:
> > > http://thread.gmane.org/gmane.linux.kernel/1215584
> > > 
> > > The backtraces look suspiciously similar to mine.
> > 
> > So everything points to commit 87fb4b7b533:
> > "net: more accurate skb truesize"
> > 
> > Can you take a look Eric?
> 
> 
> This commit was followed by a fix (for SLOB, since SLUB/SLAB were not
> affected)
> 
> Check commit bc417e30f8df (net: Add back alignment for size for
> __alloc_skb)
> 
> If current kernel still crash, I believe there is a problem elsewhere (a
> refcounting problem) that makes underlying page being reused :
> 
> The ksize(skb->head) call in pskb_expand_head() assumes skb->head is a
> valid zone, not an already freed one...

I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:


Nov 21 13:58:21 x4 kernel: Freeing unused kernel memory: 1212k freed
Nov 21 13:58:21 x4 kernel: Freeing unused kernel memory: 128k freed
Nov 21 13:58:21 x4 kernel: XFS (sda): Mounting Filesystem
Nov 21 13:58:21 x4 kernel: XFS (sda): Ending clean mount
Nov 21 13:58:21 x4 kernel: ATL1E 0000:02:00.0: irq 40 for MSI/MSI-X
Nov 21 13:58:21 x4 kernel: ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex>
Nov 21 13:58:21 x4 kernel: ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex>
Nov 21 13:58:22 x4 kernel: Adding 2097148k swap on /var/tmp/swap/swapfile.  Priority:-1 extents:2 across:2634672k 
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ce6c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ce7c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ce8c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ce9c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ceac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cebc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cecc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cedc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ceec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cefc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf0c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf1c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf2c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf3c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf4c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf5c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf6c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf7c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf8c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cf9c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cfac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cfbc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cfcc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cfdc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cfec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6cffc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d00c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d01c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d02c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d03c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d04c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d05c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d06c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d07c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d08c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d09c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d0ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d0bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d0cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d0dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d0ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d0fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d10c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d11c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d12c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d13c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d14c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d15c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d16c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d17c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d18c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d19c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d1ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d1bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d1cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d1dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d1ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d1fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d20c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d21c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d22c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d23c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d24c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d25c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d26c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d27c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d28c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d29c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d2ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d2bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d2cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d2dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d2ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d2fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d30c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d31c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d32c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d33c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d34c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d35c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d36c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d37c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d38c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d39c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d3ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d3bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d3cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d3dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d3ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d3fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d40c: 00 00 00 00 90 15 09 11 02 88 ff ff 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d41c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d42c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d43c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d44c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d45c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d46c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d47c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d48c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d49c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d4ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d4bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d4cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d4dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d4ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d4fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d50c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d51c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d52c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d53c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d54c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d55c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d56c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d57c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d58c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d59c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d5ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d5bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d5cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d5dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d5ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d5fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d60c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d61c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d62c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d63c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d64c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d65c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d66c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d67c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d68c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d69c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d6ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d6bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d6cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d6dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d6ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d6fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d70c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d71c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d72c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d73c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d74c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d75c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d76c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d77c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d78c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d79c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d7ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d7bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d7cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d7dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d7ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d7fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d80c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d81c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d82c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d83c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d84c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d85c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d86c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d87c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d88c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d89c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d8ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d8bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d8cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d8dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d8ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d8fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d90c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d91c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d92c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d93c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d94c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d95c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d96c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d97c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d98c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d99c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d9ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d9bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d9cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d9dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d9ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6d9fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da0c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da1c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da2c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da3c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da4c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da5c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da6c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da7c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da8c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6da9c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6daac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dabc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dacc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dadc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6daec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dafc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db0c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db1c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db2c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db3c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db4c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db5c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db6c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db7c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db8c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6db9c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dbac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dbbc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dbcc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dbdc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dbec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dbfc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc0c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc1c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc2c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc3c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc4c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc5c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc6c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc7c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc8c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dc9c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dcac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dcbc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dccc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dcdc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dcec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dcfc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd0c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd1c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd2c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd3c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd4c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd5c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd6c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd7c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd8c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dd9c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ddac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ddbc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ddcc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dddc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ddec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6ddfc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de0c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de1c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de2c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de3c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de4c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de5c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de6c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de7c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de8c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6de9c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6deac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6debc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6decc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dedc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6deec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6defc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df0c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df1c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df2c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df3c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df4c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df5c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df6c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df7c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df8c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6df9c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dfac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dfbc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dfcc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dfdc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dfec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6dffc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e00c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e01c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e02c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e03c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e04c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e05c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e06c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e07c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e08c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e09c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e0ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e0bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e0cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e0dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e0ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e0fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e10c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e11c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e12c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e13c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e14c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e15c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e16c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e17c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e18c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e19c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e1ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e1bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e1cc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e1dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e1ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e1fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e20c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e21c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e22c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e23c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e24c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e25c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e26c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e27c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e28c: 00 00 00 00 cc cc cc cc cc cc cc cc 48 a1 c6 15  ............H...
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e29c: 02 88 ff ff 9d 4f 8a 81 ff ff ff ff f1 64 4c 81  .....O.......dL.
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e2ac: ff ff ff ff 01 4f 10 81 ff ff ff ff 9d 4f 8a 81  .....O.......O..
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e2bc: ff ff ff ff 83 08 89 81 ff ff ff ff cc 09 89 81  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e2cc: ff ff ff ff b4 dd 4c 81 ff ff ff ff 00 00 00 00  ......L.........
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e2dc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e2ec: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e2fc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e30c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e31c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e32c: 01 00 00 00 75 6f fb ff 00 00 00 00 00 00 00 00  ....uo..........
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e33c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e34c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e35c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e36c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e37c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e38c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e39c: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e3ac: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e3bc: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e3cc: 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ....ZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e3dc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e3ec: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e3fc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e40c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e41c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e42c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e43c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e44c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e45c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e46c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e47c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e48c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e49c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e4ac: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e4bc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e4cc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e4dc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e4ec: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e4fc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e50c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e51c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e52c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e53c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e54c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e55c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e56c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e57c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e58c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e59c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e5ac: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e5bc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e5cc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e5dc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e5ec: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e5fc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e60c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e61c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e62c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e63c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e64c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e65c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e66c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e67c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e68c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e69c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e6ac: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e6bc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e6cc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e6dc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e6ec: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e6fc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e70c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e71c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e72c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e73c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e74c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e75c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e76c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e77c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e78c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e79c: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e7ac: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e7bc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e7cc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e7dc: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e7ec: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e7fc: 5a 5a 5a 5a 00 00 00 00                          ZZZZ....
Nov 21 14:02:54 x4 kernel: FIX kmalloc-8192: Restoring 0xffff880215c6cbdc-0xffff880215c6e803=0x5a
Nov 21 14:02:54 x4 kernel: 
Nov 21 14:02:54 x4 kernel: =============================================================================
Nov 21 14:02:54 x4 kernel: BUG kmalloc-8192: Redzone overwritten
Nov 21 14:02:54 x4 kernel: -----------------------------------------------------------------------------
Nov 21 14:02:54 x4 kernel: 
Nov 21 14:02:54 x4 kernel: INFO: 0xffff880215c6e290-0xffff880215c6e297. First byte 0x5a instead of 0xcc
Nov 21 14:02:54 x4 kernel: INFO: Allocated in 0x5a5a5a5a5a5a5a5a age=11936128522577594085 cpu=1515870810 pid=1515870810
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: INFO: Freed in 0x5a5a5a5a5a5a5a5a age=11936128522577594085 cpu=1515870810 pid=1515870810
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: INFO: Slab 0xffffea0008571a00 objects=3 used=3 fp=0x          (null) flags=0x4000000000004081
Nov 21 14:02:54 x4 kernel: INFO: Object 0xffff880215c6c290 @offset=17040 fp=0x5a5a5a5a5a5a5a5a
Nov 21 14:02:54 x4 kernel: 
Nov 21 14:02:54 x4 kernel: Bytes b4 ffff880215c6c280: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c290: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c2a0: e0 19 09 11 02 88 ff ff 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c2c0: 00 00 00 00 00 00 00 00 e0 02 09 11 02 88 ff ff  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c2d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c310: 00 00 00 00 00 00 00 00 50 1b 09 11 02 88 ff ff  ........P.......
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c380: b0 12 09 11 02 88 ff ff 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c390: 00 00 00 00 00 00 00 00 40 11 09 11 02 88 ff ff  ........@.......
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c3b0: 00 00 09 11 02 88 ff ff 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6c9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ca90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6caa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6caf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cb90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cbb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cbc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cbd0: 00 00 00 00 00 00 00 00 00 00 00 00 5a 5a 5a 5a  ............ZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cbe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cbf0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc00: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc10: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc20: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc30: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc40: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc50: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc60: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc70: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc80: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cc90: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cca0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ccb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ccc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ccd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cce0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ccf0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd00: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd10: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd20: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd30: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd40: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd50: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd60: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd70: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd80: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cd90: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cda0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cdb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cdc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cdd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cde0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cdf0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce00: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce10: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce20: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce30: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce40: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce50: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce60: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce70: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce80: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ce90: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cea0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ceb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cec0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6ced0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cee0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cef0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf00: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf10: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf20: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf30: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf40: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf50: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf60: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf70: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf80: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cf90: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cfa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cfb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cfc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cfd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cfe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6cff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d000: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d010: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d020: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d030: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d040: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d060: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d070: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d080: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d0c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d0d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d0e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d0f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d100: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d110: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d120: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d130: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d140: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d150: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d160: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d170: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d180: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d190: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d1a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d1b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d1c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d1d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d1e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d1f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d200: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d210: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d220: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d230: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d240: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d250: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d260: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d270: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Object ffff880215c6d280: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Redzone ffff880215c6e290: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Padding ffff880215c6e3d0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Nov 21 14:02:54 x4 kernel: Pid: 4956, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #70
Nov 21 14:02:54 x4 kernel: Call Trace:
Nov 21 14:02:54 x4 kernel: [<ffffffff81101cf8>] ? print_section+0x38/0x40
Nov 21 14:02:54 x4 kernel: [<ffffffff811021f3>] print_trailer+0xe3/0x150
Nov 21 14:02:54 x4 kernel: [<ffffffff811023f0>] check_bytes_and_report+0xe0/0x100
Nov 21 14:02:54 x4 kernel: [<ffffffff811031a3>] check_object+0x183/0x240
Nov 21 14:02:54 x4 kernel: [<ffffffff81103cc0>] validate_slab_slab+0x1c0/0x230
Nov 21 14:02:54 x4 kernel: [<ffffffff811061f6>] validate_store+0xf6/0x190
Nov 21 14:02:54 x4 kernel: [<ffffffff8110163c>] slab_attr_store+0x1c/0x30
Nov 21 14:02:54 x4 kernel: [<ffffffff811634f8>] sysfs_write_file+0xc8/0x140
Nov 21 14:02:54 x4 kernel: [<ffffffff8110dc93>] vfs_write+0xa3/0x160
Nov 21 14:02:54 x4 kernel: [<ffffffff8110de25>] sys_write+0x45/0x90
Nov 21 14:02:54 x4 kernel: [<ffffffff814ccbfb>] system_call_fastpath+0x16/0x1b
Nov 21 14:02:54 x4 kernel: FIX kmalloc-8192: Restoring 0xffff880215c6e290-0xffff880215c6e297=0xcc
Nov 21 14:02:54 x4 kernel: 
Nov 21 14:06:43 x4 kernel: =============================================================================
Nov 21 14:06:43 x4 kernel: BUG kmalloc-8192: Freepointer corrupt
Nov 21 14:06:43 x4 kernel: -----------------------------------------------------------------------------
Nov 21 14:06:43 x4 kernel: 
Nov 21 14:06:43 x4 kernel: INFO: Allocated in 0x5a5a5a5a5a5a5a5a age=11936128522577823240 cpu=1515870810 pid=1515870810
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: INFO: Freed in 0x5a5a5a5a5a5a5a5a age=11936128522577823240 cpu=1515870810 pid=1515870810
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 	0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: INFO: Slab 0xffffea0008571a00 objects=3 used=3 fp=0x          (null) flags=0x4000000000004081
Nov 21 14:06:43 x4 kernel: INFO: Object 0xffff880215c6c290 @offset=17040 fp=0x5a5a5a5a5a5a5a5a
Nov 21 14:06:43 x4 kernel: 
Nov 21 14:06:43 x4 kernel: Bytes b4 ffff880215c6c280: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c290: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c2a0: e0 19 09 11 02 88 ff ff 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c2c0: 00 00 00 00 00 00 00 00 e0 02 09 11 02 88 ff ff  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c2d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c310: 00 00 00 00 00 00 00 00 50 1b 09 11 02 88 ff ff  ........P.......
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c380: b0 12 09 11 02 88 ff ff 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c390: 00 00 00 00 00 00 00 00 40 11 09 11 02 88 ff ff  ........@.......
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c3b0: 00 00 09 11 02 88 ff ff 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6c9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ca90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6caa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6caf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cb90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cbb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cbc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cbd0: 00 00 00 00 00 00 00 00 00 00 00 00 5a 5a 5a 5a  ............ZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cbe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cbf0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc00: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc10: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc20: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc30: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc40: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc50: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc60: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc70: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc80: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cc90: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cca0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ccb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ccc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ccd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cce0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ccf0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd00: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd10: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd20: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd30: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd40: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd50: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd60: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd70: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd80: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cd90: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cda0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cdb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cdc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cdd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cde0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cdf0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce00: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce10: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce20: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce30: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce40: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce50: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce60: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce70: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce80: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ce90: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cea0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ceb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cec0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6ced0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cee0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cef0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf00: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf10: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf20: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf30: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf40: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf50: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf60: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf70: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf80: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cf90: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cfa0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cfb0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cfc0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cfd0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cfe0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6cff0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d000: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d010: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d020: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d030: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d040: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d050: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d060: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d070: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d080: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d090: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d0a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d0b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d0c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d0d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d0e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d0f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d100: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d110: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d120: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d130: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d140: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d150: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d160: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d170: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d180: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d190: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d1a0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d1b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d1c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d1d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d1e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d1f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d200: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d210: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d220: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d230: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d240: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d250: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d260: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d270: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Object ffff880215c6d280: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Redzone ffff880215c6e290: cc cc cc cc cc cc cc cc                          ........
Nov 21 14:06:43 x4 kernel: Padding ffff880215c6e3d0: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Nov 21 14:06:43 x4 kernel: Pid: 7254, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #70
Nov 21 14:06:43 x4 kernel: Call Trace:
Nov 21 14:06:43 x4 kernel: [<ffffffff81101cf8>] ? print_section+0x38/0x40
Nov 21 14:06:43 x4 kernel: [<ffffffff811021f3>] print_trailer+0xe3/0x150
Nov 21 14:06:43 x4 kernel: [<ffffffff8110229c>] object_err+0x3c/0x50
Nov 21 14:06:43 x4 kernel: [<ffffffff81103155>] check_object+0x135/0x240
Nov 21 14:06:43 x4 kernel: [<ffffffff81103cc0>] validate_slab_slab+0x1c0/0x230
Nov 21 14:06:43 x4 kernel: [<ffffffff811061f6>] validate_store+0xf6/0x190
Nov 21 14:06:43 x4 kernel: [<ffffffff8110163c>] slab_attr_store+0x1c/0x30
Nov 21 14:06:43 x4 kernel: [<ffffffff811634f8>] sysfs_write_file+0xc8/0x140
Nov 21 14:06:43 x4 kernel: [<ffffffff8110dc93>] vfs_write+0xa3/0x160
Nov 21 14:06:43 x4 kernel: [<ffffffff8110de25>] sys_write+0x45/0x90
Nov 21 14:06:43 x4 kernel: [<ffffffff814ccbfb>] system_call_fastpath+0x16/0x1b
Nov 21 14:06:49 x4 kernel: general protection fault: 0000 [#1] PREEMPT SMP 
Nov 21 14:06:49 x4 kernel: CPU 0 
Nov 21 14:06:49 x4 kernel: Pid: 4203, comm: git Not tainted 3.2.0-rc2-00274-g6fe4c6d #70 System manufacturer System Product Name/M4A78T-E
Nov 21 14:06:49 x4 kernel: RIP: 0010:[<ffffffff8110b26c>]  [<ffffffff8110b26c>] __khugepaged_enter+0x8c/0x140
Nov 21 14:06:49 x4 kernel: RSP: 0018:ffff88020020dd60  EFLAGS: 00010206
Nov 21 14:06:49 x4 kernel: RAX: 5a5a5a5a5a5a5a5a RBX: ffff880211090fd0 RCX: 0000000000000000
Nov 21 14:06:49 x4 kernel: RDX: ffff880215c6d4e0 RSI: 0000000000000000 RDI: ffffffff8195b7a0
Nov 21 14:06:49 x4 kernel: RBP: ffff88020020dd90 R08: 0000000000000000 R09: ffff880211090fd0
Nov 21 14:06:49 x4 kernel: R10: 0000000000000000 R11: 0000000000000030 R12: ffff880213de7200
Nov 21 14:06:49 x4 kernel: R13: ffff880208dc2440 R14: ffff8802144c6d40 R15: ffff880213d33200
Nov 21 14:06:49 x4 kernel: FS:  00007fa9f36d1700(0000) GS:ffff88021fc00000(0000) knlGS:0000000000000000
Nov 21 14:06:49 x4 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 21 14:06:49 x4 kernel: CR2: 00007f5514127000 CR3: 0000000200305000 CR4: 00000000000006f0
Nov 21 14:06:49 x4 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 21 14:06:49 x4 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 21 14:06:49 x4 kernel: Process git (pid: 4203, threadinfo ffff88020020c000, task ffff880216aade40)
Nov 21 14:06:49 x4 kernel: Stack:
Nov 21 14:06:49 x4 kernel: ffff880213de7200 ffff880208dc2440 ffff88020020dd90 ffff880208dc2440
Nov 21 14:06:49 x4 kernel: ffff880213de7200 ffff880208dc2440 ffff88020020de20 ffffffff8106ae68
Nov 21 14:06:49 x4 kernel: ffff88020020ddf0 0000000001200011 ffff880213de7260 ffff8802144c6da0
Nov 21 14:06:49 x4 kernel: Call Trace:
Nov 21 14:06:49 x4 kernel: [<ffffffff8106ae68>] dup_mm+0x318/0x4a0
Nov 21 14:06:49 x4 kernel: [<ffffffff81104dd8>] ? kmem_cache_alloc+0x148/0x150
Nov 21 14:06:49 x4 kernel: [<ffffffff8106bb70>] ? copy_process+0xb50/0xf60
Nov 21 14:06:49 x4 kernel: [<ffffffff8106bd28>] copy_process+0xd08/0xf60
Nov 21 14:06:49 x4 kernel: [<ffffffff8110ed38>] ? get_empty_filp+0x78/0x130
Nov 21 14:06:49 x4 kernel: [<ffffffff8106c0b0>] do_fork+0x100/0x2b0
Nov 21 14:06:49 x4 kernel: [<ffffffff81116e95>] ? do_pipe_flags+0x95/0xf0
Nov 21 14:06:49 x4 kernel: [<ffffffff81044d63>] sys_clone+0x23/0x30
Nov 21 14:06:49 x4 kernel: [<ffffffff814cce63>] stub_clone+0x13/0x20
Nov 21 14:06:49 x4 kernel: [<ffffffff814ccbfb>] ? system_call_fastpath+0x16/0x1b
Nov 21 14:06:49 x4 kernel: Code: c5 4e ec c4 4e ec c4 4e 4c 89 63 20 48 f7 e2 48 c1 ea 05 81 e2 f8 1f 00 00 48 03 15 9f a4 76 00 48 8b 02 48 85 c0 48 89 03 74 04 
Nov 21 14:06:49 x4 kernel: RIP  [<ffffffff8110b26c>] __khugepaged_enter+0x8c/0x140
Nov 21 14:06:49 x4 kernel: RSP <ffff88020020dd60>
Nov 21 14:06:49 x4 kernel: ---[ end trace ded3169f729e0239 ]---
Nov 21 14:06:49 x4 kernel: note: git[4203] exited with preempt_count 1
Nov 21 14:06:49 x4 kernel: BUG: scheduling while atomic: git/4203/0x00000002
Nov 21 14:06:49 x4 kernel: Pid: 4203, comm: git Tainted: G      D      3.2.0-rc2-00274-g6fe4c6d #70
Nov 21 14:06:49 x4 kernel: Call Trace:
Nov 21 14:06:49 x4 kernel: [<ffffffff814c44fb>] __schedule_bug+0x50/0x55
Nov 21 14:06:49 x4 kernel: [<ffffffff814c9ef8>] __schedule+0x558/0x7e0
Nov 21 14:06:49 x4 kernel: [<ffffffff810ac8f1>] ? taskstats_exit+0x281/0x380
Nov 21 14:06:49 x4 kernel: [<ffffffff814ca1ba>] schedule+0x3a/0x50
Nov 21 14:06:49 x4 kernel: [<ffffffff814cbd3d>] rwsem_down_failed_common+0xbd/0x150
Nov 21 14:06:49 x4 kernel: [<ffffffff814cbde5>] rwsem_down_read_failed+0x15/0x20
Nov 21 14:06:49 x4 kernel: [<ffffffff8120cf84>] call_rwsem_down_read_failed+0x14/0x30
Nov 21 14:06:49 x4 kernel: [<ffffffff814cb622>] ? down_read+0x12/0x20
Nov 21 14:06:49 x4 kernel: [<ffffffff8106ead7>] exit_mm+0x37/0x120
Nov 21 14:06:49 x4 kernel: [<ffffffff810702fe>] do_exit+0x13e/0x7f0
Nov 21 14:06:49 x4 kernel: [<ffffffff8106e70d>] ? kmsg_dump+0xbd/0xf0
Nov 21 14:06:49 x4 kernel: [<ffffffff8103f81f>] oops_end+0x8f/0xd0
Nov 21 14:06:49 x4 kernel: [<ffffffff8103f9a3>] die+0x53/0x80
Nov 21 14:06:49 x4 kernel: [<ffffffff8103d1c2>] do_general_protection+0x152/0x160
Nov 21 14:06:49 x4 kernel: [<ffffffff814cc7bf>] general_protection+0x1f/0x30
Nov 21 14:06:49 x4 kernel: [<ffffffff8110b26c>] ? __khugepaged_enter+0x8c/0x140
Nov 21 14:06:49 x4 kernel: [<ffffffff8106ae68>] dup_mm+0x318/0x4a0
Nov 21 14:06:49 x4 kernel: [<ffffffff81104dd8>] ? kmem_cache_alloc+0x148/0x150
Nov 21 14:06:49 x4 kernel: [<ffffffff8106bb70>] ? copy_process+0xb50/0xf60
Nov 21 14:06:49 x4 kernel: [<ffffffff8106bd28>] copy_process+0xd08/0xf60
Nov 21 14:06:49 x4 kernel: [<ffffffff8110ed38>] ? get_empty_filp+0x78/0x130
Nov 21 14:06:49 x4 kernel: [<ffffffff8106c0b0>] do_fork+0x100/0x2b0
Nov 21 14:06:49 x4 kernel: [<ffffffff81116e95>] ? do_pipe_flags+0x95/0xf0
Nov 21 14:06:49 x4 kernel: [<ffffffff81044d63>] sys_clone+0x23/0x30
Nov 21 14:06:49 x4 kernel: [<ffffffff814cce63>] stub_clone+0x13/0x20
Nov 21 14:06:49 x4 kernel: [<ffffffff814ccbfb>] ? system_call_fastpath+0x16/0x1b

Eric Dumazet Nov. 21, 2011, 2:16 p.m. UTC | #2

Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :

> I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> 

Thanks

Please continue to provide more samples.

There is something wrong somewhere, but where exactly, its hard to say.

Something is keeping a pointer to freed memory and reuse it while memory
had been reused.

Are you using hugepages ?



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Markus Trippelsdorf Nov. 21, 2011, 2:21 p.m. UTC | #3

On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> 
> > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> > 
> 
> Thanks
> 
> Please continue to provide more samples.
> 
> There is something wrong somewhere, but where exactly, its hard to say.
> 
> Something is keeping a pointer to freed memory and reuse it while memory
> had been reused.
> 
> Are you using hugepages ?

Only CONFIG_TRANSPARENT_HUGEPAGE and CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS.

I've increased LOG_BUF_SHIFT to 18, because it looks like the beginning
of the SLUB_DEBUG output was cut off last time.

Markus Trippelsdorf Nov. 21, 2011, 3:36 p.m. UTC | #4

On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> 
> > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> > 
> 
> Thanks
> 
> Please continue to provide more samples.
> 
> There is something wrong somewhere, but where exactly, its hard to say.

New sample. This one points to lib/idr.c:

[drm] Initialized drm 1.1.0 20060810
[drm] radeon defaulting to kernel modesetting.
[drm] radeon kernel modesetting enabled.
radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
radeon 0000:01:05.0: setting latency timer to 64
[drm] initializing kernel modesetting (RS780 0x1002:0x9614 0x1043:0x834D).
[drm] register mmio base: 0xFBEE0000
[drm] register mmio size: 65536
ATOM BIOS: 113
radeon 0000:01:05.0: VRAM: 128M 0x00000000C0000000 - 0x00000000C7FFFFFF (128M used)
radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[drm] Detected VRAM RAM=128M, BAR=128M
[drm] RAM width 32bits DDR
[TTM] Zone  kernel: Available graphics memory: 4083428 kiB.
[TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
[TTM] Initializing pool allocator.
[drm] radeon: 128M of VRAM memory ready
[drm] radeon: 512M of GTT memory ready.
[drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[drm] Driver supports precise vblank timestamp query.
[drm] radeon: irq initialized.
[drm] GART: num cpu pages 131072, num gpu pages 131072
[drm] Loading RS780 Microcode
[drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
radeon 0000:01:05.0: WB enabled
[drm] ring test succeeded in 1 usecs
[drm] radeon: ib pool ready.
[drm] ib test succeeded in 0 usecs
=============================================================================
BUG idr_layer_cache: Poison overwritten
-----------------------------------------------------------------------------

INFO: 0xffff880215650800-0xffff880215650803. First byte 0x0 instead of 0x6b
INFO: Slab 0xffffea0008559400 objects=18 used=18 fp=0x          (null) flags=0x4000000000004080
INFO: Object 0xffff8802156506d0 @offset=1744 fp=0xffff880215650a38

Bytes b4 ffff8802156506c0: a4 6f fb ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  .o......ZZZZZZZZ
Object ffff8802156506d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156506e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156506f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156507a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156507b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156507c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156507d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156507e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156507f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
Object ffff880215650810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215650890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156508a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156508b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156508c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156508d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156508e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Redzone ffff8802156508f0: bb bb bb bb bb bb bb bb                          ........
Padding ffff880215650a30: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Pid: 1, comm: swapper Not tainted 3.2.0-rc2-00274-g6fe4c6d #71
Call Trace:
 [<ffffffff81101cf8>] ? print_section+0x38/0x40
 [<ffffffff811021f3>] print_trailer+0xe3/0x150
 [<ffffffff811023f0>] check_bytes_and_report+0xe0/0x100
 [<ffffffff811031e6>] check_object+0x1c6/0x240
 [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
 [<ffffffff814c5c43>] alloc_debug_processing+0x62/0xe4
 [<ffffffff814c64f1>] __slab_alloc.constprop.69+0x1a4/0x1e0
 [<ffffffff8129ae77>] ? drm_property_create+0x47/0x110
 [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
 [<ffffffff81104db1>] kmem_cache_alloc+0x121/0x150
 [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
 [<ffffffff812031f0>] idr_pre_get+0x60/0x90
 [<ffffffff8129870a>] drm_mode_object_get+0x6a/0xc0
 [<ffffffff8129ae95>] drm_property_create+0x65/0x110
 [<ffffffff8129b15d>] drm_mode_config_init+0xfd/0x190
 [<ffffffff812e12ad>] radeon_modeset_init+0x1d/0x860
 [<ffffffff813211c7>] ? radeon_acpi_init+0x87/0xc0
 [<ffffffff812c37b8>] radeon_driver_load_kms+0xf8/0x150
 [<ffffffff81295a06>] drm_get_pci_dev+0x186/0x2d0
 [<ffffffff814bf1fd>] ? radeon_pci_probe+0x9e/0xb8
 [<ffffffff814bf20f>] radeon_pci_probe+0xb0/0xb8
 [<ffffffff8121be15>] pci_device_probe+0x75/0xa0
 [<ffffffff81324e1a>] ? driver_sysfs_add+0x7a/0xb0
 [<ffffffff81325021>] driver_probe_device+0x71/0x190
 [<ffffffff813251db>] __driver_attach+0x9b/0xa0
 [<ffffffff81325140>] ? driver_probe_device+0x190/0x190
 [<ffffffff81323e0d>] bus_for_each_dev+0x4d/0x90
 [<ffffffff813252f9>] driver_attach+0x19/0x20
 [<ffffffff81324598>] bus_add_driver+0x188/0x250
 [<ffffffff81325942>] driver_register+0x72/0x150
 [<ffffffff81321de5>] ? device_add+0x75/0x600
 [<ffffffff8121bb7d>] __pci_register_driver+0x5d/0xd0
 [<ffffffff81295c54>] drm_pci_init+0x104/0x120
 [<ffffffff818abefa>] ? ttm_init+0x62/0x62
 [<ffffffff818abfe1>] radeon_init+0xe7/0xe9
 [<ffffffff81890883>] do_one_initcall+0x7a/0x129
 [<ffffffff818909cc>] kernel_init+0x9a/0x114
 [<ffffffff814cddb4>] kernel_thread_helper+0x4/0x10
 [<ffffffff81890932>] ? do_one_initcall+0x129/0x129
 [<ffffffff814cddb0>] ? gs_change+0xb/0xb
FIX idr_layer_cache: Restoring 0xffff880215650800-0xffff880215650803=0x6b

FIX idr_layer_cache: Marking all objects used
[drm] Radeon Display Connectors
[drm] Connector 0:
[drm]   VGA
[drm]   DDC: 0x7e40 0x7e40 0x7e44 0x7e44 0x7e48 0x7e48 0x7e4c 0x7e4c
[drm]   Encoders:
[drm]     CRT1: INTERNAL_KLDSCP_DAC1
[drm] Connector 1:
[drm]   DVI-D
[drm]   HPD3
[drm]   DDC: 0x7e50 0x7e50 0x7e54 0x7e54 0x7e58 0x7e58 0x7e5c 0x7e5c
[drm]   Encoders:
[drm]     DFP3: INTERNAL_KLDSCP_LVTMA
[drm] radeon: power management initialized
[drm] fb mappable at 0xF0142000
[drm] vram apper at 0xF0000000
[drm] size 7299072
[drm] fb depth is 24
[drm]    pitch is 6912
fbcon: radeondrmfb (fb0) is primary device
Console: switching to colour frame buffer device 131x105
...

Eric Dumazet Nov. 21, 2011, 3:48 p.m. UTC | #5

Le lundi 21 novembre 2011 à 16:36 +0100, Markus Trippelsdorf a écrit :
> On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> > 
> > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> > > 
> > 
> > Thanks
> > 
> > Please continue to provide more samples.
> > 
> > There is something wrong somewhere, but where exactly, its hard to say.
> 
> New sample. This one points to lib/idr.c:
> 
> [drm] Initialized drm 1.1.0 20060810
> [drm] radeon defaulting to kernel modesetting.
> [drm] radeon kernel modesetting enabled.
> radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
> radeon 0000:01:05.0: setting latency timer to 64
> [drm] initializing kernel modesetting (RS780 0x1002:0x9614 0x1043:0x834D).
> [drm] register mmio base: 0xFBEE0000
> [drm] register mmio size: 65536
> ATOM BIOS: 113
> radeon 0000:01:05.0: VRAM: 128M 0x00000000C0000000 - 0x00000000C7FFFFFF (128M used)
> radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
> [drm] Detected VRAM RAM=128M, BAR=128M
> [drm] RAM width 32bits DDR
> [TTM] Zone  kernel: Available graphics memory: 4083428 kiB.
> [TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
> [TTM] Initializing pool allocator.
> [drm] radeon: 128M of VRAM memory ready
> [drm] radeon: 512M of GTT memory ready.
> [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
> [drm] Driver supports precise vblank timestamp query.
> [drm] radeon: irq initialized.
> [drm] GART: num cpu pages 131072, num gpu pages 131072
> [drm] Loading RS780 Microcode
> [drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
> radeon 0000:01:05.0: WB enabled
> [drm] ring test succeeded in 1 usecs
> [drm] radeon: ib pool ready.
> [drm] ib test succeeded in 0 usecs
> =============================================================================
> BUG idr_layer_cache: Poison overwritten
> -----------------------------------------------------------------------------

Thanks, could you now add "CONFIG_DEBUG_PAGEALLOC=y" in your config as
well ?



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Markus Trippelsdorf Nov. 21, 2011, 3:51 p.m. UTC | #6

On 2011.11.21 at 16:36 +0100, Markus Trippelsdorf wrote:
> On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> > 
> > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> > > 
> > 
> > Thanks
> > 
> > Please continue to provide more samples.
> > 
> > There is something wrong somewhere, but where exactly, its hard to say.
> 
> =============================================================================
> BUG idr_layer_cache: Poison overwritten
> -----------------------------------------------------------------------------
> 
> INFO: 0xffff880215650800-0xffff880215650803. First byte 0x0 instead of 0x6b
> INFO: Slab 0xffffea0008559400 objects=18 used=18 fp=0x          (null) flags=0x4000000000004080
> INFO: Object 0xffff8802156506d0 @offset=1744 fp=0xffff880215650a38
> 
> Bytes b4 ffff8802156506c0: a4 6f fb ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  .o......ZZZZZZZZ
> Object ffff8802156506d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156506e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156506f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> Object ffff880215650810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
> Redzone ffff8802156508f0: bb bb bb bb bb bb bb bb                          ........
> Padding ffff880215650a30: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
> Pid: 1, comm: swapper Not tainted 3.2.0-rc2-00274-g6fe4c6d #71
> Call Trace:
>  [<ffffffff81101cf8>] ? print_section+0x38/0x40
>  [<ffffffff811021f3>] print_trailer+0xe3/0x150
>  [<ffffffff811023f0>] check_bytes_and_report+0xe0/0x100
>  [<ffffffff811031e6>] check_object+0x1c6/0x240
>  [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
>  [<ffffffff814c5c43>] alloc_debug_processing+0x62/0xe4
>  [<ffffffff814c64f1>] __slab_alloc.constprop.69+0x1a4/0x1e0
>  [<ffffffff8129ae77>] ? drm_property_create+0x47/0x110
>  [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
>  [<ffffffff81104db1>] kmem_cache_alloc+0x121/0x150
>  [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
>  [<ffffffff812031f0>] idr_pre_get+0x60/0x90
>  [<ffffffff8129870a>] drm_mode_object_get+0x6a/0xc0
>  [<ffffffff8129ae95>] drm_property_create+0x65/0x110
>  [<ffffffff8129b15d>] drm_mode_config_init+0xfd/0x190
>  [<ffffffff812e12ad>] radeon_modeset_init+0x1d/0x860
>  [<ffffffff813211c7>] ? radeon_acpi_init+0x87/0xc0
>  [<ffffffff812c37b8>] radeon_driver_load_kms+0xf8/0x150
>  [<ffffffff81295a06>] drm_get_pci_dev+0x186/0x2d0
>  [<ffffffff814bf1fd>] ? radeon_pci_probe+0x9e/0xb8
>  [<ffffffff814bf20f>] radeon_pci_probe+0xb0/0xb8
>  [<ffffffff8121be15>] pci_device_probe+0x75/0xa0
>  [<ffffffff81324e1a>] ? driver_sysfs_add+0x7a/0xb0
>  [<ffffffff81325021>] driver_probe_device+0x71/0x190
>  [<ffffffff813251db>] __driver_attach+0x9b/0xa0
>  [<ffffffff81325140>] ? driver_probe_device+0x190/0x190
>  [<ffffffff81323e0d>] bus_for_each_dev+0x4d/0x90
>  [<ffffffff813252f9>] driver_attach+0x19/0x20
>  [<ffffffff81324598>] bus_add_driver+0x188/0x250
>  [<ffffffff81325942>] driver_register+0x72/0x150
>  [<ffffffff81321de5>] ? device_add+0x75/0x600
>  [<ffffffff8121bb7d>] __pci_register_driver+0x5d/0xd0
>  [<ffffffff81295c54>] drm_pci_init+0x104/0x120
>  [<ffffffff818abefa>] ? ttm_init+0x62/0x62
>  [<ffffffff818abfe1>] radeon_init+0xe7/0xe9
>  [<ffffffff81890883>] do_one_initcall+0x7a/0x129
>  [<ffffffff818909cc>] kernel_init+0x9a/0x114
>  [<ffffffff814cddb4>] kernel_thread_helper+0x4/0x10
>  [<ffffffff81890932>] ? do_one_initcall+0x129/0x129
>  [<ffffffff814cddb0>] ? gs_change+0xb/0xb
> FIX idr_layer_cache: Restoring 0xffff880215650800-0xffff880215650803=0x6b
> 
> FIX idr_layer_cache: Marking all objects used

Running "slabinfo -v" later:

Nov 21 16:41:03 x4 kernel: =============================================================================
Nov 21 16:41:03 x4 kernel: BUG idr_layer_cache: Redzone overwritten
Nov 21 16:41:03 x4 kernel: -----------------------------------------------------------------------------
Nov 21 16:41:03 x4 kernel:
Nov 21 16:41:03 x4 kernel: INFO: 0xffff8802156508f0-0xffff8802156508f7. First byte 0xbb instead of 0xcc
Nov 21 16:41:03 x4 kernel: INFO: Slab 0xffffea0008559400 objects=18 used=18 fp=0x          (null) flags=0x4000000000004081
Nov 21 16:41:03 x4 kernel: INFO: Object 0xffff8802156506d0 @offset=1744 fp=0xffff880215650a38
Nov 21 16:41:03 x4 kernel:
Nov 21 16:41:03 x4 kernel: Bytes b4 ffff8802156506c0: a4 6f fb ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  .o......ZZZZZZZZ
Nov 21 16:41:03 x4 kernel: Object ffff8802156506d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156506e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156506f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156507a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156507b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156507c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156507d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156507e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156507f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff880215650890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156508a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156508b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156508c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156508d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:41:03 x4 kernel: Object ffff8802156508e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Nov 21 16:41:03 x4 kernel: Redzone ffff8802156508f0: bb bb bb bb bb bb bb bb                          ........
Nov 21 16:41:03 x4 kernel: Padding ffff880215650a30: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Nov 21 16:41:03 x4 kernel: Pid: 12278, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #71
Nov 21 16:41:03 x4 kernel: Call Trace:
Nov 21 16:41:03 x4 kernel: [<ffffffff81101cf8>] ? print_section+0x38/0x40
Nov 21 16:41:03 x4 kernel: [<ffffffff811021f3>] print_trailer+0xe3/0x150
Nov 21 16:41:03 x4 kernel: [<ffffffff811023f0>] check_bytes_and_report+0xe0/0x100
Nov 21 16:41:03 x4 kernel: [<ffffffff811031a3>] check_object+0x183/0x240
Nov 21 16:41:03 x4 kernel: [<ffffffff81103cc0>] validate_slab_slab+0x1c0/0x230
Nov 21 16:41:03 x4 kernel: [<ffffffff811061f6>] validate_store+0xf6/0x190
Nov 21 16:41:03 x4 kernel: [<ffffffff8110163c>] slab_attr_store+0x1c/0x30
Nov 21 16:41:03 x4 kernel: [<ffffffff811634f8>] sysfs_write_file+0xc8/0x140
Nov 21 16:41:03 x4 kernel: [<ffffffff8110dc93>] vfs_write+0xa3/0x160
Nov 21 16:41:03 x4 kernel: [<ffffffff8110de25>] sys_write+0x45/0x90
Nov 21 16:41:03 x4 kernel: [<ffffffff814ccbfb>] system_call_fastpath+0x16/0x1b
Nov 21 16:41:03 x4 kernel: FIX idr_layer_cache: Restoring 0xffff8802156508f0-0xffff8802156508f7=0xcc
Nov 21 16:41:03 x4 kernel:
Nov 21 16:42:07 x4 kernel: =============================================================================
Nov 21 16:42:07 x4 kernel: BUG idr_layer_cache: Redzone overwritten
Nov 21 16:42:07 x4 kernel: -----------------------------------------------------------------------------
Nov 21 16:42:07 x4 kernel:
Nov 21 16:42:07 x4 kernel: INFO: 0xffff880215650c58-0xffff880215650c5f. First byte 0xbb instead of 0xcc
Nov 21 16:42:07 x4 kernel: INFO: Slab 0xffffea0008559400 objects=18 used=18 fp=0x          (null) flags=0x4000000000004081
Nov 21 16:42:07 x4 kernel: INFO: Object 0xffff880215650a38 @offset=2616 fp=0xffff880215650da0
Nov 21 16:42:07 x4 kernel:
Nov 21 16:42:07 x4 kernel: Bytes b4 ffff880215650a28: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
Nov 21 16:42:07 x4 kernel: Object ffff880215650a38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650a48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650a58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650a68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650a78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650a88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650a98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650aa8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650ab8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650ac8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650ad8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650ae8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650af8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b58: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b68: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b78: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650b98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650ba8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650bb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650bc8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650bd8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650be8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650bf8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650c08: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650c18: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650c28: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650c38: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 16:42:07 x4 kernel: Object ffff880215650c48: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Nov 21 16:42:07 x4 kernel: Redzone ffff880215650c58: bb bb bb bb bb bb bb bb                          ........
Nov 21 16:42:07 x4 kernel: Padding ffff880215650d98: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Nov 21 16:42:07 x4 kernel: Pid: 12924, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #71
Nov 21 16:42:07 x4 kernel: Call Trace:
Nov 21 16:42:07 x4 kernel: [<ffffffff81101cf8>] ? print_section+0x38/0x40
Nov 21 16:42:07 x4 kernel: [<ffffffff811021f3>] print_trailer+0xe3/0x150
Nov 21 16:42:07 x4 kernel: [<ffffffff811023f0>] check_bytes_and_report+0xe0/0x100
Nov 21 16:42:07 x4 kernel: [<ffffffff811031a3>] check_object+0x183/0x240
Nov 21 16:42:07 x4 kernel: [<ffffffff81103cc0>] validate_slab_slab+0x1c0/0x230
Nov 21 16:42:07 x4 kernel: [<ffffffff811061f6>] validate_store+0xf6/0x190
Nov 21 16:42:07 x4 kernel: [<ffffffff8110163c>] slab_attr_store+0x1c/0x30
Nov 21 16:42:07 x4 kernel: [<ffffffff811634f8>] sysfs_write_file+0xc8/0x140
Nov 21 16:42:07 x4 kernel: [<ffffffff8110dc93>] vfs_write+0xa3/0x160
Nov 21 16:42:07 x4 kernel: [<ffffffff8110de25>] sys_write+0x45/0x90
Nov 21 16:42:07 x4 kernel: [<ffffffff814ccbfb>] system_call_fastpath+0x16/0x1b
Nov 21 16:42:07 x4 kernel: FIX idr_layer_cache: Restoring 0xffff880215650c58-0xffff880215650c5f=0xcc

Markus Trippelsdorf Nov. 21, 2011, 4:10 p.m. UTC | #7

On 2011.11.21 at 16:48 +0100, Eric Dumazet wrote:
> Le lundi 21 novembre 2011 à 16:36 +0100, Markus Trippelsdorf a écrit :
> > On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> > > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> > > 
> > > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> > > > 
> > > 
> > > Thanks
> > > 
> > > Please continue to provide more samples.
> > > 
> > > There is something wrong somewhere, but where exactly, its hard to say.
> > 
> > New sample. This one points to lib/idr.c:
> > 
> > [drm] Initialized drm 1.1.0 20060810
> > [drm] radeon defaulting to kernel modesetting.
> > [drm] radeon kernel modesetting enabled.
> > radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
> > radeon 0000:01:05.0: setting latency timer to 64
> > [drm] initializing kernel modesetting (RS780 0x1002:0x9614 0x1043:0x834D).
> > [drm] register mmio base: 0xFBEE0000
> > [drm] register mmio size: 65536
> > ATOM BIOS: 113
> > radeon 0000:01:05.0: VRAM: 128M 0x00000000C0000000 - 0x00000000C7FFFFFF (128M used)
> > radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
> > [drm] Detected VRAM RAM=128M, BAR=128M
> > [drm] RAM width 32bits DDR
> > [TTM] Zone  kernel: Available graphics memory: 4083428 kiB.
> > [TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
> > [TTM] Initializing pool allocator.
> > [drm] radeon: 128M of VRAM memory ready
> > [drm] radeon: 512M of GTT memory ready.
> > [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
> > [drm] Driver supports precise vblank timestamp query.
> > [drm] radeon: irq initialized.
> > [drm] GART: num cpu pages 131072, num gpu pages 131072
> > [drm] Loading RS780 Microcode
> > [drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
> > radeon 0000:01:05.0: WB enabled
> > [drm] ring test succeeded in 1 usecs
> > [drm] radeon: ib pool ready.
> > [drm] ib test succeeded in 0 usecs
> > =============================================================================
> > BUG idr_layer_cache: Poison overwritten
> > -----------------------------------------------------------------------------
> 
> Thanks, could you now add "CONFIG_DEBUG_PAGEALLOC=y" in your config as
> well ?

Sure. This one happend with CONFIG_DEBUG_PAGEALLOC=y:

[drm] Initialized radeon 2.11.0 20080528 for 0000:01:05.0 on minor 0
loop: module loaded
ahci 0000:00:11.0: version 3.0
ahci 0000:00:11.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
ahci 0000:00:11.0: AHCI 0001.0100 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part ccc 
scsi0 : ahci
scsi1 : ahci
=============================================================================
BUG task_struct: Poison overwritten
-----------------------------------------------------------------------------

INFO: 0xffff880215c43800-0xffff880215c43803. First byte 0x0 instead of 0x6b
INFO: Allocated in copy_process+0xc4/0xf60 age=168 cpu=1 pid=5
	__slab_alloc.constprop.70+0x1a4/0x1e0
	kmem_cache_alloc+0x126/0x160
	copy_process+0xc4/0xf60
	do_fork+0x100/0x2b0
	kernel_thread+0x6c/0x70
	__call_usermodehelper+0x31/0xa0
	process_one_work+0x11a/0x430
	worker_thread+0x126/0x2d0
	kthread+0x87/0x90
	kernel_thread_helper+0x4/0x10
INFO: Freed in free_task+0x3e/0x50 age=156 cpu=2 pid=13
	__slab_free+0x33/0x2d0
	kmem_cache_free+0x104/0x120
	free_task+0x3e/0x50
	__put_task_struct+0xb0/0x110
	delayed_put_task_struct+0x3b/0xa0
	__rcu_process_callbacks+0x12a/0x350
	rcu_process_callbacks+0x62/0x140
	__do_softirq+0xa8/0x200
	run_ksoftirqd+0x107/0x210
	kthread+0x87/0x90
	kernel_thread_helper+0x4/0x10
INFO: Slab 0xffffea0008571000 objects=17 used=17 fp=0x          (null) flags=0x4000000000004080
INFO: Object 0xffff880215c432c0 @offset=12992 fp=0xffff880215c41d00

Bytes b4 ffff880215c432b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Object ffff880215c432c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c432d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c432e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c432f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43300: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43310: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43320: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43330: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43340: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43350: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43360: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43370: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43380: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43390: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c433a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c433b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c433c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c433d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c433e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c433f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43400: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43440: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43450: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43460: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43470: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43480: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43490: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c434a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c434b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c434c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c434d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c434e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c434f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43500: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43510: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43520: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43530: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43540: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43550: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43560: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43570: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43580: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43590: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c435a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c435b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c435c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c435d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c435e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c435f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43600: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43610: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43620: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43630: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43640: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43650: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43660: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43670: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43680: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43690: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c436a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c436b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c436c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c436d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c436e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c436f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c437a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c437b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c437c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c437d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c437e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c437f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
Object ffff880215c43810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c43890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215c438a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Redzone ffff880215c438b0: bb bb bb bb bb bb bb bb                          ........
Padding ffff880215c439f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Pid: 5, comm: kworker/u:0 Not tainted 3.2.0-rc2-00274-g6fe4c6d #72
Call Trace:
 [<ffffffff81101ca8>] ? print_section+0x38/0x40
 [<ffffffff811021a3>] print_trailer+0xe3/0x150
 [<ffffffff811023a0>] check_bytes_and_report+0xe0/0x100
 [<ffffffff81103196>] check_object+0x1c6/0x240
 [<ffffffff8106b034>] ? copy_process+0xc4/0xf60
 [<ffffffff814c5bb3>] alloc_debug_processing+0x62/0xe4
 [<ffffffff814c6461>] __slab_alloc.constprop.70+0x1a4/0x1e0
 [<ffffffff8106b034>] ? copy_process+0xc4/0xf60
 [<ffffffff814ca12a>] ? schedule+0x3a/0x50
 [<ffffffff81104d66>] kmem_cache_alloc+0x126/0x160
 [<ffffffff8106b034>] ? copy_process+0xc4/0xf60
 [<ffffffff81065f18>] ? enqueue_task_fair+0xf8/0x140
 [<ffffffff8106b034>] copy_process+0xc4/0xf60
 [<ffffffff8106c000>] do_fork+0x100/0x2b0
 [<ffffffff810920fd>] ? sched_clock_local+0x1d/0x90
 [<ffffffff81044dec>] kernel_thread+0x6c/0x70
 [<ffffffff81084430>] ? proc_cap_handler+0x180/0x180
 [<ffffffff814cdd30>] ? gs_change+0xb/0xb
 [<ffffffff810845a1>] __call_usermodehelper+0x31/0xa0
 [<ffffffff810869ba>] process_one_work+0x11a/0x430
 [<ffffffff81084570>] ? call_usermodehelper_freeinfo+0x30/0x30
 [<ffffffff81087026>] worker_thread+0x126/0x2d0
 [<ffffffff81086f00>] ? rescuer_thread+0x1f0/0x1f0
 [<ffffffff8108bb87>] kthread+0x87/0x90
 [<ffffffff814cdd34>] kernel_thread_helper+0x4/0x10
 [<ffffffff8108bb00>] ? kthread_flush_work_fn+0x10/0x10
 [<ffffffff814cdd30>] ? gs_change+0xb/0xb
FIX task_struct: Restoring 0xffff880215c43800-0xffff880215c43803=0x6b

FIX task_struct: Marking all objects used
...
debug: unmapping init memory ffffffff8187d000..ffffffff818ea000
Write protecting the kernel read-only data: 8192k
debug: unmapping init memory ffff8800014d1000..ffff880001600000
debug: unmapping init memory ffff8800017e0000..ffff880001800000
...

slabinfo -v gives:

SLUB: task_struct 10 slabs counted but counter=11

Markus Trippelsdorf Nov. 21, 2011, 4:34 p.m. UTC | #8

On 2011.11.21 at 17:10 +0100, Markus Trippelsdorf wrote:
> On 2011.11.21 at 16:48 +0100, Eric Dumazet wrote:
> > Le lundi 21 novembre 2011 à 16:36 +0100, Markus Trippelsdorf a écrit :
> > > On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> > > > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> > > > 
> > > > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> > > > > 
> > > > 
> > > > Thanks
> > > > 
> > > > Please continue to provide more samples.
> > > > 
> > > > There is something wrong somewhere, but where exactly, its hard to say.
> > > 
> > > New sample. This one points to lib/idr.c:
> > > 
> > > =============================================================================
> > > BUG idr_layer_cache: Poison overwritten
> > > -----------------------------------------------------------------------------
> > 
> > Thanks, could you now add "CONFIG_DEBUG_PAGEALLOC=y" in your config as
> > well ?
> 
> Sure. This one happend with CONFIG_DEBUG_PAGEALLOC=y:
> 
> =============================================================================
> BUG task_struct: Poison overwritten
> -----------------------------------------------------------------------------

And sometimes this one that I've reported earlier already:

(see: http://thread.gmane.org/gmane.linux.kernel/1215023 )

 ------------[ cut here ]------------
 WARNING: at fs/sysfs/sysfs.h:195 sysfs_get_inode+0x136/0x140()
 Hardware name: System Product Name
 Pid: 1876, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #72
 Call Trace:
 [<ffffffff8106cac5>] warn_slowpath_common+0x75/0xb0
 [<ffffffff8106cbc5>] warn_slowpath_null+0x15/0x20
 [<ffffffff81163236>] sysfs_get_inode+0x136/0x140
 [<ffffffff81164cef>] sysfs_lookup+0x6f/0x110
 [<ffffffff811173f9>] d_alloc_and_lookup+0x39/0x80
 [<ffffffff81118774>] do_lookup+0x294/0x3a0
 [<ffffffff8111798a>] ? inode_permission+0x7a/0xb0
 [<ffffffff8111a3f7>] do_last.isra.46+0x137/0x7f0
 [<ffffffff8111ab76>] path_openat+0xc6/0x370
 [<ffffffff81117606>] ? getname_flags+0x36/0x230
 [<ffffffff810ec852>] ? handle_mm_fault+0x192/0x290
 [<ffffffff8111ae5c>] do_filp_open+0x3c/0x90
 [<ffffffff81127c8c>] ? alloc_fd+0xdc/0x120
 [<ffffffff8110ce77>] do_sys_open+0xe7/0x1c0
 [<ffffffff8110cf6b>] sys_open+0x1b/0x20
 [<ffffffff814ccb7b>] system_call_fastpath+0x16/0x1b
 ---[ end trace b1377eb8b131d37d ]---

Markus Trippelsdorf Nov. 22, 2011, 8:36 a.m. UTC | #9

On 2011.11.21 at 17:34 +0100, Markus Trippelsdorf wrote:
> On 2011.11.21 at 17:10 +0100, Markus Trippelsdorf wrote:
> > On 2011.11.21 at 16:48 +0100, Eric Dumazet wrote:
> > > Le lundi 21 novembre 2011 à 16:36 +0100, Markus Trippelsdorf a écrit :
> > > > On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> > > > > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> > > > > 
> > > > > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> > > > > > 
> > > > > 
> > > > > Thanks
> > > > > 
> > > > > Please continue to provide more samples.
> > > > > 
> > > > > There is something wrong somewhere, but where exactly, its hard to say.
> > > > 
> > > > New sample. This one points to lib/idr.c:
> > > > 
> > > > =============================================================================
> > > > BUG idr_layer_cache: Poison overwritten
> > > > -----------------------------------------------------------------------------
> > > 
> > > Thanks, could you now add "CONFIG_DEBUG_PAGEALLOC=y" in your config as
> > > well ?
> > 
> > Sure. This one happend with CONFIG_DEBUG_PAGEALLOC=y:
> > 
> > =============================================================================
> > BUG task_struct: Poison overwritten
> > -----------------------------------------------------------------------------
> 
> And sometimes this one that I've reported earlier already:
> 
> (see: http://thread.gmane.org/gmane.linux.kernel/1215023 )
> 
>  ------------[ cut here ]------------
>  WARNING: at fs/sysfs/sysfs.h:195 sysfs_get_inode+0x136/0x140()
>  Hardware name: System Product Name
>  Pid: 1876, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #72
>  Call Trace:
>  [<ffffffff8106cac5>] warn_slowpath_common+0x75/0xb0
>  [<ffffffff8106cbc5>] warn_slowpath_null+0x15/0x20
>  [<ffffffff81163236>] sysfs_get_inode+0x136/0x140
>  [<ffffffff81164cef>] sysfs_lookup+0x6f/0x110
>  [<ffffffff811173f9>] d_alloc_and_lookup+0x39/0x80
>  [<ffffffff81118774>] do_lookup+0x294/0x3a0
>  [<ffffffff8111798a>] ? inode_permission+0x7a/0xb0
>  [<ffffffff8111a3f7>] do_last.isra.46+0x137/0x7f0
>  [<ffffffff8111ab76>] path_openat+0xc6/0x370
>  [<ffffffff81117606>] ? getname_flags+0x36/0x230
>  [<ffffffff810ec852>] ? handle_mm_fault+0x192/0x290
>  [<ffffffff8111ae5c>] do_filp_open+0x3c/0x90
>  [<ffffffff81127c8c>] ? alloc_fd+0xdc/0x120
>  [<ffffffff8110ce77>] do_sys_open+0xe7/0x1c0
>  [<ffffffff8110cf6b>] sys_open+0x1b/0x20
>  [<ffffffff814ccb7b>] system_call_fastpath+0x16/0x1b
>  ---[ end trace b1377eb8b131d37d ]---

Hm, the "sysfs: use rb-tree" thing hit again during boot. Could this be
the root cause of this all?

I wrote down the following:

RIP : rb_next

Trace:
 sysfs_dir_pos
 sysfs_readdir
 ? sys_ioctl
 vfs_readdir
 sys_getdents

Markus Trippelsdorf Nov. 23, 2011, 4:03 p.m. UTC | #10

On 2011.11.21 at 16:36 +0100, Markus Trippelsdorf wrote:
> On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> > 
> > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> > > 
> > 
> > Thanks
> > 
> > Please continue to provide more samples.
> > 
> > There is something wrong somewhere, but where exactly, its hard to say.
> 
> New sample. This one points to lib/idr.c:
> 
> [drm] Initialized drm 1.1.0 20060810
> [drm] radeon defaulting to kernel modesetting.
> [drm] radeon kernel modesetting enabled.
> radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
> radeon 0000:01:05.0: setting latency timer to 64
> [drm] initializing kernel modesetting (RS780 0x1002:0x9614 0x1043:0x834D).
> [drm] register mmio base: 0xFBEE0000
> [drm] register mmio size: 65536
> ATOM BIOS: 113
> radeon 0000:01:05.0: VRAM: 128M 0x00000000C0000000 - 0x00000000C7FFFFFF (128M used)
> radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
> [drm] Detected VRAM RAM=128M, BAR=128M
> [drm] RAM width 32bits DDR
> [TTM] Zone  kernel: Available graphics memory: 4083428 kiB.
> [TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
> [TTM] Initializing pool allocator.
> [drm] radeon: 128M of VRAM memory ready
> [drm] radeon: 512M of GTT memory ready.
> [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
> [drm] Driver supports precise vblank timestamp query.
> [drm] radeon: irq initialized.
> [drm] GART: num cpu pages 131072, num gpu pages 131072
> [drm] Loading RS780 Microcode
> [drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
> radeon 0000:01:05.0: WB enabled
> [drm] ring test succeeded in 1 usecs
> [drm] radeon: ib pool ready.
> [drm] ib test succeeded in 0 usecs
> =============================================================================
> BUG idr_layer_cache: Poison overwritten
> -----------------------------------------------------------------------------
> 
> INFO: 0xffff880215650800-0xffff880215650803. First byte 0x0 instead of 0x6b
> INFO: Slab 0xffffea0008559400 objects=18 used=18 fp=0x          (null) flags=0x4000000000004080
> INFO: Object 0xffff8802156506d0 @offset=1744 fp=0xffff880215650a38
> 
> Bytes b4 ffff8802156506c0: a4 6f fb ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  .o......ZZZZZZZZ
> Object ffff8802156506d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156506e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156506f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156507f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> Object ffff880215650810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215650890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156508e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
> Redzone ffff8802156508f0: bb bb bb bb bb bb bb bb                          ........
> Padding ffff880215650a30: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
> Pid: 1, comm: swapper Not tainted 3.2.0-rc2-00274-g6fe4c6d #71
> Call Trace:
>  [<ffffffff81101cf8>] ? print_section+0x38/0x40
>  [<ffffffff811021f3>] print_trailer+0xe3/0x150
>  [<ffffffff811023f0>] check_bytes_and_report+0xe0/0x100
>  [<ffffffff811031e6>] check_object+0x1c6/0x240
>  [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
>  [<ffffffff814c5c43>] alloc_debug_processing+0x62/0xe4
>  [<ffffffff814c64f1>] __slab_alloc.constprop.69+0x1a4/0x1e0
>  [<ffffffff8129ae77>] ? drm_property_create+0x47/0x110
>  [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
>  [<ffffffff81104db1>] kmem_cache_alloc+0x121/0x150
>  [<ffffffff812031f0>] ? idr_pre_get+0x60/0x90
>  [<ffffffff812031f0>] idr_pre_get+0x60/0x90
>  [<ffffffff8129870a>] drm_mode_object_get+0x6a/0xc0
>  [<ffffffff8129ae95>] drm_property_create+0x65/0x110
>  [<ffffffff8129b15d>] drm_mode_config_init+0xfd/0x190
>  [<ffffffff812e12ad>] radeon_modeset_init+0x1d/0x860
>  [<ffffffff813211c7>] ? radeon_acpi_init+0x87/0xc0
>  [<ffffffff812c37b8>] radeon_driver_load_kms+0xf8/0x150
>  [<ffffffff81295a06>] drm_get_pci_dev+0x186/0x2d0
>  [<ffffffff814bf1fd>] ? radeon_pci_probe+0x9e/0xb8
>  [<ffffffff814bf20f>] radeon_pci_probe+0xb0/0xb8
>  [<ffffffff8121be15>] pci_device_probe+0x75/0xa0
>  [<ffffffff81324e1a>] ? driver_sysfs_add+0x7a/0xb0
>  [<ffffffff81325021>] driver_probe_device+0x71/0x190
>  [<ffffffff813251db>] __driver_attach+0x9b/0xa0
>  [<ffffffff81325140>] ? driver_probe_device+0x190/0x190
>  [<ffffffff81323e0d>] bus_for_each_dev+0x4d/0x90
>  [<ffffffff813252f9>] driver_attach+0x19/0x20
>  [<ffffffff81324598>] bus_add_driver+0x188/0x250
>  [<ffffffff81325942>] driver_register+0x72/0x150
>  [<ffffffff81321de5>] ? device_add+0x75/0x600
>  [<ffffffff8121bb7d>] __pci_register_driver+0x5d/0xd0
>  [<ffffffff81295c54>] drm_pci_init+0x104/0x120
>  [<ffffffff818abefa>] ? ttm_init+0x62/0x62
>  [<ffffffff818abfe1>] radeon_init+0xe7/0xe9
>  [<ffffffff81890883>] do_one_initcall+0x7a/0x129
>  [<ffffffff818909cc>] kernel_init+0x9a/0x114
>  [<ffffffff814cddb4>] kernel_thread_helper+0x4/0x10
>  [<ffffffff81890932>] ? do_one_initcall+0x129/0x129
>  [<ffffffff814cddb0>] ? gs_change+0xb/0xb
> FIX idr_layer_cache: Restoring 0xffff880215650800-0xffff880215650803=0x6b
> 
> FIX idr_layer_cache: Marking all objects used

Yesterday I couldn't reproduce the issue at all. But today I've hit
exactly the same spot again. (CCing the drm list)


[drm] Initialized drm 1.1.0 20060810
[drm] radeon defaulting to kernel modesetting.
[drm] radeon kernel modesetting enabled.
radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
radeon 0000:01:05.0: setting latency timer to 64
[drm] initializing kernel modesetting (RS780 0x1002:0x9614 0x1043:0x834D).
[drm] register mmio base: 0xFBEE0000
[drm] register mmio size: 65536
ATOM BIOS: 113
radeon 0000:01:05.0: VRAM: 128M 0x00000000C0000000 - 0x00000000C7FFFFFF (128M used)
radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[drm] Detected VRAM RAM=128M, BAR=128M
[drm] RAM width 32bits DDR
[TTM] Zone  kernel: Available graphics memory: 4083428 kiB.
[TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
[TTM] Initializing pool allocator.
[drm] radeon: 128M of VRAM memory ready
[drm] radeon: 512M of GTT memory ready.
[drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[drm] Driver supports precise vblank timestamp query.
[drm] radeon: irq initialized.
[drm] GART: num cpu pages 131072, num gpu pages 131072
[drm] Loading RS780 Microcode
[drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
radeon 0000:01:05.0: WB enabled
[drm] ring test succeeded in 1 usecs
[drm] radeon: ib pool ready.
[drm] ib test succeeded in 0 usecs
=============================================================================
BUG idr_layer_cache: Poison overwritten
-----------------------------------------------------------------------------

INFO: 0xffff880215648800-0xffff880215648803. First byte 0x0 instead of 0x6b
INFO: Slab 0xffffea0008559200 objects=18 used=18 fp=0x          (null) flags=0x4000000000004080
INFO: Object 0xffff8802156486d0 @offset=1744 fp=0xffff880215648a38

Bytes b4 ffff8802156486c0: 66 75 fb ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  fu......ZZZZZZZZ
Object ffff8802156486d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156486e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156486f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648700: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648710: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648720: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648730: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648740: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648750: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648760: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648770: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648780: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648790: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156487a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156487b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156487c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156487d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156487e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156487f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
Object ffff880215648810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff880215648890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156488a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156488b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156488c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156488d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802156488e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Redzone ffff8802156488f0: bb bb bb bb bb bb bb bb                          ........
Padding ffff880215648a30: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
Pid: 1, comm: swapper Not tainted 3.2.0-rc2-00438-g4d8c62a-dirty #105
Call Trace:
 [<ffffffff81101fb8>] ? print_section+0x38/0x40
 [<ffffffff811024b3>] print_trailer+0xe3/0x150
 [<ffffffff811026b0>] check_bytes_and_report+0xe0/0x100
 [<ffffffff811034a6>] check_object+0x1c6/0x240
 [<ffffffff81203550>] ? idr_pre_get+0x60/0x90
 [<ffffffff814c5fab>] alloc_debug_processing+0x62/0xe4
 [<ffffffff814c6887>] __slab_alloc.constprop.69+0x1a4/0x1e0
 [<ffffffff8129b287>] ? drm_property_create+0x47/0x110
 [<ffffffff81203550>] ? idr_pre_get+0x60/0x90
 [<ffffffff81105071>] kmem_cache_alloc+0x121/0x150
 [<ffffffff81203550>] ? idr_pre_get+0x60/0x90
 [<ffffffff81203550>] idr_pre_get+0x60/0x90
 [<ffffffff81298b0a>] drm_mode_object_get+0x6a/0xc0
 [<ffffffff8129b2a5>] drm_property_create+0x65/0x110
 [<ffffffff8129b56d>] drm_mode_config_init+0xfd/0x190
 [<ffffffff812e157d>] radeon_modeset_init+0x1d/0x860
 [<ffffffff81321547>] ? radeon_acpi_init+0x87/0xc0
 [<ffffffff812c3bc8>] radeon_driver_load_kms+0xf8/0x150
 [<ffffffff81295e06>] drm_get_pci_dev+0x186/0x2d0
 [<ffffffff814bf56d>] ? radeon_pci_probe+0x9e/0xb8
 [<ffffffff814bf57f>] radeon_pci_probe+0xb0/0xb8
 [<ffffffff8121c175>] pci_device_probe+0x75/0xa0
 [<ffffffff8132519a>] ? driver_sysfs_add+0x7a/0xb0
 [<ffffffff813253a1>] driver_probe_device+0x71/0x190
 [<ffffffff8132555b>] __driver_attach+0x9b/0xa0
 [<ffffffff813254c0>] ? driver_probe_device+0x190/0x190
 [<ffffffff8132418d>] bus_for_each_dev+0x4d/0x90
 [<ffffffff81325679>] driver_attach+0x19/0x20
 [<ffffffff81324918>] bus_add_driver+0x188/0x250
 [<ffffffff81325cc2>] driver_register+0x72/0x150
 [<ffffffff81322165>] ? device_add+0x75/0x600
 [<ffffffff8121bedd>] __pci_register_driver+0x5d/0xd0
 [<ffffffff81296054>] drm_pci_init+0x104/0x120
 [<ffffffff818abf51>] ? ttm_init+0x62/0x62
 [<ffffffff818ac038>] radeon_init+0xe7/0xe9
 [<ffffffff81890883>] do_one_initcall+0x7a/0x129
 [<ffffffff818909cc>] kernel_init+0x9a/0x114
 [<ffffffff814ce174>] kernel_thread_helper+0x4/0x10
 [<ffffffff81890932>] ? do_one_initcall+0x129/0x129
 [<ffffffff814ce170>] ? gs_change+0xb/0xb
FIX idr_layer_cache: Restoring 0xffff880215648800-0xffff880215648803=0x6b

FIX idr_layer_cache: Marking all objects used
[drm] Radeon Display Connectors
[drm] Connector 0:
[drm]   VGA
[drm]   DDC: 0x7e40 0x7e40 0x7e44 0x7e44 0x7e48 0x7e48 0x7e4c 0x7e4c
[drm]   Encoders:
[drm]     CRT1: INTERNAL_KLDSCP_DAC1
[drm] Connector 1:
[drm]   DVI-D
[drm]   HPD3
[drm]   DDC: 0x7e50 0x7e50 0x7e54 0x7e54 0x7e58 0x7e58 0x7e5c 0x7e5c
[drm]   Encoders:
[drm]     DFP3: INTERNAL_KLDSCP_LVTMA
[drm] radeon: power management initialized
[drm] fb mappable at 0xF0142000
[drm] vram apper at 0xF0000000
[drm] size 7299072
[drm] fb depth is 24
[drm]    pitch is 6912
fbcon: radeondrmfb (fb0) is primary device
Console: switching to colour frame buffer device 131x105
fb0: radeondrmfb frame buffer device
drm: registered panic notifier
[drm] Initialized radeon 2.12.0 20080528 for 0000:01:05.0 on minor 0

Christoph Lameter (Ampere) Nov. 23, 2011, 4:06 p.m. UTC | #11

On Wed, 23 Nov 2011, Markus Trippelsdorf wrote:

> > FIX idr_layer_cache: Marking all objects used
>
> Yesterday I couldn't reproduce the issue at all. But today I've hit
> exactly the same spot again. (CCing the drm list)

Well this is looks like write after free.

> =============================================================================
> BUG idr_layer_cache: Poison overwritten
> -----------------------------------------------------------------------------
> Object ffff8802156487c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156487d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156487e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff8802156487f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Object ffff880215648800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> Object ffff880215648810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk

And its an integer sized write of 0. If you look at the struct definition
and lookup the offset you should be able to locate the field that
was modified.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Markus Trippelsdorf Nov. 24, 2011, 8:50 a.m. UTC | #12

On 2011.11.23 at 10:06 -0600, Christoph Lameter wrote:
> On Wed, 23 Nov 2011, Markus Trippelsdorf wrote:
> 
> > > FIX idr_layer_cache: Marking all objects used
> >
> > Yesterday I couldn't reproduce the issue at all. But today I've hit
> > exactly the same spot again. (CCing the drm list)
> 
> Well this is looks like write after free.
> 
> > =============================================================================
> > BUG idr_layer_cache: Poison overwritten
> > -----------------------------------------------------------------------------
> > Object ffff8802156487c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > Object ffff8802156487d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > Object ffff8802156487e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > Object ffff8802156487f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > Object ffff880215648800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> > Object ffff880215648810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> 
> And its an integer sized write of 0. If you look at the struct definition
> and lookup the offset you should be able to locate the field that
> was modified.

Here are two more BUGs that seem to point to the same bug:

1)
...
Nov 21 18:30:30 x4 kernel: [drm] radeon: irq initialized.
Nov 21 18:30:30 x4 kernel: [drm] GART: num cpu pages 131072, num gpu pages 131072
Nov 21 18:30:30 x4 kernel: [drm] Loading RS780 Microcode
Nov 21 18:30:30 x4 kernel: [drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
Nov 21 18:30:30 x4 kernel: radeon 0000:01:05.0: WB enabled
Nov 21 18:30:30 x4 kernel: =============================================================================
Nov 21 18:30:30 x4 kernel: BUG task_xstate: Not a valid slab page
Nov 21 18:30:30 x4 kernel: -----------------------------------------------------------------------------
Nov 21 18:30:30 x4 kernel:
Nov 21 18:30:30 x4 kernel: INFO: Slab 0xffffea0000044300 objects=32767 used=65535 fp=0x          (null) flags=0x0401
Nov 21 18:30:30 x4 kernel: Pid: 9, comm: ksoftirqd/1 Not tainted 3.2.0-rc2-00274-g6fe4c6d-dirty #75
Nov 21 18:30:30 x4 kernel: Call Trace:
Nov 21 18:30:30 x4 kernel: [<ffffffff81101c1d>] slab_err+0x7d/0x90
Nov 21 18:30:30 x4 kernel: [<ffffffff8103e29f>] ? dump_trace+0x16f/0x2e0
Nov 21 18:30:30 x4 kernel: [<ffffffff81044764>] ? free_thread_xstate+0x24/0x40
Nov 21 18:30:30 x4 kernel: [<ffffffff81044764>] ? free_thread_xstate+0x24/0x40
Nov 21 18:30:30 x4 kernel: [<ffffffff81102566>] check_slab+0x96/0xc0
Nov 21 18:30:30 x4 kernel: [<ffffffff814c5c29>] free_debug_processing+0x34/0x19c
Nov 21 18:30:30 x4 kernel: [<ffffffff81101d9a>] ? set_track+0x5a/0x190
Nov 21 18:30:30 x4 kernel: [<ffffffff8110cf2b>] ? sys_open+0x1b/0x20
Nov 21 18:30:30 x4 kernel: [<ffffffff814c5e55>] __slab_free+0x33/0x2d0
Nov 21 18:30:30 x4 kernel: [<ffffffff8110cf2b>] ? sys_open+0x1b/0x20
Nov 21 18:30:30 x4 kernel: [<ffffffff81105134>] kmem_cache_free+0x104/0x120
Nov 21 18:30:30 x4 kernel: [<ffffffff81044764>] free_thread_xstate+0x24/0x40
Nov 21 18:30:30 x4 kernel: [<ffffffff81044794>] free_thread_info+0x14/0x30
Nov 21 18:30:30 x4 kernel: [<ffffffff8106a4ff>] free_task+0x2f/0x50
Nov 21 18:30:30 x4 kernel: [<ffffffff8106a5d0>] __put_task_struct+0xb0/0x110
Nov 21 18:30:30 x4 kernel: [<ffffffff8106eb4b>] delayed_put_task_struct+0x3b/0xa0
Nov 21 18:30:30 x4 kernel: [<ffffffff810aa01a>] __rcu_process_callbacks+0x12a/0x350
Nov 21 18:30:30 x4 kernel: [<ffffffff810aa2a2>] rcu_process_callbacks+0x62/0x140
Nov 21 18:30:30 x4 kernel: [<ffffffff81072e18>] __do_softirq+0xa8/0x200
Nov 21 18:30:30 x4 kernel: [<ffffffff81073077>] run_ksoftirqd+0x107/0x210
Nov 21 18:30:30 x4 kernel: [<ffffffff81072f70>] ? __do_softirq+0x200/0x200
Nov 21 18:30:30 x4 kernel: [<ffffffff8108bb87>] kthread+0x87/0x90
Nov 21 18:30:30 x4 kernel: [<ffffffff814cdcf4>] kernel_thread_helper+0x4/0x10
Nov 21 18:30:30 x4 kernel: [<ffffffff8108bb00>] ? kthread_flush_work_fn+0x10/0x10
Nov 21 18:30:30 x4 kernel: [<ffffffff814cdcf0>] ? gs_change+0xb/0xb
Nov 21 18:30:30 x4 kernel: FIX task_xstate: Object at 0xffffffff8110cf2b not freed
Nov 21 18:30:30 x4 kernel: [drm] ring test succeeded in 1 usecs
Nov 21 18:30:30 x4 kernel: [drm] radeon: ib pool ready.
Nov 21 18:30:30 x4 kernel: [drm] ib test succeeded in 0 usecs
Nov 21 18:30:30 x4 kernel: [drm] Radeon Display Connectors
Nov 21 18:30:30 x4 kernel: [drm] Connector 0

2)
...
Nov 21 17:04:38 x4 kernel: fbcon: radeondrmfb (fb0) is primary device
Nov 21 17:04:38 x4 kernel: Console: switching to colour frame buffer device 131x105
Nov 21 17:04:38 x4 kernel: fb0: radeondrmfb frame buffer device
Nov 21 17:04:38 x4 kernel: drm: registered panic notifier
Nov 21 17:04:38 x4 kernel: [drm] Initialized radeon 2.11.0 20080528 for 0000:01:05.0 on minor 0
Nov 21 17:04:38 x4 kernel: loop: module loaded
Nov 21 17:04:38 x4 kernel: ahci 0000:00:11.0: version 3.0
Nov 21 17:04:38 x4 kernel: ahci 0000:00:11.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
Nov 21 17:04:38 x4 kernel: ahci 0000:00:11.0: AHCI 0001.0100 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
Nov 21 17:04:38 x4 kernel: ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part ccc
Nov 21 17:04:38 x4 kernel: scsi0 : ahci
Nov 21 17:04:38 x4 kernel: scsi1 : ahci
Nov 21 17:04:38 x4 kernel: =============================================================================
Nov 21 17:04:38 x4 kernel: BUG task_struct: Poison overwritten
Nov 21 17:04:38 x4 kernel: -----------------------------------------------------------------------------
Nov 21 17:04:38 x4 kernel:
Nov 21 17:04:38 x4 kernel: INFO: 0xffff880215c43800-0xffff880215c43803. First byte 0x0 instead of 0x6b
Nov 21 17:04:38 x4 kernel: INFO: Allocated in copy_process+0xc4/0xf60 age=168 cpu=1 pid=5
Nov 21 17:04:38 x4 kernel:      __slab_alloc.constprop.70+0x1a4/0x1e0
Nov 21 17:04:38 x4 kernel:      kmem_cache_alloc+0x126/0x160
Nov 21 17:04:38 x4 kernel:      copy_process+0xc4/0xf60
Nov 21 17:04:38 x4 kernel:      do_fork+0x100/0x2b0
Nov 21 17:04:38 x4 kernel:      kernel_thread+0x6c/0x70
Nov 21 17:04:38 x4 kernel:      __call_usermodehelper+0x31/0xa0
Nov 21 17:04:38 x4 kernel:      process_one_work+0x11a/0x430
Nov 21 17:04:38 x4 kernel:      worker_thread+0x126/0x2d0
Nov 21 17:04:38 x4 kernel:      kthread+0x87/0x90
Nov 21 17:04:38 x4 kernel:      kernel_thread_helper+0x4/0x10
Nov 21 17:04:38 x4 kernel: INFO: Freed in free_task+0x3e/0x50 age=156 cpu=2 pid=13
Nov 21 17:04:38 x4 kernel:      __slab_free+0x33/0x2d0
Nov 21 17:04:38 x4 kernel:      kmem_cache_free+0x104/0x120
Nov 21 17:04:38 x4 kernel:      free_task+0x3e/0x50
Nov 21 17:04:38 x4 kernel:      __put_task_struct+0xb0/0x110
Nov 21 17:04:38 x4 kernel:      delayed_put_task_struct+0x3b/0xa0
Nov 21 17:04:38 x4 kernel:      __rcu_process_callbacks+0x12a/0x350
Nov 21 17:04:38 x4 kernel:      rcu_process_callbacks+0x62/0x140
Nov 21 17:04:38 x4 kernel:      __do_softirq+0xa8/0x200
Nov 21 17:04:38 x4 kernel:      run_ksoftirqd+0x107/0x210
Nov 21 17:04:38 x4 kernel:      kthread+0x87/0x90
Nov 21 17:04:38 x4 kernel:      kernel_thread_helper+0x4/0x10
Nov 21 17:04:38 x4 kernel: INFO: Slab 0xffffea0008571000 objects=17 used=17 fp=0x          (null) flags=0x4000000000004080
Nov 21 17:04:38 x4 kernel: INFO: Object 0xffff880215c432c0 @offset=12992 fp=0xffff880215c41d00
Nov 21 17:04:38 x4 kernel:
Nov 21 17:04:38 x4 kernel: Bytes b4 ffff880215c432b0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 17:04:38 x4 kernel: Object ffff880215c432c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
...
Nov 21 17:04:38 x4 kernel: Object ffff880215c437f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c43890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Nov 21 17:04:38 x4 kernel: Object ffff880215c438a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5  kkkkkkkkkkkkkkk.
Nov 21 17:04:38 x4 kernel: Redzone ffff880215c438b0: bb bb bb bb bb bb bb bb                          ........
Nov 21 17:04:38 x4 kernel: Padding ffff880215c439f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
Nov 21 17:04:38 x4 kernel: Pid: 5, comm: kworker/u:0 Not tainted 3.2.0-rc2-00274-g6fe4c6d #72
Nov 21 17:04:38 x4 kernel: Call Trace:
Nov 21 17:04:38 x4 kernel: [<ffffffff81101ca8>] ? print_section+0x38/0x40
Nov 21 17:04:38 x4 kernel: [<ffffffff811021a3>] print_trailer+0xe3/0x150
Nov 21 17:04:38 x4 kernel: [<ffffffff811023a0>] check_bytes_and_report+0xe0/0x100
Nov 21 17:04:38 x4 kernel: [<ffffffff81103196>] check_object+0x1c6/0x240
Nov 21 17:04:38 x4 kernel: [<ffffffff8106b034>] ? copy_process+0xc4/0xf60
Nov 21 17:04:38 x4 kernel: [<ffffffff814c5bb3>] alloc_debug_processing+0x62/0xe4
Nov 21 17:04:38 x4 kernel: [<ffffffff814c6461>] __slab_alloc.constprop.70+0x1a4/0x1e0
Nov 21 17:04:38 x4 kernel: [<ffffffff8106b034>] ? copy_process+0xc4/0xf60
Nov 21 17:04:38 x4 kernel: [<ffffffff814ca12a>] ? schedule+0x3a/0x50
Nov 21 17:04:38 x4 kernel: [<ffffffff81104d66>] kmem_cache_alloc+0x126/0x160
Nov 21 17:04:38 x4 kernel: [<ffffffff8106b034>] ? copy_process+0xc4/0xf60
Nov 21 17:04:38 x4 kernel: [<ffffffff81065f18>] ? enqueue_task_fair+0xf8/0x140
Nov 21 17:04:38 x4 kernel: [<ffffffff8106b034>] copy_process+0xc4/0xf60
Nov 21 17:04:38 x4 kernel: [<ffffffff8106c000>] do_fork+0x100/0x2b0
Nov 21 17:04:38 x4 kernel: [<ffffffff810920fd>] ? sched_clock_local+0x1d/0x90
Nov 21 17:04:38 x4 kernel: [<ffffffff81044dec>] kernel_thread+0x6c/0x70
Nov 21 17:04:38 x4 kernel: [<ffffffff81084430>] ? proc_cap_handler+0x180/0x180
Nov 21 17:04:38 x4 kernel: [<ffffffff814cdd30>] ? gs_change+0xb/0xb
Nov 21 17:04:38 x4 kernel: [<ffffffff810845a1>] __call_usermodehelper+0x31/0xa0
Nov 21 17:04:38 x4 kernel: [<ffffffff810869ba>] process_one_work+0x11a/0x430
Nov 21 17:04:38 x4 kernel: [<ffffffff81084570>] ? call_usermodehelper_freeinfo+0x30/0x30
Nov 21 17:04:38 x4 kernel: [<ffffffff81087026>] worker_thread+0x126/0x2d0
Nov 21 17:04:38 x4 kernel: [<ffffffff81086f00>] ? rescuer_thread+0x1f0/0x1f0
Nov 21 17:04:38 x4 kernel: [<ffffffff8108bb87>] kthread+0x87/0x90
Nov 21 17:04:38 x4 kernel: [<ffffffff814cdd34>] kernel_thread_helper+0x4/0x10
Nov 21 17:04:38 x4 kernel: [<ffffffff8108bb00>] ? kthread_flush_work_fn+0x10/0x10
Nov 21 17:04:38 x4 kernel: [<ffffffff814cdd30>] ? gs_change+0xb/0xb
Nov 21 17:04:38 x4 kernel: FIX task_struct: Restoring 0xffff880215c43800-0xffff880215c43803=0x6b
Nov 21 17:04:38 x4 kernel:
Nov 21 17:04:38 x4 kernel: FIX task_struct: Marking all objects used

Markus Trippelsdorf Dec. 1, 2011, 8:44 a.m. UTC | #13

On 2011.11.24 at 09:50 +0100, Markus Trippelsdorf wrote:
> On 2011.11.23 at 10:06 -0600, Christoph Lameter wrote:
> > On Wed, 23 Nov 2011, Markus Trippelsdorf wrote:
> > 
> > > > FIX idr_layer_cache: Marking all objects used
> > >
> > > Yesterday I couldn't reproduce the issue at all. But today I've hit
> > > exactly the same spot again. (CCing the drm list)
> > 
> > Well this is looks like write after free.
> > 
> > > =============================================================================
> > > BUG idr_layer_cache: Poison overwritten
> > > -----------------------------------------------------------------------------
> > > Object ffff8802156487c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > Object ffff8802156487d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > Object ffff8802156487e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > Object ffff8802156487f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > Object ffff880215648800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> > > Object ffff880215648810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > 
> > And its an integer sized write of 0. If you look at the struct definition
> > and lookup the offset you should be able to locate the field that
> > was modified.

It also happens with CONFIG_SLAB. 
(If someone wants to reproduce the issue, just run a kexec boot loop and
the bug will occur after a few (~10) iterations.)

Dec  1 05:04:52 x4 kernel: [drm] Initialized drm 1.1.0 20060810
Dec  1 05:04:52 x4 kernel: [drm] radeon defaulting to kernel modesetting.
Dec  1 05:04:52 x4 kernel: [drm] radeon kernel modesetting enabled.
Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: setting latency timer to 64
Dec  1 05:04:52 x4 kernel: [drm] initializing kernel modesetting (RS780 0x1002:0x9614 0x1043:0x834D).
Dec  1 05:04:52 x4 kernel: [drm] register mmio base: 0xFBEE0000
Dec  1 05:04:52 x4 kernel: [drm] register mmio size: 65536
Dec  1 05:04:52 x4 kernel: ATOM BIOS: 113
Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: VRAM: 128M 0x00000000C0000000 - 0x00000000C7FFFFFF (128M used)
Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
Dec  1 05:04:52 x4 kernel: [drm] Detected VRAM RAM=128M, BAR=128M
Dec  1 05:04:52 x4 kernel: [drm] RAM width 32bits DDR
Dec  1 05:04:52 x4 kernel: [TTM] Zone  kernel: Available graphics memory: 4090750 kiB.
Dec  1 05:04:52 x4 kernel: [TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
Dec  1 05:04:52 x4 kernel: [TTM] Initializing pool allocator.
Dec  1 05:04:52 x4 kernel: [drm] radeon: 128M of VRAM memory ready
Dec  1 05:04:52 x4 kernel: [drm] radeon: 512M of GTT memory ready.
Dec  1 05:04:52 x4 kernel: [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
Dec  1 05:04:52 x4 kernel: [drm] Driver supports precise vblank timestamp query.
Dec  1 05:04:52 x4 kernel: [drm] radeon: irq initialized.
Dec  1 05:04:52 x4 kernel: [drm] GART: num cpu pages 131072, num gpu pages 131072
Dec  1 05:04:52 x4 kernel: [drm] Loading RS780 Microcode
Dec  1 05:04:52 x4 kernel: [drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: WB enabled
Dec  1 05:04:52 x4 kernel: Slab corruption: size-1024 start=ffff880216cbc730, len=1024
Dec  1 05:04:52 x4 kernel: Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Dec  1 05:04:52 x4 kernel: Last user: [<          (null)>](0x0)
Dec  1 05:04:52 x4 kernel: 0d0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
Dec  1 05:04:52 x4 kernel: Prev obj: start=ffff880216cbc318, len=1024
Dec  1 05:04:52 x4 kernel: Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Dec  1 05:04:52 x4 kernel: Last user: [<          (null)>](0x0)
Dec  1 05:04:52 x4 kernel: 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Dec  1 05:04:52 x4 kernel: 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Dec  1 05:04:52 x4 kernel: Next obj: start=ffff880216cbcb48, len=1024
Dec  1 05:04:52 x4 kernel: Redzone: 0xd84156c5635688c0/0xd84156c5635688c0.
Dec  1 05:04:52 x4 kernel: Last user: [<ffffffff81299874>](radeon_bo_create+0xb4/0x240)
Dec  1 05:04:52 x4 kernel: 000: 48 cb cb 16 02 88 ff ff 48 cb cb 16 02 88 ff ff  H.......H.......
Dec  1 05:04:52 x4 kernel: 010: 02 00 27 00 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
Dec  1 05:04:52 x4 kernel: [drm] ring test succeeded in 0 usecs
Dec  1 05:04:52 x4 kernel: [drm] radeon: ib pool ready.
Dec  1 05:04:52 x4 kernel: [drm] ib test succeeded in 0 usecs
Dec  1 05:04:52 x4 kernel: [drm] Radeon Display Connectors
Dec  1 05:04:52 x4 kernel: [drm] Connector 0:
Dec  1 05:04:52 x4 kernel: [drm]   VGA
Dec  1 05:04:52 x4 kernel: [drm]   DDC: 0x7e40 0x7e40 0x7e44 0x7e44 0x7e48 0x7e48 0x7e4c 0x7e4c
Dec  1 05:04:52 x4 kernel: [drm]   Encoders:
Dec  1 05:04:52 x4 kernel: [drm]     CRT1: INTERNAL_KLDSCP_DAC1
Dec  1 05:04:52 x4 kernel: [drm] Connector 1:
Dec  1 05:04:52 x4 kernel: [drm]   DVI-D
Dec  1 05:04:52 x4 kernel: [drm]   HPD3
Dec  1 05:04:52 x4 kernel: [drm]   DDC: 0x7e50 0x7e50 0x7e54 0x7e54 0x7e58 0x7e58 0x7e5c 0x7e5c
Dec  1 05:04:52 x4 kernel: [drm]   Encoders:
Dec  1 05:04:52 x4 kernel: [drm]     DFP3: INTERNAL_KLDSCP_LVTMA
Dec  1 05:04:52 x4 kernel: [drm] radeon: power management initialized
Dec  1 05:04:52 x4 kernel: [drm] fb mappable at 0xF0142000
Dec  1 05:04:52 x4 kernel: [drm] vram apper at 0xF0000000
Dec  1 05:04:52 x4 kernel: [drm] size 7299072
Dec  1 05:04:52 x4 kernel: [drm] fb depth is 24
Dec  1 05:04:52 x4 kernel: [drm]    pitch is 6912
Dec  1 05:04:52 x4 kernel: fbcon: radeondrmfb (fb0) is primary device
Dec  1 05:04:52 x4 kernel: Console: switching to colour frame buffer device 131x105
Dec  1 05:04:52 x4 kernel: fb0: radeondrmfb frame buffer device
Dec  1 05:04:52 x4 kernel: drm: registered panic notifier
Dec  1 05:04:52 x4 kernel: [drm] Initialized radeon 2.12.0 20080528 for 0000:01:05.0 on minor 0


Dec  1 05:09:35 x4 kernel: radeon 0000:01:05.0: WB enabled
Dec  1 05:09:35 x4 kernel: [drm] ring test succeeded in 1 usecs
Dec  1 05:09:35 x4 kernel: [drm] radeon: ib pool ready.
Dec  1 05:09:35 x4 kernel: [drm] ib test succeeded in 0 usecs
Dec  1 05:09:35 x4 kernel: Slab corruption: size-512 start=ffff880216f7e760, len=512
Dec  1 05:09:35 x4 kernel: Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Dec  1 05:09:35 x4 kernel: Last user: [<          (null)>](0x0)
Dec  1 05:09:35 x4 kernel: 0a0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
Dec  1 05:09:35 x4 kernel: Prev obj: start=ffff880216f7e548, len=512
Dec  1 05:09:35 x4 kernel: Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Dec  1 05:09:35 x4 kernel: Last user: [<          (null)>](0x0)
Dec  1 05:09:35 x4 kernel: 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Dec  1 05:09:35 x4 kernel: 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Dec  1 05:09:35 x4 kernel: Next obj: start=ffff880216f7e978, len=512
Dec  1 05:09:35 x4 kernel: Redzone: 0xd84156c5635688c0/0xd84156c5635688c0.
Dec  1 05:09:35 x4 kernel: Last user: [<ffffffff812e519d>](radeon_add_atom_encoder+0x7d/0x280)
Dec  1 05:09:35 x4 kernel: 000: f8 d3 f6 16 02 88 ff ff 18 d8 f6 16 02 88 ff ff  ................
Dec  1 05:09:35 x4 kernel: 010: 18 d8 f6 16 02 88 ff ff 0c 00 00 00 e0 e0 e0 e0  ................
Dec  1 05:09:35 x4 kernel: [drm] Radeon Display Connectors
Dec  1 05:09:35 x4 kernel: [drm] Connector 0:
Dec  1 05:09:35 x4 kernel: [drm]   VGA
Dec  1 05:09:35 x4 kernel: [drm]   DDC: 0x7e40 0x7e40 0x7e44 0x7e44 0x7e48 0x7e48 0x7e4c 0x7e4c
Dec  1 05:09:35 x4 kernel: [drm]   Encoders:
Dec  1 05:09:35 x4 kernel: [drm]     CRT1: INTERNAL_KLDSCP_DAC1
Dec  1 05:09:35 x4 kernel: [drm] Connector 1:
Dec  1 05:09:35 x4 kernel: [drm]   DVI-D
Dec  1 05:09:35 x4 kernel: [drm]   HPD3
Dec  1 05:09:35 x4 kernel: [drm]   DDC: 0x7e50 0x7e50 0x7e54 0x7e54 0x7e58 0x7e58 0x7e5c 0x7e5c
Dec  1 05:09:35 x4 kernel: [drm]   Encoders:
Dec  1 05:09:35 x4 kernel: [drm]     DFP3: INTERNAL_KLDSCP_LVTMA
Dec  1 05:09:35 x4 kernel: [drm] radeon: power management initialized
Dec  1 05:09:35 x4 kernel: [drm] fb mappable at 0xF0142000
Dec  1 05:09:35 x4 kernel: [drm] vram apper at 0xF0000000
Dec  1 05:09:35 x4 kernel: [drm] size 7299072
Dec  1 05:09:35 x4 kernel: [drm] fb depth is 24
Dec  1 05:09:35 x4 kernel: [drm]    pitch is 6912
Dec  1 05:09:35 x4 kernel: fbcon: radeondrmfb (fb0) is primary device
Dec  1 05:09:35 x4 kernel: Console: switching to colour frame buffer device 131x105
Dec  1 05:09:35 x4 kernel: fb0: radeondrmfb frame buffer device
Dec  1 05:09:35 x4 kernel: drm: registered panic notifier
Dec  1 05:09:35 x4 kernel: [drm] Initialized radeon 2.12.0 20080528 for 0000:01:05.0 on minor 0

Pekka Enberg Dec. 1, 2011, 8:53 a.m. UTC | #14

On Thu, 1 Dec 2011, Markus Trippelsdorf wrote:
> On 2011.11.24 at 09:50 +0100, Markus Trippelsdorf wrote:
>> On 2011.11.23 at 10:06 -0600, Christoph Lameter wrote:
>>> On Wed, 23 Nov 2011, Markus Trippelsdorf wrote:
>>>
>>>>> FIX idr_layer_cache: Marking all objects used
>>>>
>>>> Yesterday I couldn't reproduce the issue at all. But today I've hit
>>>> exactly the same spot again. (CCing the drm list)
>>>
>>> Well this is looks like write after free.
>>>
>>>> =============================================================================
>>>> BUG idr_layer_cache: Poison overwritten
>>>> -----------------------------------------------------------------------------
>>>> Object ffff8802156487c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
>>>> Object ffff8802156487d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
>>>> Object ffff8802156487e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
>>>> Object ffff8802156487f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
>>>> Object ffff880215648800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
>>>> Object ffff880215648810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
>>>
>>> And its an integer sized write of 0. If you look at the struct definition
>>> and lookup the offset you should be able to locate the field that
>>> was modified.
>
> It also happens with CONFIG_SLAB.
> (If someone wants to reproduce the issue, just run a kexec boot loop and
> the bug will occur after a few (~10) iterations.)
>
> Dec  1 05:04:52 x4 kernel: [drm] Initialized drm 1.1.0 20060810
> Dec  1 05:04:52 x4 kernel: [drm] radeon defaulting to kernel modesetting.
> Dec  1 05:04:52 x4 kernel: [drm] radeon kernel modesetting enabled.
> Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
> Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: setting latency timer to 64
> Dec  1 05:04:52 x4 kernel: [drm] initializing kernel modesetting (RS780 0x1002:0x9614 0x1043:0x834D).
> Dec  1 05:04:52 x4 kernel: [drm] register mmio base: 0xFBEE0000
> Dec  1 05:04:52 x4 kernel: [drm] register mmio size: 65536
> Dec  1 05:04:52 x4 kernel: ATOM BIOS: 113
> Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: VRAM: 128M 0x00000000C0000000 - 0x00000000C7FFFFFF (128M used)
> Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
> Dec  1 05:04:52 x4 kernel: [drm] Detected VRAM RAM=128M, BAR=128M
> Dec  1 05:04:52 x4 kernel: [drm] RAM width 32bits DDR
> Dec  1 05:04:52 x4 kernel: [TTM] Zone  kernel: Available graphics memory: 4090750 kiB.
> Dec  1 05:04:52 x4 kernel: [TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
> Dec  1 05:04:52 x4 kernel: [TTM] Initializing pool allocator.
> Dec  1 05:04:52 x4 kernel: [drm] radeon: 128M of VRAM memory ready
> Dec  1 05:04:52 x4 kernel: [drm] radeon: 512M of GTT memory ready.
> Dec  1 05:04:52 x4 kernel: [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
> Dec  1 05:04:52 x4 kernel: [drm] Driver supports precise vblank timestamp query.
> Dec  1 05:04:52 x4 kernel: [drm] radeon: irq initialized.
> Dec  1 05:04:52 x4 kernel: [drm] GART: num cpu pages 131072, num gpu pages 131072
> Dec  1 05:04:52 x4 kernel: [drm] Loading RS780 Microcode
> Dec  1 05:04:52 x4 kernel: [drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
> Dec  1 05:04:52 x4 kernel: radeon 0000:01:05.0: WB enabled
> Dec  1 05:04:52 x4 kernel: Slab corruption: size-1024 start=ffff880216cbc730, len=1024
> Dec  1 05:04:52 x4 kernel: Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
> Dec  1 05:04:52 x4 kernel: Last user: [<          (null)>](0x0)
> Dec  1 05:04:52 x4 kernel: 0d0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> Dec  1 05:04:52 x4 kernel: Prev obj: start=ffff880216cbc318, len=1024
> Dec  1 05:04:52 x4 kernel: Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
> Dec  1 05:04:52 x4 kernel: Last user: [<          (null)>](0x0)
> Dec  1 05:04:52 x4 kernel: 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Dec  1 05:04:52 x4 kernel: 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Dec  1 05:04:52 x4 kernel: Next obj: start=ffff880216cbcb48, len=1024
> Dec  1 05:04:52 x4 kernel: Redzone: 0xd84156c5635688c0/0xd84156c5635688c0.
> Dec  1 05:04:52 x4 kernel: Last user: [<ffffffff81299874>](radeon_bo_create+0xb4/0x240)
> Dec  1 05:04:52 x4 kernel: 000: 48 cb cb 16 02 88 ff ff 48 cb cb 16 02 88 ff ff  H.......H.......
> Dec  1 05:04:52 x4 kernel: 010: 02 00 27 00 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
> Dec  1 05:04:52 x4 kernel: [drm] ring test succeeded in 0 usecs
> Dec  1 05:04:52 x4 kernel: [drm] radeon: ib pool ready.
> Dec  1 05:04:52 x4 kernel: [drm] ib test succeeded in 0 usecs
> Dec  1 05:04:52 x4 kernel: [drm] Radeon Display Connectors
> Dec  1 05:04:52 x4 kernel: [drm] Connector 0:
> Dec  1 05:04:52 x4 kernel: [drm]   VGA
> Dec  1 05:04:52 x4 kernel: [drm]   DDC: 0x7e40 0x7e40 0x7e44 0x7e44 0x7e48 0x7e48 0x7e4c 0x7e4c
> Dec  1 05:04:52 x4 kernel: [drm]   Encoders:
> Dec  1 05:04:52 x4 kernel: [drm]     CRT1: INTERNAL_KLDSCP_DAC1
> Dec  1 05:04:52 x4 kernel: [drm] Connector 1:
> Dec  1 05:04:52 x4 kernel: [drm]   DVI-D
> Dec  1 05:04:52 x4 kernel: [drm]   HPD3
> Dec  1 05:04:52 x4 kernel: [drm]   DDC: 0x7e50 0x7e50 0x7e54 0x7e54 0x7e58 0x7e58 0x7e5c 0x7e5c
> Dec  1 05:04:52 x4 kernel: [drm]   Encoders:
> Dec  1 05:04:52 x4 kernel: [drm]     DFP3: INTERNAL_KLDSCP_LVTMA
> Dec  1 05:04:52 x4 kernel: [drm] radeon: power management initialized
> Dec  1 05:04:52 x4 kernel: [drm] fb mappable at 0xF0142000
> Dec  1 05:04:52 x4 kernel: [drm] vram apper at 0xF0000000
> Dec  1 05:04:52 x4 kernel: [drm] size 7299072
> Dec  1 05:04:52 x4 kernel: [drm] fb depth is 24
> Dec  1 05:04:52 x4 kernel: [drm]    pitch is 6912
> Dec  1 05:04:52 x4 kernel: fbcon: radeondrmfb (fb0) is primary device
> Dec  1 05:04:52 x4 kernel: Console: switching to colour frame buffer device 131x105
> Dec  1 05:04:52 x4 kernel: fb0: radeondrmfb frame buffer device
> Dec  1 05:04:52 x4 kernel: drm: registered panic notifier
> Dec  1 05:04:52 x4 kernel: [drm] Initialized radeon 2.12.0 20080528 for 0000:01:05.0 on minor 0
>
>
> Dec  1 05:09:35 x4 kernel: radeon 0000:01:05.0: WB enabled
> Dec  1 05:09:35 x4 kernel: [drm] ring test succeeded in 1 usecs
> Dec  1 05:09:35 x4 kernel: [drm] radeon: ib pool ready.
> Dec  1 05:09:35 x4 kernel: [drm] ib test succeeded in 0 usecs
> Dec  1 05:09:35 x4 kernel: Slab corruption: size-512 start=ffff880216f7e760, len=512
> Dec  1 05:09:35 x4 kernel: Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
> Dec  1 05:09:35 x4 kernel: Last user: [<          (null)>](0x0)
> Dec  1 05:09:35 x4 kernel: 0a0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> Dec  1 05:09:35 x4 kernel: Prev obj: start=ffff880216f7e548, len=512
> Dec  1 05:09:35 x4 kernel: Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
> Dec  1 05:09:35 x4 kernel: Last user: [<          (null)>](0x0)
> Dec  1 05:09:35 x4 kernel: 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Dec  1 05:09:35 x4 kernel: 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> Dec  1 05:09:35 x4 kernel: Next obj: start=ffff880216f7e978, len=512
> Dec  1 05:09:35 x4 kernel: Redzone: 0xd84156c5635688c0/0xd84156c5635688c0.
> Dec  1 05:09:35 x4 kernel: Last user: [<ffffffff812e519d>](radeon_add_atom_encoder+0x7d/0x280)
> Dec  1 05:09:35 x4 kernel: 000: f8 d3 f6 16 02 88 ff ff 18 d8 f6 16 02 88 ff ff  ................
> Dec  1 05:09:35 x4 kernel: 010: 18 d8 f6 16 02 88 ff ff 0c 00 00 00 e0 e0 e0 e0  ................
> Dec  1 05:09:35 x4 kernel: [drm] Radeon Display Connectors
> Dec  1 05:09:35 x4 kernel: [drm] Connector 0:
> Dec  1 05:09:35 x4 kernel: [drm]   VGA
> Dec  1 05:09:35 x4 kernel: [drm]   DDC: 0x7e40 0x7e40 0x7e44 0x7e44 0x7e48 0x7e48 0x7e4c 0x7e4c
> Dec  1 05:09:35 x4 kernel: [drm]   Encoders:
> Dec  1 05:09:35 x4 kernel: [drm]     CRT1: INTERNAL_KLDSCP_DAC1
> Dec  1 05:09:35 x4 kernel: [drm] Connector 1:
> Dec  1 05:09:35 x4 kernel: [drm]   DVI-D
> Dec  1 05:09:35 x4 kernel: [drm]   HPD3
> Dec  1 05:09:35 x4 kernel: [drm]   DDC: 0x7e50 0x7e50 0x7e54 0x7e54 0x7e58 0x7e58 0x7e5c 0x7e5c
> Dec  1 05:09:35 x4 kernel: [drm]   Encoders:
> Dec  1 05:09:35 x4 kernel: [drm]     DFP3: INTERNAL_KLDSCP_LVTMA
> Dec  1 05:09:35 x4 kernel: [drm] radeon: power management initialized
> Dec  1 05:09:35 x4 kernel: [drm] fb mappable at 0xF0142000
> Dec  1 05:09:35 x4 kernel: [drm] vram apper at 0xF0000000
> Dec  1 05:09:35 x4 kernel: [drm] size 7299072
> Dec  1 05:09:35 x4 kernel: [drm] fb depth is 24
> Dec  1 05:09:35 x4 kernel: [drm]    pitch is 6912
> Dec  1 05:09:35 x4 kernel: fbcon: radeondrmfb (fb0) is primary device
> Dec  1 05:09:35 x4 kernel: Console: switching to colour frame buffer device 131x105
> Dec  1 05:09:35 x4 kernel: fb0: radeondrmfb frame buffer device
> Dec  1 05:09:35 x4 kernel: drm: registered panic notifier
> Dec  1 05:09:35 x4 kernel: [drm] Initialized radeon 2.12.0 20080528 for 0000:01:05.0 on minor 0

The SLAB debugging output points to DRM and more specifically to the 
radeon driver.

Dave, Alex, could you please take a look at this or fix up the CC to 
include the right people?

 			Pekka
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Jerome Glisse Dec. 2, 2011, 7:43 p.m. UTC | #15

On Thu, Dec 01, 2011 at 09:44:37AM +0100, Markus Trippelsdorf wrote:
> On 2011.11.24 at 09:50 +0100, Markus Trippelsdorf wrote:
> > On 2011.11.23 at 10:06 -0600, Christoph Lameter wrote:
> > > On Wed, 23 Nov 2011, Markus Trippelsdorf wrote:
> > > 
> > > > > FIX idr_layer_cache: Marking all objects used
> > > >
> > > > Yesterday I couldn't reproduce the issue at all. But today I've hit
> > > > exactly the same spot again. (CCing the drm list)
> > > 
> > > Well this is looks like write after free.
> > > 
> > > > =============================================================================
> > > > BUG idr_layer_cache: Poison overwritten
> > > > -----------------------------------------------------------------------------
> > > > Object ffff8802156487c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > Object ffff8802156487d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > Object ffff8802156487e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > Object ffff8802156487f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > Object ffff880215648800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> > > > Object ffff880215648810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > 
> > > And its an integer sized write of 0. If you look at the struct definition
> > > and lookup the offset you should be able to locate the field that
> > > was modified.
> 
> It also happens with CONFIG_SLAB. 
> (If someone wants to reproduce the issue, just run a kexec boot loop and
> the bug will occur after a few (~10) iterations.)
> 

Can you provide the kexec command line you are using and full kernel
log (mostly interested in kernel option).

Cheers,
Jerome
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Markus Trippelsdorf Dec. 2, 2011, 8:06 p.m. UTC | #16

On 2011.12.02 at 14:43 -0500, Jerome Glisse wrote:
> On Thu, Dec 01, 2011 at 09:44:37AM +0100, Markus Trippelsdorf wrote:
> > On 2011.11.24 at 09:50 +0100, Markus Trippelsdorf wrote:
> > > On 2011.11.23 at 10:06 -0600, Christoph Lameter wrote:
> > > > On Wed, 23 Nov 2011, Markus Trippelsdorf wrote:
> > > > 
> > > > > > FIX idr_layer_cache: Marking all objects used
> > > > >
> > > > > Yesterday I couldn't reproduce the issue at all. But today I've hit
> > > > > exactly the same spot again. (CCing the drm list)
> > > > 
> > > > Well this is looks like write after free.
> > > > 
> > > > > =============================================================================
> > > > > BUG idr_layer_cache: Poison overwritten
> > > > > -----------------------------------------------------------------------------
> > > > > Object ffff8802156487c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > Object ffff8802156487d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > Object ffff8802156487e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > Object ffff8802156487f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > Object ffff880215648800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> > > > > Object ffff880215648810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > 
> > > > And its an integer sized write of 0. If you look at the struct definition
> > > > and lookup the offset you should be able to locate the field that
> > > > was modified.
> > 
> > It also happens with CONFIG_SLAB. 
> > (If someone wants to reproduce the issue, just run a kexec boot loop and
> > the bug will occur after a few (~10) iterations.)
> > 
> 
> Can you provide the kexec command line you are using and full kernel
> log (mostly interested in kernel option).

/usr/sbin/kexec -l "/usr/src/linux/arch/x86/boot/bzImage" --append="root=PARTUUID=6d6a4009-3a90-40df-806a-e63f48189719 init=/sbin/minit rootflags=logbsize=262144 fbcon=rotate:3 drm_kms_helper.poll=0 quiet"
/usr/sbin/kexec -e

(The loop happens after autologin in .zprofile:
sleep 4 && sudo /etc/minit/ctrlaltdel/run
(the last script kills, unmounts and then runs the two kexec commands
above))

Linux version 3.2.0-rc4-00089-g621fc1e-dirty (markus@x4.trippels.de) (gcc version 4.6.3 20111202 (prerelease) (GCC) ) #134 SMP PREEMPT Fri Dec 2 11:06:20 CET 2011
Command line: root=PARTUUID=6d6a4009-3a90-40df-806a-e63f48189719 init=/sbin/minit rootflags=logbsize=262144 fbcon=rotate:3 drm_kms_helper.poll=0 quiet
KERNEL supported cpus:
  AMD AuthenticAMD
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000100 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000dfe90000 (usable)
 BIOS-e820: 00000000dfe90000 - 00000000dfea8000 (ACPI data)
 BIOS-e820: 00000000dfea8000 - 00000000dfed0000 (ACPI NVS)
 BIOS-e820: 00000000dfed0000 - 00000000dff00000 (reserved)
 BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000220000000 (usable)
NX (Execute Disable) protection: active
DMI present.
DMI: System manufacturer System Product Name/M4A78T-E, BIOS 3406    08/20/2010
e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x220000 max_arch_pfn = 0x400000000
MTRR default type: uncachable
MTRR fixed ranges enabled:
  00000-9FFFF write-back
  A0000-EFFFF uncachable
  F0000-FFFFF write-protect
MTRR variable ranges enabled:
  0 base 000000000000 mask FFFF80000000 write-back
  1 base 000080000000 mask FFFFC0000000 write-back
  2 base 0000C0000000 mask FFFFE0000000 write-back
  3 base 0000F0000000 mask FFFFF8000000 write-combining
  4 disabled
  5 disabled
  6 disabled
  7 disabled
TOM2: 0000000220000000 aka 8704M
x86 PAT enabled: cpu 0, old 0x7010600070106, new 0x7010600070106
last_pfn = 0xdfe90 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 20000000
Base memory trampoline at [ffff88000009d000] 9d000 size 8192
Using GB pages for direct mapping
init_memory_mapping: 0000000000000000-00000000dfe90000
 0000000000 - 00c0000000 page 1G
 00c0000000 - 00dfe00000 page 2M
 00dfe00000 - 00dfe90000 page 4k
kernel direct mapping tables up to dfe90000 @ 1fffd000-20000000
init_memory_mapping: 0000000100000000-0000000220000000
 0100000000 - 0200000000 page 1G
 0200000000 - 0220000000 page 2M
kernel direct mapping tables up to 220000000 @ dfe8e000-dfe90000
ACPI: RSDP 00000000000fb880 00024 (v02 ACPIAM)
ACPI: XSDT 00000000dfe90100 0005C (v01 082010 XSDT1403 20100820 MSFT 00000097)
ACPI: FACP 00000000dfe90290 000F4 (v03 082010 FACP1403 20100820 MSFT 00000097)
ACPI Warning: Optional field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20110623/tbfadt-560)
ACPI: DSDT 00000000dfe90450 0E6FE (v01  A1152 A1152000 00000000 INTL 20060113)
ACPI: FACS 00000000dfea8000 00040
ACPI: APIC 00000000dfe90390 0007C (v01 082010 APIC1403 20100820 MSFT 00000097)
ACPI: MCFG 00000000dfe90410 0003C (v01 082010 OEMMCFG  20100820 MSFT 00000097)
ACPI: OEMB 00000000dfea8040 00072 (v01 082010 OEMB1403 20100820 MSFT 00000097)
ACPI: SRAT 00000000dfe9f450 000E8 (v01 AMD    FAM_F_10 00000002 AMD  00000001)
ACPI: HPET 00000000dfe9f540 00038 (v01 082010 OEMHPET  20100820 MSFT 00000097)
ACPI: SSDT 00000000dfe9f580 0088C (v01 A M I  POWERNOW 00000001 AMD  00000001)
ACPI: Local APIC address 0xfee00000
 [ffffea0000000000-ffffea00087fffff] PMD -> [ffff880217600000-ffff88021f5fffff] on node 0
Zone PFN ranges:
  DMA32    0x00000010 -> 0x00100000
  Normal   0x00100000 -> 0x00220000
Movable zone start PFN for each node
early_node_map[3] active PFN ranges
    0: 0x00000010 -> 0x0000009f
    0: 0x00000100 -> 0x000dfe90
    0: 0x00100000 -> 0x00220000
On node 0 totalpages: 2096671
  DMA32 zone: 16384 pages used for memmap
  DMA32 zone: 2 pages reserved
  DMA32 zone: 900637 pages, LIFO batch:31
  Normal zone: 18432 pages used for memmap
  Normal zone: 1161216 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled)
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x8300 base: 0xfed00000
SMP: Allowing 4 CPUs, 0 hotplug CPUs
nr_irqs_gsi: 40
Allocating PCI resources starting at dff00000 (gap: dff00000:20000000)
setup_percpu: NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 23 pages/cpu @ffff88021fc00000 s71680 r0 d22528 u524288
pcpu-alloc: s71680 r0 d22528 u524288 alloc=1*2097152
pcpu-alloc: [0] 0 1 2 3 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 2061853
Kernel command line: root=PARTUUID=6d6a4009-3a90-40df-806a-e63f48189719 init=/sbin/minit rootflags=logbsize=262144 fbcon=rotate:3 drm_kms_helper.poll=0 quiet
PID hash table entries: 4096 (order: 3, 32768 bytes)
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Memory: 8167156k/8912896k available (4549k kernel code, 526212k absent, 219528k reserved, 4000k data, 420k init)
SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Preemptible hierarchical RCU implementation.
	Verbose stalled-CPUs detection is disabled.
NR_IRQS:4352 nr_irqs:712 16
Extended CMOS year: 2000
Console: colour VGA+ 80x25
console [tty0] enabled
hpet clockevent registered
Fast TSC calibration using PIT
Detected 3210.181 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 6420.36 BogoMIPS (lpj=3210181)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 256
tseg: 0000000000
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
mce: CPU supports 6 MCE banks
using AMD E400 aware idle routine
Freeing SMP alternatives: 12k freed
ACPI: Core revision 20110623
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: AMD Phenom(tm) II X4 955 Processor stepping 02
Performance Events: AMD PMU driver.
... version:                0
... bit width:              48
... generic registers:      4
... value mask:             0000ffffffffffff
... max period:             00007fffffffffff
... fixed-purpose events:   0
... event mask:             000000000000000f
System has AMD C1E enabled
Switch to broadcast mode on CPU0
MCE: In-kernel MCE decoding enabled.
Booting Node   0, Processors  #1
smpboot cpu 1: start_ip = 9d000
Switch to broadcast mode on CPU1
 #2
smpboot cpu 2: start_ip = 9d000
Switch to broadcast mode on CPU2
 #3 Ok.
smpboot cpu 3: start_ip = 9d000
Brought up 4 CPUs
Total of 4 processors activated (25685.44 BogoMIPS).
Switch to broadcast mode on CPU3
devtmpfs: initialized
NET: Registered protocol family 16
node 0 link 0: io port [1000, ffffff]
TOM: 00000000e0000000 aka 3584M
Fam 10h mmconf [e0000000, efffffff]
node 0 link 0: mmio [a0000, bffff]
node 0 link 0: mmio [e0000000, efffffff] ==> none
node 0 link 0: mmio [f0000000, fbcfffff]
node 0 link 0: mmio [fbd00000, fbefffff]
node 0 link 0: mmio [fbf00000, ffefffff]
TOM2: 0000000220000000 aka 8704M
bus: [00, 07] on node 0 link 0
bus: 00 index 0 [io  0x0000-0xffff]
bus: 00 index 1 [mem 0x000a0000-0x000bffff]
bus: 00 index 2 [mem 0xf0000000-0xffffffff]
bus: 00 index 3 [mem 0x220000000-0xfcffffffff]
Extended Config Space enabled on 1 nodes
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
PCI: Using configuration type 1 for extended access
bio: create slab <bio-0> at 0
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
ACPI: EC: Look up EC in DSDT
ACPI: Executed 3 blocks of module-level executable AML code
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
pci_root PNP0A03:00: host bridge window [io  0x0000-0x0cf7]
pci_root PNP0A03:00: host bridge window [io  0x0d00-0xffff]
pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff]
pci_root PNP0A03:00: host bridge window [mem 0x000d0000-0x000dffff]
pci_root PNP0A03:00: host bridge window [mem 0xdff00000-0xdfffffff]
pci_root PNP0A03:00: host bridge window [mem 0xf0000000-0xfebfffff]
pci 0000:00:00.0: [1022:9600] type 0 class 0x000600
pci 0000:00:01.0: [1022:9602] type 1 class 0x000604
pci 0000:00:06.0: [1022:9606] type 1 class 0x000604
pci 0000:00:06.0: PME# supported from D0 D3hot D3cold
pci 0000:00:06.0: PME# disabled
pci 0000:00:11.0: [1002:4391] type 0 class 0x000106
pci 0000:00:11.0: reg 10: [io  0xc000-0xc007]
pci 0000:00:11.0: reg 14: [io  0xb000-0xb003]
pci 0000:00:11.0: reg 18: [io  0xa000-0xa007]
pci 0000:00:11.0: reg 1c: [io  0x9000-0x9003]
pci 0000:00:11.0: reg 20: [io  0x8000-0x800f]
pci 0000:00:11.0: reg 24: [mem 0xfbcffc00-0xfbcfffff]
pci 0000:00:12.0: [1002:4397] type 0 class 0x000c03
pci 0000:00:12.0: reg 10: [mem 0xfbcfd000-0xfbcfdfff]
pci 0000:00:12.1: [1002:4398] type 0 class 0x000c03
pci 0000:00:12.1: reg 10: [mem 0xfbcfe000-0xfbcfefff]
pci 0000:00:12.2: [1002:4396] type 0 class 0x000c03
pci 0000:00:12.2: reg 10: [mem 0xfbcff800-0xfbcff8ff]
pci 0000:00:12.2: supports D1 D2
pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
pci 0000:00:12.2: PME# disabled
pci 0000:00:13.0: [1002:4397] type 0 class 0x000c03
pci 0000:00:13.0: reg 10: [mem 0xfbcfb000-0xfbcfbfff]
pci 0000:00:13.1: [1002:4398] type 0 class 0x000c03
pci 0000:00:13.1: reg 10: [mem 0xfbcfc000-0xfbcfcfff]
pci 0000:00:13.2: [1002:4396] type 0 class 0x000c03
pci 0000:00:13.2: reg 10: [mem 0xfbcff400-0xfbcff4ff]
pci 0000:00:13.2: supports D1 D2
pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
pci 0000:00:13.2: PME# disabled
pci 0000:00:14.0: [1002:4385] type 0 class 0x000c05
pci 0000:00:14.1: [1002:439c] type 0 class 0x000101
pci 0000:00:14.1: reg 10: [io  0x0000-0x0007]
pci 0000:00:14.1: reg 14: [io  0x0000-0x0003]
pci 0000:00:14.1: reg 18: [io  0x0000-0x0007]
pci 0000:00:14.1: reg 1c: [io  0x0000-0x0003]
pci 0000:00:14.1: reg 20: [io  0xff00-0xff0f]
pci 0000:00:14.3: [1002:439d] type 0 class 0x000601
pci 0000:00:14.4: [1002:4384] type 1 class 0x000604
pci 0000:00:14.5: [1002:4399] type 0 class 0x000c03
pci 0000:00:14.5: reg 10: [mem 0xfbcfa000-0xfbcfafff]
pci 0000:00:18.0: [1022:1200] type 0 class 0x000600
pci 0000:00:18.1: [1022:1201] type 0 class 0x000600
pci 0000:00:18.2: [1022:1202] type 0 class 0x000600
pci 0000:00:18.3: [1022:1203] type 0 class 0x000600
pci 0000:00:18.4: [1022:1204] type 0 class 0x000600
pci 0000:01:05.0: [1002:9614] type 0 class 0x000300
pci 0000:01:05.0: reg 10: [mem 0xf0000000-0xf7ffffff pref]
pci 0000:01:05.0: reg 14: [io  0xd000-0xd0ff]
pci 0000:01:05.0: reg 18: [mem 0xfbee0000-0xfbeeffff]
pci 0000:01:05.0: reg 24: [mem 0xfbd00000-0xfbdfffff]
pci 0000:01:05.0: supports D1 D2
pci 0000:01:05.1: [1002:960f] type 0 class 0x000403
pci 0000:01:05.1: reg 10: [mem 0xfbefc000-0xfbefffff]
pci 0000:01:05.1: supports D1 D2
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0:   bridge window [io  0xd000-0xdfff]
pci 0000:00:01.0:   bridge window [mem 0xfbd00000-0xfbefffff]
pci 0000:00:01.0:   bridge window [mem 0xf0000000-0xf7ffffff 64bit pref]
pci 0000:02:00.0: [1969:1026] type 0 class 0x000200
pci 0000:02:00.0: reg 10: [mem 0xfbfc0000-0xfbffffff 64bit]
pci 0000:02:00.0: reg 18: [io  0xec00-0xec7f]
pci 0000:02:00.0: PME# supported from D3hot D3cold
pci 0000:02:00.0: PME# disabled
pci 0000:00:06.0: PCI bridge to [bus 02-02]
pci 0000:00:06.0:   bridge window [io  0xe000-0xefff]
pci 0000:00:06.0:   bridge window [mem 0xfbf00000-0xfbffffff]
pci 0000:00:14.4: PCI bridge to [bus 03-03] (subtractive decode)
pci 0000:00:14.4:   bridge window [io  0x0000-0x0cf7] (subtractive decode)
pci 0000:00:14.4:   bridge window [io  0x0d00-0xffff] (subtractive decode)
pci 0000:00:14.4:   bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
pci 0000:00:14.4:   bridge window [mem 0x000d0000-0x000dffff] (subtractive decode)
pci 0000:00:14.4:   bridge window [mem 0xdff00000-0xdfffffff] (subtractive decode)
pci 0000:00:14.4:   bridge window [mem 0xf0000000-0xfebfffff] (subtractive decode)
pci_bus 0000:00: on NUMA node 0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE6._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
 pci0000:00: Unable to request _OSC control (_OSC support mask: 0x19)
ACPI: PCI Interrupt Link [LNKA] (IRQs 4 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKB] (IRQs 4 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 4 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKD] (IRQs 4 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKE] (IRQs 4 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 4 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 4 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 4 7 10 11 12 14 15) *0, disabled.
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Advanced Linux Sound Architecture Driver Version 1.0.24.
PCI: Using ACPI for IRQ routing
PCI: pci_cache_line_size set to 64 bytes
reserve RAM buffer: 000000000009fc00 - 000000000009ffff 
reserve RAM buffer: 00000000dfe90000 - 00000000dfffffff 
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
hpet0: 4 comparators, 32-bit 14.318180 MHz counter
Switching to clocksource hpet
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:00: [bus 00-ff]
pnp 00:00: [io  0x0cf8-0x0cff]
pnp 00:00: [io  0x0000-0x0cf7 window]
pnp 00:00: [io  0x0d00-0xffff window]
pnp 00:00: [mem 0x000a0000-0x000bffff window]
pnp 00:00: [mem 0x000d0000-0x000dffff window]
pnp 00:00: [mem 0xdff00000-0xdfffffff window]
pnp 00:00: [mem 0xf0000000-0xfebfffff window]
pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:02: [dma 4]
pnp 00:02: [io  0x0000-0x000f]
pnp 00:02: [io  0x0081-0x0083]
pnp 00:02: [io  0x0087]
pnp 00:02: [io  0x0089-0x008b]
pnp 00:02: [io  0x008f]
pnp 00:02: [io  0x00c0-0x00df]
pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
pnp 00:03: [io  0x0070-0x0071]
pnp 00:03: [irq 8]
pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active)
pnp 00:04: [io  0x0061]
pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
pnp 00:05: [io  0x00f0-0x00ff]
pnp 00:05: [irq 13]
pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
pnp 00:06: [mem 0xfed00000-0xfed003ff]
pnp 00:06: Plug and Play ACPI device, IDs PNP0103 (active)
pnp 00:07: [io  0x0060]
pnp 00:07: [io  0x0064]
pnp 00:07: [mem 0xfec00000-0xfec00fff]
pnp 00:07: [mem 0xfee00000-0xfee00fff]
system 00:07: [mem 0xfec00000-0xfec00fff] could not be reserved
system 00:07: [mem 0xfee00000-0xfee00fff] has been reserved
system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:08: [io  0x0010-0x001f]
pnp 00:08: [io  0x0022-0x003f]
pnp 00:08: [io  0x0062-0x0063]
pnp 00:08: [io  0x0065-0x006f]
pnp 00:08: [io  0x0072-0x007f]
pnp 00:08: [io  0x0080]
pnp 00:08: [io  0x0084-0x0086]
pnp 00:08: [io  0x0088]
pnp 00:08: [io  0x008c-0x008e]
pnp 00:08: [io  0x0090-0x009f]
pnp 00:08: [io  0x00a2-0x00bf]
pnp 00:08: [io  0x00b1]
pnp 00:08: [io  0x00e0-0x00ef]
pnp 00:08: [io  0x04d0-0x04d1]
pnp 00:08: [io  0x040b]
pnp 00:08: [io  0x04d6]
pnp 00:08: [io  0x0c00-0x0c01]
pnp 00:08: [io  0x0c14]
pnp 00:08: [io  0x0c50-0x0c51]
pnp 00:08: [io  0x0c52]
pnp 00:08: [io  0x0c6c]
pnp 00:08: [io  0x0c6f]
pnp 00:08: [io  0x0cd0-0x0cd1]
pnp 00:08: [io  0x0cd2-0x0cd3]
pnp 00:08: [io  0x0cd4-0x0cd5]
pnp 00:08: [io  0x0cd6-0x0cd7]
pnp 00:08: [io  0x0cd8-0x0cdf]
pnp 00:08: [io  0x0b00-0x0b3f]
pnp 00:08: [io  0x0800-0x089f]
pnp 00:08: [io  0x0000-0xffffffffffffffff disabled]
pnp 00:08: [io  0x0b00-0x0b0f]
pnp 00:08: [io  0x0b20-0x0b3f]
pnp 00:08: [io  0x0900-0x090f]
pnp 00:08: [io  0x0910-0x091f]
pnp 00:08: [io  0xfe00-0xfefe]
pnp 00:08: [io  0x0060]
pnp 00:08: [io  0x0064]
pnp 00:08: [mem 0xdff00000-0xdfffffff]
pnp 00:08: [mem 0xffb80000-0xffbfffff]
pnp 00:08: [mem 0xfec10000-0xfec1001f]
system 00:08: [io  0x04d0-0x04d1] has been reserved
system 00:08: [io  0x040b] has been reserved
system 00:08: [io  0x04d6] has been reserved
system 00:08: [io  0x0c00-0x0c01] has been reserved
system 00:08: [io  0x0c14] has been reserved
system 00:08: [io  0x0c50-0x0c51] has been reserved
system 00:08: [io  0x0c52] has been reserved
system 00:08: [io  0x0c6c] has been reserved
system 00:08: [io  0x0c6f] has been reserved
system 00:08: [io  0x0cd0-0x0cd1] has been reserved
system 00:08: [io  0x0cd2-0x0cd3] has been reserved
system 00:08: [io  0x0cd4-0x0cd5] has been reserved
system 00:08: [io  0x0cd6-0x0cd7] has been reserved
system 00:08: [io  0x0cd8-0x0cdf] has been reserved
system 00:08: [io  0x0b00-0x0b3f] has been reserved
system 00:08: [io  0x0800-0x089f] has been reserved
system 00:08: [io  0x0b00-0x0b0f] has been reserved
system 00:08: [io  0x0b20-0x0b3f] has been reserved
system 00:08: [io  0x0900-0x090f] has been reserved
system 00:08: [io  0x0910-0x091f] has been reserved
system 00:08: [io  0xfe00-0xfefe] has been reserved
system 00:08: [mem 0xdff00000-0xdfffffff] has been reserved
system 00:08: [mem 0xffb80000-0xffbfffff] has been reserved
system 00:08: [mem 0xfec10000-0xfec1001f] has been reserved
system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:09: [io  0x0000-0xffffffffffffffff disabled]
pnp 00:09: [io  0x0230-0x023f]
pnp 00:09: [io  0x0290-0x029f]
pnp 00:09: [io  0x0f40-0x0f4f]
pnp 00:09: [io  0x0a30-0x0a3f]
system 00:09: [io  0x0230-0x023f] has been reserved
system 00:09: [io  0x0290-0x029f] has been reserved
system 00:09: [io  0x0f40-0x0f4f] has been reserved
system 00:09: [io  0x0a30-0x0a3f] has been reserved
system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:0a: [mem 0xe0000000-0xefffffff]
system 00:0a: [mem 0xe0000000-0xefffffff] has been reserved
system 00:0a: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:0b: [mem 0x00000000-0x0009ffff]
pnp 00:0b: [mem 0x000c0000-0x000cffff]
pnp 00:0b: [mem 0x000e0000-0x000fffff]
pnp 00:0b: [mem 0x00100000-0xdfefffff]
pnp 00:0b: [mem 0xfec00000-0xffffffff]
system 00:0b: [mem 0x00000000-0x0009ffff] could not be reserved
system 00:0b: [mem 0x000c0000-0x000cffff] could not be reserved
system 00:0b: [mem 0x000e0000-0x000fffff] could not be reserved
system 00:0b: [mem 0x00100000-0xdfefffff] could not be reserved
system 00:0b: [mem 0xfec00000-0xffffffff] could not be reserved
system 00:0b: Plug and Play ACPI device, IDs PNP0c01 (active)
pnp: PnP ACPI: found 12 devices
ACPI: ACPI bus type pnp unregistered
PCI: max bus depth: 1 pci_try_num: 2
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0:   bridge window [io  0xd000-0xdfff]
pci 0000:00:01.0:   bridge window [mem 0xfbd00000-0xfbefffff]
pci 0000:00:01.0:   bridge window [mem 0xf0000000-0xf7ffffff 64bit pref]
pci 0000:00:06.0: PCI bridge to [bus 02-02]
pci 0000:00:06.0:   bridge window [io  0xe000-0xefff]
pci 0000:00:06.0:   bridge window [mem 0xfbf00000-0xfbffffff]
pci 0000:00:14.4: PCI bridge to [bus 03-03]
pci 0000:00:06.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
pci 0000:00:06.0: setting latency timer to 64
pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
pci_bus 0000:00: resource 7 [mem 0x000d0000-0x000dffff]
pci_bus 0000:00: resource 8 [mem 0xdff00000-0xdfffffff]
pci_bus 0000:00: resource 9 [mem 0xf0000000-0xfebfffff]
pci_bus 0000:01: resource 0 [io  0xd000-0xdfff]
pci_bus 0000:01: resource 1 [mem 0xfbd00000-0xfbefffff]
pci_bus 0000:01: resource 2 [mem 0xf0000000-0xf7ffffff 64bit pref]
pci_bus 0000:02: resource 0 [io  0xe000-0xefff]
pci_bus 0000:02: resource 1 [mem 0xfbf00000-0xfbffffff]
pci_bus 0000:03: resource 4 [io  0x0000-0x0cf7]
pci_bus 0000:03: resource 5 [io  0x0d00-0xffff]
pci_bus 0000:03: resource 6 [mem 0x000a0000-0x000bffff]
pci_bus 0000:03: resource 7 [mem 0x000d0000-0x000dffff]
pci_bus 0000:03: resource 8 [mem 0xdff00000-0xdfffffff]
pci_bus 0000:03: resource 9 [mem 0xf0000000-0xfebfffff]
NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
UDP hash table entries: 4096 (order: 5, 131072 bytes)
UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
NET: Registered protocol family 1
pci 0000:00:01.0: MSI quirk detected; subordinate MSI disabled
pci 0000:01:05.0: Boot video device
PCI: CLS 64 bytes, default 64
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff8800dbe8e000 - ffff8800dfe8e000
software IO TLB at phys 0xdbe8e000 - 0xdfe8e000
kvm: Nested Virtualization enabled
kvm: Nested Paging enabled
perf: AMD IBS detected (0x0000001f)
SGI XFS with security attributes, large block/inode numbers, no debug enabled
msgmni has been set to 15951
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
input: Power Button as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0
ACPI: Power Button [PWRB]
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
ACPI: Power Button [PWRF]
ACPI: processor limited to max C-state 1
[drm] Initialized drm 1.1.0 20060810
[drm] radeon defaulting to kernel modesetting.
[drm] radeon kernel modesetting enabled.
radeon 0000:01:05.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
radeon 0000:01:05.0: setting latency timer to 64
[drm] initializing kernel modesetting (RS780 0x1002:0x9614 0x1043:0x834D).
[drm] register mmio base: 0xFBEE0000
[drm] register mmio size: 65536
ATOM BIOS: 113
radeon 0000:01:05.0: VRAM: 128M 0x00000000C0000000 - 0x00000000C7FFFFFF (128M used)
radeon 0000:01:05.0: GTT: 512M 0x00000000A0000000 - 0x00000000BFFFFFFF
[drm] Detected VRAM RAM=128M, BAR=128M
[drm] RAM width 32bits DDR
[TTM] Zone  kernel: Available graphics memory: 4083584 kiB.
[TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
[TTM] Initializing pool allocator.
[drm] radeon: 128M of VRAM memory ready
[drm] radeon: 512M of GTT memory ready.
[drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[drm] Driver supports precise vblank timestamp query.
[drm] radeon: irq initialized.
[drm] GART: num cpu pages 131072, num gpu pages 131072
[drm] Loading RS780 Microcode
[drm] PCIE GART of 512M enabled (table at 0x00000000C0040000).
radeon 0000:01:05.0: WB enabled
[drm] ring test succeeded in 1 usecs
[drm] radeon: ib pool ready.
[drm] ib test succeeded in 0 usecs
[drm] Radeon Display Connectors
[drm] Connector 0:
[drm]   VGA
[drm]   DDC: 0x7e40 0x7e40 0x7e44 0x7e44 0x7e48 0x7e48 0x7e4c 0x7e4c
[drm]   Encoders:
[drm]     CRT1: INTERNAL_KLDSCP_DAC1
[drm] Connector 1:
[drm]   DVI-D
[drm]   HPD3
[drm]   DDC: 0x7e50 0x7e50 0x7e54 0x7e54 0x7e58 0x7e58 0x7e5c 0x7e5c
[drm]   Encoders:
[drm]     DFP3: INTERNAL_KLDSCP_LVTMA
[drm] radeon: power management initialized
[drm] fb mappable at 0xF0142000
[drm] vram apper at 0xF0000000
[drm] size 7299072
[drm] fb depth is 24
[drm]    pitch is 6912
fbcon: radeondrmfb (fb0) is primary device
Console: switching to colour frame buffer device 131x105
fb0: radeondrmfb frame buffer device
drm: registered panic notifier
[drm] Initialized radeon 2.12.0 20080528 for 0000:01:05.0 on minor 0
loop: module loaded
ahci 0000:00:11.0: version 3.0
ahci 0000:00:11.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
ahci 0000:00:11.0: AHCI 0001.0100 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part ccc 
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 abar m1024@0xfbcffc00 port 0xfbcffd00 irq 22
ata2: SATA max UDMA/133 abar m1024@0xfbcffc00 port 0xfbcffd80 irq 22
ata3: SATA max UDMA/133 abar m1024@0xfbcffc00 port 0xfbcffe00 irq 22
ata4: SATA max UDMA/133 abar m1024@0xfbcffc00 port 0xfbcffe80 irq 22
ata5: SATA max UDMA/133 abar m1024@0xfbcffc00 port 0xfbcfff00 irq 22
ata6: SATA max UDMA/133 abar m1024@0xfbcffc00 port 0xfbcfff80 irq 22
pata_atiixp 0000:00:14.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
scsi6 : pata_atiixp
scsi7 : pata_atiixp
ata7: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xff00 irq 14
ata8: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xff08 irq 15
ATL1E 0000:02:00.0: BAR 0: set to [mem 0xfbfc0000-0xfbffffff 64bit] (PCI address [0xfbfc0000-0xfbffffff])
ATL1E 0000:02:00.0: BAR 2: set to [io  0xec00-0xec7f] (PCI address [0xec00-0xec7f])
ATL1E 0000:02:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
ATL1E 0000:02:00.0: setting latency timer to 64
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:12.2: PCI INT B -> GSI 17 (level, low) -> IRQ 17
ehci_hcd 0000:00:12.2: EHCI Host Controller
ehci_hcd 0000:00:12.2: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:12.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
QUIRK: Enable AMD PLL fix
ehci_hcd 0000:00:12.2: applying AMD SB600/SB700 USB freeze workaround
ehci_hcd 0000:00:12.2: debug port 1
ehci_hcd 0000:00:12.2: irq 17, io mem 0xfbcff800
ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 6 ports detected
ehci_hcd 0000:00:13.2: PCI INT B -> GSI 19 (level, low) -> IRQ 19
ehci_hcd 0000:00:13.2: EHCI Host Controller
ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:13.2: applying AMD SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
ehci_hcd 0000:00:13.2: applying AMD SB600/SB700 USB freeze workaround
ehci_hcd 0000:00:13.2: debug port 1
ehci_hcd 0000:00:13.2: irq 19, io mem 0xfbcff400
ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 6 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci_hcd 0000:00:12.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ohci_hcd 0000:00:12.0: OHCI Host Controller
ohci_hcd 0000:00:12.0: new USB bus registered, assigned bus number 3
ohci_hcd 0000:00:12.0: irq 16, io mem 0xfbcfd000
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 3 ports detected
ohci_hcd 0000:00:12.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ohci_hcd 0000:00:12.1: OHCI Host Controller
ohci_hcd 0000:00:12.1: new USB bus registered, assigned bus number 4
ohci_hcd 0000:00:12.1: irq 16, io mem 0xfbcfe000
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 3 ports detected
ohci_hcd 0000:00:13.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
ohci_hcd 0000:00:13.0: OHCI Host Controller
ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 5
ohci_hcd 0000:00:13.0: irq 18, io mem 0xfbcfb000
ata7.00: ATAPI: HL-DT-STDVD-RAM GH22NP20, 1.03, max UDMA/66
ata7.00: configured for UDMA/66
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 3 ports detected
ohci_hcd 0000:00:13.1: PCI INT A -> GSI 18 (level, low) -> IRQ 18
ohci_hcd 0000:00:13.1: OHCI Host Controller
ohci_hcd 0000:00:13.1: new USB bus registered, assigned bus number 6
ohci_hcd 0000:00:13.1: irq 18, io mem 0xfbcfc000
hub 6-0:1.0: USB hub found
hub 6-0:1.0: 3 ports detected
ohci_hcd 0000:00:14.5: PCI INT C -> GSI 18 (level, low) -> IRQ 18
ohci_hcd 0000:00:14.5: OHCI Host Controller
ohci_hcd 0000:00:14.5: new USB bus registered, assigned bus number 7
ohci_hcd 0000:00:14.5: irq 18, io mem 0xfbcfa000
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata6: SATA link down (SStatus 0 SControl 300)
ata5: SATA link down (SStatus 0 SControl 300)
ata4: SATA link down (SStatus 0 SControl 300)
ata3.00: ATA-8: OCZ-VERTEX, 1.6, max UDMA/133
ata3.00: 62533296 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
ata1.00: ATA-8: ST1500DL003-9VT16L, CC32, max UDMA/133
ata1.00: 2930277168 sectors, multi 16: LBA48 NCQ (depth 31/32)
ata2: SATA link down (SStatus 0 SControl 300)
ata3.00: configured for UDMA/133
ata1.00: configured for UDMA/133
scsi 0:0:0:0: Direct-Access     ATA      ST1500DL003-9VT1 CC32 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 2930277168 512-byte logical blocks: (1.50 TB/1.36 TiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 2:0:0:0: Direct-Access     ATA      OCZ-VERTEX       1.6  PQ: 0 ANSI: 5
sd 2:0:0:0: [sdb] 62533296 512-byte logical blocks: (32.0 GB/29.8 GiB)
sd 2:0:0:0: Attached scsi generic sg1 type 0
sd 2:0:0:0: [sdb] Write Protect is off
sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdb: sdb1 sdb2
sd 2:0:0:0: [sdb] Attached SCSI disk
scsi 6:0:0:0: CD-ROM            HL-DT-ST DVD-RAM GH22NP20 1.03 PQ: 0 ANSI: 5
hub 7-0:1.0: USB hub found
hub 7-0:1.0: 2 ports detected
usbcore: registered new interface driver usblp
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver usbserial
USB Serial support registered for generic
 sda: unknown partition table
sd 0:0:0:0: [sda] Attached SCSI disk
sr0: scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray
cdrom: Uniform CD-ROM driver Revision: 3.20
sr 6:0:0:0: Attached scsi CD-ROM sr0
sr 6:0:0:0: Attached scsi generic sg2 type 5
usb 4-1: new full-speed USB device number 2 using ohci_hcd
Refined TSC clocksource calibration: 3210.826 MHz.
Switching to clocksource tsc
usb 4-2: new full-speed USB device number 3 using ohci_hcd
usb 4-3: new low-speed USB device number 4 using ohci_hcd
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial Driver core
USB Serial support registered for GSM modem (1-port)
usbcore: registered new interface driver option
option: v0.7.2:USB Driver for GSM modems
i8042: PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mousedev: PS/2 mouse device common for all mice
rtc_cmos 00:03: RTC can wake from S4
rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
i2c /dev entries driver
EDAC MC: Ver: 2.1.0
AMD64 EDAC driver v3.4.0
EDAC amd64: DRAM ECC enabled.
EDAC amd64: F10h detected (node 0).
EDAC MC: DCT0 chip selects:
EDAC amd64: MC: 0:  1024MB 1:  1024MB
EDAC amd64: MC: 2:  1024MB 3:  1024MB
EDAC amd64: MC: 4:     0MB 5:     0MB
EDAC amd64: MC: 6:     0MB 7:     0MB
EDAC MC: DCT1 chip selects:
EDAC amd64: MC: 0:  1024MB 1:  1024MB
EDAC amd64: MC: 2:  1024MB 3:  1024MB
EDAC amd64: MC: 4:     0MB 5:     0MB
EDAC amd64: MC: 6:     0MB 7:     0MB
EDAC amd64: using x4 syndromes.
EDAC amd64: MCT channel count: 2
EDAC amd64: CS0: Unbuffered DDR3 RAM
EDAC amd64: CS1: Unbuffered DDR3 RAM
EDAC amd64: CS2: Unbuffered DDR3 RAM
EDAC amd64: CS3: Unbuffered DDR3 RAM
EDAC MC0: Giving out device to 'amd64_edac' 'F10h': DEV 0000:00:18.2
EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.2' (POLLED)
cpuidle: using governor ladder
cpuidle: using governor menu
input: C-Media USB Headphone Set   as /devices/pci0000:00/0000:00:12.1/usb4/4-1/4-1:1.3/input/input2
generic-usb 0003:0D8C:000C.0001: input,hidraw0: USB HID v1.00 Device [C-Media USB Headphone Set  ] on usb-0000:00:12.1-1/input3
logitech-djreceiver 0003:046D:C52B.0004: hiddev0,hidraw1: USB HID v1.11 Device [Logitech USB Receiver] on usb-0000:00:12.1-2/input2
input: Logitech Unifying Device. Wireless PID:101b as /devices/pci0000:00/0000:00:12.1/usb4/4-2/4-2:1.2/0003:046D:C52B.0004/input/input3
logitech-djdevice 0003:046D:C52B.0006: input,hidraw2: USB HID v1.11 Mouse [Logitech Unifying Device. Wireless PID:101b] on usb-0000:00:12.1-2:1
input: HID 046a:0011 as /devices/pci0000:00/0000:00:12.1/usb4/4-3/4-3:1.0/input/input4
generic-usb 0003:046A:0011.0005: input,hidraw3: USB HID v1.10 Keyboard [HID 046a:0011] on usb-0000:00:12.1-3/input0
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
usbcore: registered new interface driver snd-usb-audio
ALSA device list:
  #0: C-Media USB Headphone Set at usb-0000:00:12.1-1, full speed
Netfilter messages via NETLINK v0.30.
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ctnetlink v0.93: registering with nfnetlink.
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos 00:03: setting system clock to 2011-12-02 19:50:18 UTC (1322855418)
powernow-k8: Found 1 AMD Phenom(tm) II X4 955 Processor (4 cpu cores) (version 2.20.00)
powernow-k8:    0 : pstate 0 (3200 MHz)
powernow-k8:    1 : pstate 1 (2500 MHz)
powernow-k8:    2 : pstate 2 (2100 MHz)
powernow-k8:    3 : pstate 3 (800 MHz)
XFS (sdb2): Mounting Filesystem
XFS (sdb2): Ending clean mount
VFS: Mounted root (xfs filesystem) readonly on device 8:18.
devtmpfs: mounted
Freeing unused kernel memory: 420k freed
Write protecting the kernel read-only data: 8192k
Freeing unused kernel memory: 1584k freed
Freeing unused kernel memory: 400k freed
XFS (sda): Mounting Filesystem
XFS (sda): Ending clean mount
ATL1E 0000:02:00.0: irq 40 for MSI/MSI-X
ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex>
ATL1E 0000:02:00.0: eth0: NIC Link is Up <100 Mbps Full Duplex>
udevd[868]: starting version 171
Adding 2097148k swap on /var/tmp/swap/swapfile.  Priority:-1 extents:2 across:2634672k

Markus Trippelsdorf Dec. 2, 2011, 8:48 p.m. UTC | #17

On 2011.12.02 at 21:06 +0100, Markus Trippelsdorf wrote:
> On 2011.12.02 at 14:43 -0500, Jerome Glisse wrote:
> > On Thu, Dec 01, 2011 at 09:44:37AM +0100, Markus Trippelsdorf wrote:
> > > On 2011.11.24 at 09:50 +0100, Markus Trippelsdorf wrote:
> > > > On 2011.11.23 at 10:06 -0600, Christoph Lameter wrote:
> > > > > On Wed, 23 Nov 2011, Markus Trippelsdorf wrote:
> > > > > 
> > > > > > > FIX idr_layer_cache: Marking all objects used
> > > > > >
> > > > > > Yesterday I couldn't reproduce the issue at all. But today I've hit
> > > > > > exactly the same spot again. (CCing the drm list)
> > > > > 
> > > > > Well this is looks like write after free.
> > > > > 
> > > > > > =============================================================================
> > > > > > BUG idr_layer_cache: Poison overwritten
> > > > > > -----------------------------------------------------------------------------
> > > > > > Object ffff8802156487c0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > > Object ffff8802156487d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > > Object ffff8802156487e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > > Object ffff8802156487f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > > Object ffff880215648800: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  ....kkkkkkkkkkkk
> > > > > > Object ffff880215648810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> > > > > 
> > > > > And its an integer sized write of 0. If you look at the struct definition
> > > > > and lookup the offset you should be able to locate the field that
> > > > > was modified.
> > > 
> > > It also happens with CONFIG_SLAB. 
> > > (If someone wants to reproduce the issue, just run a kexec boot loop and
> > > the bug will occur after a few (~10) iterations.)
> > > 
> > 
> > Can you provide the kexec command line you are using and full kernel
> > log (mostly interested in kernel option).
> 
> /usr/sbin/kexec -l "/usr/src/linux/arch/x86/boot/bzImage" --append="root=PARTUUID=6d6a4009-3a90-40df-806a-e63f48189719 init=/sbin/minit rootflags=logbsize=262144 fbcon=rotate:3 drm_kms_helper.poll=0 quiet"
> /usr/sbin/kexec -e
> 
> (The loop happens after autologin in .zprofile:
> sleep 4 && sudo /etc/minit/ctrlaltdel/run
> (the last script kills, unmounts and then runs the two kexec commands
> above))

BTW I always see (mostly only on screen, sometimes in the logs):

[Firmware Bug]: cpu 2, try to use APIC500 (LVT offset 0) for vector 0x10400, but the register is already in use for vector 0xf9 on another cpu
[Firmware Bug]: cpu 2, IBS interrupt offset 0 not available (MSRC001103A=0x0000000000000100)
[Firmware Bug]: using offset 1 for IBS interrupts
[Firmware Bug]: workaround enabled for IBS LVT offset
perf: AMD IBS detected (0x0000001f) 

But I hope that it is only a harmless warning. 
(perf Instruction-Based Sampling)

Robert?

Robert Richter Dec. 7, 2011, 2:32 p.m. UTC | #18

On 02.12.11 21:48:20, Markus Trippelsdorf wrote:
> BTW I always see (mostly only on screen, sometimes in the logs):
> 
> [Firmware Bug]: cpu 2, try to use APIC500 (LVT offset 0) for vector 0x10400, but the register is already in use for vector 0xf9 on another cpu
> [Firmware Bug]: cpu 2, IBS interrupt offset 0 not available (MSRC001103A=0x0000000000000100)
> [Firmware Bug]: using offset 1 for IBS interrupts
> [Firmware Bug]: workaround enabled for IBS LVT offset
> perf: AMD IBS detected (0x0000001f) 
> 
> But I hope that it is only a harmless warning. 
> (perf Instruction-Based Sampling)

Yes, the message always apears on AMD family 10h. Nothing to worry
about.

A patch is on the way to soften the message to not scare the people:

 http://git.kernel.org/?p=linux/kernel/git/tip/tip.git;a=commit;h=16e5294e5f8303756a179cf218e37dfb9ed34417

-Robert

Markus Trippelsdorf Dec. 7, 2011, 2:39 p.m. UTC | #19

On 2011.12.07 at 15:32 +0100, Robert Richter wrote:
> On 02.12.11 21:48:20, Markus Trippelsdorf wrote:
> > BTW I always see (mostly only on screen, sometimes in the logs):
> > 
> > [Firmware Bug]: cpu 2, try to use APIC500 (LVT offset 0) for vector 0x10400, but the register is already in use for vector 0xf9 on another cpu
> > [Firmware Bug]: cpu 2, IBS interrupt offset 0 not available (MSRC001103A=0x0000000000000100)
> > [Firmware Bug]: using offset 1 for IBS interrupts
> > [Firmware Bug]: workaround enabled for IBS LVT offset
> > perf: AMD IBS detected (0x0000001f) 
> > 
> > But I hope that it is only a harmless warning. 
> > (perf Instruction-Based Sampling)
> 
> Yes, the message always apears on AMD family 10h. Nothing to worry
> about.
> 
> A patch is on the way to soften the message to not scare the people:
> 
>  http://git.kernel.org/?p=linux/kernel/git/tip/tip.git;a=commit;h=16e5294e5f8303756a179cf218e37dfb9ed34417

Thanks.
It's already in mainline and the message is gone now.

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=16e5294e5f8303756a179cf218e37dfb9ed34417

Eric W. Biederman Dec. 19, 2011, 3:21 a.m. UTC | #20

Markus Trippelsdorf <markus@trippelsdorf.de> writes:

> On 2011.11.21 at 17:34 +0100, Markus Trippelsdorf wrote:
>> On 2011.11.21 at 17:10 +0100, Markus Trippelsdorf wrote:
>> > On 2011.11.21 at 16:48 +0100, Eric Dumazet wrote:
>> > > Le lundi 21 novembre 2011 à 16:36 +0100, Markus Trippelsdorf a écrit :
>> > > > On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
>> > > > > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
>> > > > > 
>> > > > > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
>> > > > > > 
>> > > > > 
>> > > > > Thanks
>> > > > > 
>> > > > > Please continue to provide more samples.
>> > > > > 
>> > > > > There is something wrong somewhere, but where exactly, its hard to say.
>> > > > 
>> > > > New sample. This one points to lib/idr.c:
>> > > > 
>> > > > =============================================================================
>> > > > BUG idr_layer_cache: Poison overwritten
>> > > > -----------------------------------------------------------------------------
>> > > 
>> > > Thanks, could you now add "CONFIG_DEBUG_PAGEALLOC=y" in your config as
>> > > well ?
>> > 
>> > Sure. This one happend with CONFIG_DEBUG_PAGEALLOC=y:
>> > 
>> > =============================================================================
>> > BUG task_struct: Poison overwritten
>> > -----------------------------------------------------------------------------
>> 
>> And sometimes this one that I've reported earlier already:
>> 
>> (see: http://thread.gmane.org/gmane.linux.kernel/1215023 )
>> 
>>  ------------[ cut here ]------------
>>  WARNING: at fs/sysfs/sysfs.h:195 sysfs_get_inode+0x136/0x140()
>>  Hardware name: System Product Name
>>  Pid: 1876, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #72
>>  Call Trace:
>>  [<ffffffff8106cac5>] warn_slowpath_common+0x75/0xb0
>>  [<ffffffff8106cbc5>] warn_slowpath_null+0x15/0x20
>>  [<ffffffff81163236>] sysfs_get_inode+0x136/0x140
>>  [<ffffffff81164cef>] sysfs_lookup+0x6f/0x110
>>  [<ffffffff811173f9>] d_alloc_and_lookup+0x39/0x80
>>  [<ffffffff81118774>] do_lookup+0x294/0x3a0
>>  [<ffffffff8111798a>] ? inode_permission+0x7a/0xb0
>>  [<ffffffff8111a3f7>] do_last.isra.46+0x137/0x7f0
>>  [<ffffffff8111ab76>] path_openat+0xc6/0x370
>>  [<ffffffff81117606>] ? getname_flags+0x36/0x230
>>  [<ffffffff810ec852>] ? handle_mm_fault+0x192/0x290
>>  [<ffffffff8111ae5c>] do_filp_open+0x3c/0x90
>>  [<ffffffff81127c8c>] ? alloc_fd+0xdc/0x120
>>  [<ffffffff8110ce77>] do_sys_open+0xe7/0x1c0
>>  [<ffffffff8110cf6b>] sys_open+0x1b/0x20
>>  [<ffffffff814ccb7b>] system_call_fastpath+0x16/0x1b
>>  ---[ end trace b1377eb8b131d37d ]---
>
> Hm, the "sysfs: use rb-tree" thing hit again during boot. Could this be
> the root cause of this all?
>
> I wrote down the following:
>
> RIP : rb_next
>
> Trace:
>  sysfs_dir_pos
>  sysfs_readdir
>  ? sys_ioctl
>  vfs_readdir
>  sys_getdents

Thanks for reporting this.

Has this by any chance been resolved or stopped happening?

This looks for all of the world like something is stomping your sysfs
dirents.   I haven't seen anyone else complaining so this seems like the
problem is unique to your configuration.  Which suggests that it is not
sysfs itself that is wrong.

I have been through the code a time or two and I haven't seen anything
obviously wrong.  Everything that sysfs does is protected by the
sysfs_mutex so the locking is very very simple.

My best guess of why now is that the rbtree code make a sysfs dirent
48 bytes larger.  And so it is much more exposed to these kinds of
problems.

Eric
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Eric W. Biederman Dec. 19, 2011, 9:06 a.m. UTC | #21

Markus Trippelsdorf <markus@trippelsdorf.de> writes:

> On 2011.12.18 at 19:21 -0800, Eric W. Biederman wrote:
>> Markus Trippelsdorf <markus@trippelsdorf.de> writes:
>> 
>> > On 2011.11.21 at 17:34 +0100, Markus Trippelsdorf wrote:
>> >> On 2011.11.21 at 17:10 +0100, Markus Trippelsdorf wrote:
>> >> > On 2011.11.21 at 16:48 +0100, Eric Dumazet wrote:
>> >> > > Le lundi 21 novembre 2011 à 16:36 +0100, Markus Trippelsdorf a écrit :
>> >> > > > On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
>> >> > > > > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
>> >> > > > > 
>> >> > > > > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
>> >> > > > > > 
>> >> > > > > 
>> >> > > > > Thanks
>> >> > > > > 
>> >> > > > > Please continue to provide more samples.
>> >> > > > > 
>> >> > > > > There is something wrong somewhere, but where exactly, its hard to say.
>> >> > > > 
>> >> > > > New sample. This one points to lib/idr.c:
>> >> > > > 
>> >> > > > =============================================================================
>> >> > > > BUG idr_layer_cache: Poison overwritten
>> >> > > > -----------------------------------------------------------------------------
>> >> > > 
>> >> > > Thanks, could you now add "CONFIG_DEBUG_PAGEALLOC=y" in your config as
>> >> > > well ?
>> >> > 
>> >> > Sure. This one happend with CONFIG_DEBUG_PAGEALLOC=y:
>> >> > 
>> >> > =============================================================================
>> >> > BUG task_struct: Poison overwritten
>> >> > -----------------------------------------------------------------------------
>> >> 
>> >> And sometimes this one that I've reported earlier already:
>> >> 
>> >> (see: http://thread.gmane.org/gmane.linux.kernel/1215023 )
>> >> 
>> >>  ------------[ cut here ]------------
>> >>  WARNING: at fs/sysfs/sysfs.h:195 sysfs_get_inode+0x136/0x140()
>> >>  Hardware name: System Product Name
>> >>  Pid: 1876, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #72
>> >>  Call Trace:
>> >>  [<ffffffff8106cac5>] warn_slowpath_common+0x75/0xb0
>> >>  [<ffffffff8106cbc5>] warn_slowpath_null+0x15/0x20
>> >>  [<ffffffff81163236>] sysfs_get_inode+0x136/0x140
>> >>  [<ffffffff81164cef>] sysfs_lookup+0x6f/0x110
>> >>  [<ffffffff811173f9>] d_alloc_and_lookup+0x39/0x80
>> >>  [<ffffffff81118774>] do_lookup+0x294/0x3a0
>> >>  [<ffffffff8111798a>] ? inode_permission+0x7a/0xb0
>> >>  [<ffffffff8111a3f7>] do_last.isra.46+0x137/0x7f0
>> >>  [<ffffffff8111ab76>] path_openat+0xc6/0x370
>> >>  [<ffffffff81117606>] ? getname_flags+0x36/0x230
>> >>  [<ffffffff810ec852>] ? handle_mm_fault+0x192/0x290
>> >>  [<ffffffff8111ae5c>] do_filp_open+0x3c/0x90
>> >>  [<ffffffff81127c8c>] ? alloc_fd+0xdc/0x120
>> >>  [<ffffffff8110ce77>] do_sys_open+0xe7/0x1c0
>> >>  [<ffffffff8110cf6b>] sys_open+0x1b/0x20
>> >>  [<ffffffff814ccb7b>] system_call_fastpath+0x16/0x1b
>> >>  ---[ end trace b1377eb8b131d37d ]---
>> >
>> > Hm, the "sysfs: use rb-tree" thing hit again during boot. Could this be
>> > the root cause of this all?
>> >
>> > I wrote down the following:
>> >
>> > RIP : rb_next
>> >
>> > Trace:
>> >  sysfs_dir_pos
>> >  sysfs_readdir
>> >  ? sys_ioctl
>> >  vfs_readdir
>> >  sys_getdents
>> 
>> Thanks for reporting this.
>> 
>> Has this by any chance been resolved or stopped happening?
>
> Yes.
>
>> This looks for all of the world like something is stomping your sysfs
>> dirents.   I haven't seen anyone else complaining so this seems like the
>> problem is unique to your configuration.  Which suggests that it is not
>> sysfs itself that is wrong.
>> 
>> I have been through the code a time or two and I haven't seen anything
>> obviously wrong.  Everything that sysfs does is protected by the
>> sysfs_mutex so the locking is very very simple.
>> 
>> My best guess of why now is that the rbtree code make a sysfs dirent
>> 48 bytes larger.  And so it is much more exposed to these kinds of
>> problems.
>
> Sorry, but your subsystem was just accidentally hit by a bug in the
> Radeon driver, that sometimes randomly writes 0 dwords somewhere to
> memory after a kexec boot (see the rest of this huge thread).
> It's still not fixed in mainline, because Linus refused to take the fix
> this late in the series.

Awesome.  

I guess that means I am only responsible for the Radeon driver having
the opportunity to take that code path. It is nice to see kexec being
used and being well known enough I didn't have to get involved.

Eric
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Markus Trippelsdorf Dec. 19, 2011, 9:19 a.m. UTC | #22

On 2011.12.18 at 19:21 -0800, Eric W. Biederman wrote:
> Markus Trippelsdorf <markus@trippelsdorf.de> writes:
> 
> > On 2011.11.21 at 17:34 +0100, Markus Trippelsdorf wrote:
> >> On 2011.11.21 at 17:10 +0100, Markus Trippelsdorf wrote:
> >> > On 2011.11.21 at 16:48 +0100, Eric Dumazet wrote:
> >> > > Le lundi 21 novembre 2011 à 16:36 +0100, Markus Trippelsdorf a écrit :
> >> > > > On 2011.11.21 at 15:16 +0100, Eric Dumazet wrote:
> >> > > > > Le lundi 21 novembre 2011 à 14:15 +0100, Markus Trippelsdorf a écrit :
> >> > > > > 
> >> > > > > > I've enabled CONFIG_SLUB_DEBUG_ON and this is what happend:
> >> > > > > > 
> >> > > > > 
> >> > > > > Thanks
> >> > > > > 
> >> > > > > Please continue to provide more samples.
> >> > > > > 
> >> > > > > There is something wrong somewhere, but where exactly, its hard to say.
> >> > > > 
> >> > > > New sample. This one points to lib/idr.c:
> >> > > > 
> >> > > > =============================================================================
> >> > > > BUG idr_layer_cache: Poison overwritten
> >> > > > -----------------------------------------------------------------------------
> >> > > 
> >> > > Thanks, could you now add "CONFIG_DEBUG_PAGEALLOC=y" in your config as
> >> > > well ?
> >> > 
> >> > Sure. This one happend with CONFIG_DEBUG_PAGEALLOC=y:
> >> > 
> >> > =============================================================================
> >> > BUG task_struct: Poison overwritten
> >> > -----------------------------------------------------------------------------
> >> 
> >> And sometimes this one that I've reported earlier already:
> >> 
> >> (see: http://thread.gmane.org/gmane.linux.kernel/1215023 )
> >> 
> >>  ------------[ cut here ]------------
> >>  WARNING: at fs/sysfs/sysfs.h:195 sysfs_get_inode+0x136/0x140()
> >>  Hardware name: System Product Name
> >>  Pid: 1876, comm: slabinfo Not tainted 3.2.0-rc2-00274-g6fe4c6d #72
> >>  Call Trace:
> >>  [<ffffffff8106cac5>] warn_slowpath_common+0x75/0xb0
> >>  [<ffffffff8106cbc5>] warn_slowpath_null+0x15/0x20
> >>  [<ffffffff81163236>] sysfs_get_inode+0x136/0x140
> >>  [<ffffffff81164cef>] sysfs_lookup+0x6f/0x110
> >>  [<ffffffff811173f9>] d_alloc_and_lookup+0x39/0x80
> >>  [<ffffffff81118774>] do_lookup+0x294/0x3a0
> >>  [<ffffffff8111798a>] ? inode_permission+0x7a/0xb0
> >>  [<ffffffff8111a3f7>] do_last.isra.46+0x137/0x7f0
> >>  [<ffffffff8111ab76>] path_openat+0xc6/0x370
> >>  [<ffffffff81117606>] ? getname_flags+0x36/0x230
> >>  [<ffffffff810ec852>] ? handle_mm_fault+0x192/0x290
> >>  [<ffffffff8111ae5c>] do_filp_open+0x3c/0x90
> >>  [<ffffffff81127c8c>] ? alloc_fd+0xdc/0x120
> >>  [<ffffffff8110ce77>] do_sys_open+0xe7/0x1c0
> >>  [<ffffffff8110cf6b>] sys_open+0x1b/0x20
> >>  [<ffffffff814ccb7b>] system_call_fastpath+0x16/0x1b
> >>  ---[ end trace b1377eb8b131d37d ]---
> >
> > Hm, the "sysfs: use rb-tree" thing hit again during boot. Could this be
> > the root cause of this all?
> >
> > I wrote down the following:
> >
> > RIP : rb_next
> >
> > Trace:
> >  sysfs_dir_pos
> >  sysfs_readdir
> >  ? sys_ioctl
> >  vfs_readdir
> >  sys_getdents
> 
> Thanks for reporting this.
> 
> Has this by any chance been resolved or stopped happening?

Yes.

> This looks for all of the world like something is stomping your sysfs
> dirents.   I haven't seen anyone else complaining so this seems like the
> problem is unique to your configuration.  Which suggests that it is not
> sysfs itself that is wrong.
> 
> I have been through the code a time or two and I haven't seen anything
> obviously wrong.  Everything that sysfs does is protected by the
> sysfs_mutex so the locking is very very simple.
> 
> My best guess of why now is that the rbtree code make a sysfs dirent
> 48 bytes larger.  And so it is much more exposed to these kinds of
> problems.

Sorry, but your subsystem was just accidentally hit by a bug in the
Radeon driver, that sometimes randomly writes 0 dwords somewhere to
memory after a kexec boot (see the rest of this huge thread).
It's still not fixed in mainline, because Linus refused to take the fix
this late in the series.

WARNING: at mm/slub.c:3357, kernel BUG at mm/slub.c:3413

Commit Message

Comments

Patch