diff mbox

Exciting :-( adventures in metadata checksumming

Message ID 20120806164700.GA7896@thunk.org
State Accepted, archived
Headers show

Commit Message

Theodore Ts'o Aug. 6, 2012, 4:47 p.m. UTC 
On Sat, Aug 04, 2012 at 06:41:39PM -0400, George Spelvin wrote:
> > commit 037b728b8a6a775e9a5e03fd24b1008d633c1cb4
> > Author: Theodore Ts'o <tytso@mit.edu>
> > Date:   Sat Aug 4 16:56:55 2012 -0400
> >
> >     Put ELF_OTHER_LIBS in the right place for the linker
> 
> Thanks for the update.  That produces (following your procedure from
> the previous e-mail exactly, modulo directory names) a *different* error...

Oops.  Sorry, I screwed up that last patch, because I forgot that we
were in the elfshared directory.  Since I had comerr-dev, et. al.,
installed, I didn't notice because I was just linking against the
system libraries.  Thanks for testing and pointing out this!

Here's a replacement patch which should work; I've tested it after
deleting comerr-dev, e2fslibs-dev, etc., and it works, so I'm pretty
confident I got it right this time.

Regards,

						- Ted

commit d5aa6a82b37a0e78d8882601e6ad9da9d9dcb4da
Author: Theodore Ts'o <tytso@mit.edu>
Date:   Sat Aug 4 16:56:55 2012 -0400

    Put ELF_OTHER_LIBS in the right place for the linker
    
    Commit a7c17431b9 attempted to fix a problem where the system
    libraries might get used instead of local libraries for things like
    -lcom_err.  It tried to accomplish this by moving $(ELF_OTHER_LIBS) to
    before $(LDFLAGS).
    
    Unfortunately, this was the wrong fix; $(ELF_OTHER_LIBS) *MUST* be
    after the object files, or the linker might not pull in the necessary
    library and not include it into the DT_NEEDED section of the shared
    library.  The proper fix is to add a -L$(LIB) before $(LDFLAGS), and
    then remove the -L option from all of the ELF_OTHER_LIBS definitions
    in the library Makefiles.
    
    Addresses-Sourceforge-Bug: #3554345
    
    Cc: Olivier Blin <olivier.blin@softathome.com>
    Reported-by:  Mike Frysinger <vapier@gentoo.org>
    Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

George Spelvin Aug. 6, 2012, 6:14 p.m. UTC | #1 
> Here's a replacement patch which should work; I've tested it after
> deleting comerr-dev, e2fslibs-dev, etc., and it works, so I'm pretty
> confident I got it right this time.

Thanks!  Successful build on Ubuntu 12.04.
(No patch required on Debian/unstable.)

(Well, some problems with multiarch and not wanting to *install* without
a new libcomerr2:i386 to match the new libcomerr2:amd64, but that's
something I have to figure out.)
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Theodore Ts'o Aug. 6, 2012, 10:12 p.m. UTC | #2 
On Mon, Aug 06, 2012 at 02:14:41PM -0400, George Spelvin wrote:
> 
> (Well, some problems with multiarch and not wanting to *install* without
> a new libcomerr2:i386 to match the new libcomerr2:amd64, but that's
> something I have to figure out.)

Obvious stupid question --- you don't have a 32-bit krb5 package or
something else which depewnds on libcomerr2:i386?

I'm using Debain testing and haven't noted any issues like this...

    	  	 	     	     	       - Ted




--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George Spelvin Aug. 6, 2012, 10:59 p.m. UTC | #3 
> Obvious stupid question --- you don't have a 32-bit krb5 package or
> something else which depewnds on libcomerr2:i386?

Yes, exactly.  Just deleting the damn thing was the first thing I
thought of.  The dependency chain is long and annoying and ends in
something that's not 64-bit safe and so depends on ia32-libs.

That's a catch-all package; I don't know if the dependency is real
or not.  I worked around it in the simple stupid way by editing
/var/lig/dpkg/status and telling it that the installed verion is
1.43~WIP-2012-08-02-1; it's not like there's a meaningful difference.

> I'm using Debain testing and haven't noted any issues like this...

It's the Ubuntu machine.  Not your problem!

If you care, it ends up dying with the following, but I don't expect or need
your help.

i686-linux-gnu-gcc -c -I. -I../lib -I/tmp/e2fsprogs/lib -D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -D__NO_STRING_INLINES /root/e2fsprogs/e2fsck/sigcatcher.c -o sigcatcher.o
i686-linux-gnu-gcc -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-rpath-link,../lib -rdynamic -o e2fsck dict.o unix.o e2fsck.o super.o pass1.o pass1b.o pass2.o pass3.o pass4.o pass5.o journal.o badblocks.o util.o dirinfo.o dx_dirinfo.o ehandler.o problem.o message.o quota.o recovery.o region.o revoke.o ea_refcount.o rehash.o profile.o prof_err.o logfile.o sigcatcher.o  ../lib/libquota.a ../lib/libext2fs.so ../lib/libcom_err.so     ../lib/libe2p.so 
unix.o: In function `PRS':
/tmp/e2fsprogs/e2fsck/unix.c:765: undefined reference to `blkid_get_cache'
/tmp/e2fsprogs/e2fsck/unix.c:857: undefined reference to `blkid_get_devname'
/tmp/e2fsprogs/e2fsck/unix.c:931: undefined reference to `blkid_get_devname'
e2fsck.o: In function `e2fsck_free_context':
/tmp/e2fsprogs/e2fsck/e2fsck.c:179: undefined reference to `blkid_put_cache'
super.o: In function `check_super_block':
/tmp/e2fsprogs/e2fsck/super.c:728: undefined reference to `uuid_is_null'
/tmp/e2fsprogs/e2fsck/super.c:730: undefined reference to `uuid_generate'
journal.o: In function `e2fsck_journal_load':
/tmp/e2fsprogs/e2fsck/journal.c:581: undefined reference to `uuid_is_null'
journal.o: In function `e2fsck_get_journal':
/tmp/e2fsprogs/e2fsck/journal.c:315: undefined reference to `uuid_is_null'
/tmp/e2fsprogs/e2fsck/journal.c:397: undefined reference to `uuid_unparse'
/tmp/e2fsprogs/e2fsck/journal.c:398: undefined reference to `blkid_get_devname'
/tmp/e2fsprogs/e2fsck/journal.c:401: undefined reference to `blkid_devno_to_devname'
journal.o: In function `e2fsck_check_ext3_journal':
/tmp/e2fsprogs/e2fsck/journal.c:766: undefined reference to `uuid_is_null'
journal.o: In function `e2fsck_journal_reset_super':
/tmp/e2fsprogs/e2fsck/journal.c:683: undefined reference to `uuid_generate'
journal.o: In function `e2fsck_fix_ext3_journal_hint':
/tmp/e2fsprogs/e2fsck/journal.c:1117: undefined reference to `uuid_is_null'
/tmp/e2fsprogs/e2fsck/journal.c:1120: undefined reference to `uuid_unparse'
/tmp/e2fsprogs/e2fsck/journal.c:1121: undefined reference to `blkid_get_devname'
dirinfo.o: In function `setup_tdb':
/tmp/e2fsprogs/e2fsck/dirinfo.c:63: undefined reference to `uuid_unparse'
collect2: error: ld returned 1 exit status
make[3]: *** [e2fsck] Error 1
make[3]: Leaving directory `/tmp/e2fsprogs/debian/BUILD-STD/e2fsck'
make[2]: *** [all-progs-recursive] Error 1
make[2]: Leaving directory `/tmp/e2fsprogs/debian/BUILD-STD'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/tmp/e2fsprogs/debian/BUILD-STD'
make: *** [debian/stampdir/build-std-stamp] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
debuild: fatal error at line 1350:
dpkg-buildpackage -rfakeroot -d -us -uc -b -ai386 failed

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Theodore Ts'o Aug. 6, 2012, 11:25 p.m. UTC | #4 
On Mon, Aug 06, 2012 at 06:59:37PM -0400, George Spelvin wrote:
> dpkg-buildpackage: error: debian/rules build gave error exit status 2
> debuild: fatal error at line 1350:
> dpkg-buildpackage -rfakeroot -d -us -uc -b -ai386 failed

I don't think -ai386 will work for e2fsprogs since we need external
libraries; specifically, libblkid-dev and libuuid-dev and the dev
packages aren't multiarch compatible.  (In fact, I'm not sure -ai386
to build 32-bit packages in an x86 environment will work in general.
It's certainly not the standard way 32-bit packages are built.)

The failures you're seeing is because "pkg-config --libs blkid" is
returning a null string when run in the dpkg-buildpackage -ai386
environment --- which is not surprising, since we don't have a -32/-64
bit link libraries in libblkid-dev.

So if you want to build a 32-bit set of packages of e2fsprogs, you'll
need to make a 32-bit build environment as a chroot, using debootstrap
and build e2fsprogs in the 32-bit chroot.

That's actually how I build my packages for Debian, BTW --- I have a
32-bit chroot, and I build the binary packages for i386 and upload
them from there.  I let the autobuilders build the 64-bit binary
packages from the source upload.  That way, the most commonly used
binary packages are built in a standard autobuilder environment, and
are not subject to the vagracies of my build environment.  It also
means that I don't have to worry about the build scripts bitrot for
the 32-bit packages.  (I also build 64-bit debs for my own use --- but
I don't upload them.)

Regards,

	   	      	     	      	 - Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George Spelvin Aug. 8, 2012, 1:39 p.m. UTC | #5 
The machine I reported earlier issues has not glitched yet.
However, I have some more fun to report.

To further test metadata checksums, I enabled it on my desktop machine.
(Also has SSE4.2, so the checksum overhead should be minimal.)  This is
a Debian/unstable lachine, with 64-bit kernel (v3.5 + ext4-for-linus)
and 32-bit userland, with e2fsprogs from the next branch.

This time I included the root FS, which gave some interesting issues
last night during the network backup run.  This morning I was greeted by:

BUG: unable to handle kernel paging request at fffffffffffffff8
IP: [<ffffffff810ffbf9>] ext4_readdir+0x1e2/0x5a8
PGD 1589067 PUD 158a067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:  battery nfds exportfs deflate zlib_deflate zlib_inflate ctr <whole bunch of crypto modules snipped> crypto_null af_key xfrm_algo fuse ftdi_sio usbserial r8199
Pid: 31650, comm: rsync Not tainted 3.5.0-00032-g0e1gf37 #55 Gigabyte Technology Co., Ltd. H55M-UD2H/H55M-UD2H
RIP: 0010:[<ffffffff810ffbf9>]  [<ffffffff810ffbf9>] ext4_readdir+0x1e2/0x5a8
RSP: 0018:ffff88010eb65e38  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8800b6763300 RCX: 0000000000000000
RDX: ffffffff810e5d99 RSI: ffff88010eb65f40 RDI: ffff8800b6763300
RBP: ffff88010eb65ed8 R08: 0000000000013750 R09: ffffea000445bd40
R10: 0000000000000000 R11: ffffffff8111a578 R12: ffff880108808740
R13: ffff88003ab13748 R14: ffff8801137e8400 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880117c80000(0063) knlGS:00000000f760d6c0
CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
CR2: fffffffffffffff8 CR3: 000000011177e000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process rsync (pid: 31650, threadinfo ffff88010eb64000, task ffff880111d6a5e0)
Stack:
 ffff88010eb65f08 ffffffff810bc5e9 ffff880113782220 ffff880000000000
 00000005756e69e4 ffff88011379501e ffffffff810e5d99 ffff88010eb65f40 
 ffff88003ab13748 ffff88003ab13748 0000000000000000 ffffffff813dc01c
Call Trace:
 [<ffffffff810bc5e9>] ? do_filp_open+0x33/0x81
 [<ffffffff810e5d99>] ? compat_filldir+0xdd/0xdd
 [<ffffffff813dc01c>] ? _cond_resched+0x9/0x1d
 [<ffffffff8104437b>] ? shoud_resched+0x9/0x28
 [<ffffffff810e5d99>] ? compat_filldir+0xdd/0xdd
 [<ffffffff810be52e>] vfs_readdir+0x61/0x9a
 [<ffffffff810aa3b5>] ? kmem_cache_free+0x15/0x6e
 [<ffffffff810e729e>] compat_sys_getdents64+0x72/0xcc
 [<ffffffff813de59b>] sysenter_dispatch+0x7/0x1e
Code: 00 83 f8 00 0f 8c a6 00 00 00 75 02 eb 6d 4c 89 e7 e8 49 5b 09 00 49 89 44 24 08 49 8b 4c 24 08 48 8b 55 90 48 89 df 48 8b 75 98 <8b> 41 f8 41 89 44 24 20 8b 41 fc 48 83 e9 08 41 89 44 24 24 e8
RIP  [<ffffffff810ffbf9>] ext4_readdir+0x1e2/0x5a8
 RSP <ffff88010eb65e38>
CR2: fffffffffffffff8

(The above was hand-transcribed, so I *hope* I got all the hex correct!)

Anyway, on reboot, the system came up, but would not fsck, printing:

fsck.ext4: Superblock checksum does not match superblock while trying to open /dev/sda2
/dev/sda2:
The superblock could not be read or does not describe a correct ext2
filesstem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

fsck died with exit code 8


So I logged in by hand and tried to run the recommended command, but!
Including "-b 8193" produced the above message, while *omitting* it
actually ran successfully.  I'm a little confused by that.
(The "fsck" wrapper invoked from /etc/init.d/checkroot.sh is 2.20.1-5.1.)

For some limited values of "successfully"; it sure found a lot of problems:

(This is the second run; I stopped the first and started capturing it
when it was obvious pencil and paper was impractical.)

Script started on Wed Aug  8 08:40:45 2012
/run# e2fsck -y -v -C0 /dev/sda2
e2fsck 1.43-WIP (1-Aug-2012)
root was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
... some similar errors from earlier run omitted ...
Inode 947580 checksum does not match inode.  Clear? yes
Inode 947581 checksum does not match inode.  Clear? yes
Inode 947582 checksum does not match inode.  Clear? yes
Inode 947583 checksum does not match inode.  Clear? yes
Inode 947584 checksum does not match inode.  Clear? yes
Inode 947585 checksum does not match inode.  Clear? yes
Inode 947586 checksum does not match inode.  Clear? yes
Inode 947587 checksum does not match inode.  Clear? yes
Inode 947588 checksum does not match inode.  Clear? yes
Inode 947589 checksum does not match inode.  Clear? yes
Inode 947590 checksum does not match inode.  Clear? yes
Inode 947591 checksum does not match inode.  Clear? yes
... 315 additional deleted ...
Inode 947920 checksum does not match inode.  Clear? yes

Pass 2: Checking directory structure
Entry 'linux' in /usr/arm-linux-gnueabi/include (534036) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534570) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534582) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534660) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534681) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534695) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534737) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534785) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534796) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534805) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534820) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534824) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534860) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534911) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534947) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534959) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (534978) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (535012) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (535026) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (535045) has deleted/unused inode 534518.  Clear? yes
Entry '..' in ??? (535065) has deleted/unused inode 534518.  Clear? yes

Pass 3: Checking directory connectivity
Unconnected directory inode 534570 (...)
Connect to /lost+found? yes
... suplicates snipped ...

Pass 4: Checking reference counts
Inode 534036 ref count is 30, should be 29.  Fix? yes
Inode 534570 ref count is 3, should be 2.  Fix? yes
Inode 534582 ref count is 4, should be 3.  Fix? yes
Inode 534660 ref count is 3, should be 2.  Fix? yes
Inode 534681 ref count is 3, should be 2.  Fix? yes
Inode 534695 ref count is 3, should be 2.  Fix? yes
Inode 534737 ref count is 3, should be 2.  Fix? yes
Inode 534785 ref count is 3, should be 2.  Fix? yes
Unattached inode 534795  Connect to /lost+found? yes
Inode 534795 ref count is 2, should be 1.  Fix? yes
Inode 534796 ref count is 3, should be 2.  Fix? yes
Unattached inode 534797  Connect to /lost+found? yes
Inode 534797 ref count is 2, should be 1.  Fix? yes
... snip ...
Unattached inode 542215  Connect to /lost+found? yes
Inode 542215 ref count is 2, should be 1.  Fix? yes

Pass 5: Checking group summary information
Block bitmap differences:  -(3833858--3833859) -3833862 -(3833864--3833865) -3833867 -3833870 -(3833872--3833873)  ...2.5 MB snipped...
Fix? yes

Free blocks count wrong for group #0 (27905, counted=27767).
Fix? yes
Free blocks count wrong for group #1 (2607, counted=2109).
Fix? yes
Free blocks count wrong for group #2 (4062, counted=4164).
Fix? yes
... snip ...
Free blocks count wrong for group #289 (21421, counted=21608).
Fix? yes
Free blocks count wrong (6043152, counted=6026661).
Fix? yes

Inode bitmap differences:  -(26241--26243) -(26245--26247) -26253 -26263 -26268 -(26271--26272) -(26275--26276) -26278 ... 1,000,000 bytes snipped ...
Fix? yes

Free inodes count wrong for group #0 (2, counted=31).
Fix? yes
Free inodes count wrong for group #1 (0, counted=934).
Fix? yes
Free inodes count wrong for group #2 (0, counted=1847).
Fix? yes
... snip ...
Free inodes count wrong for group #288 (486, counted=468).
Fix? yes

Free inodes count wrong (693942, counted=693943).
Fix? yes

root: ***** FILE SYSTEM WAS MODIFIED *****
root: ***** REBOOT LINUX *****

      286777 inodes used (29.24%, out of 980720)
         278 non-contiguous files (0.1%)
         172 non-contiguous directories (0.1%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 262385/65
     3738850 blocks used (38.29%, out of 9765511)
           0 bad blocks
           0 large files

      238864 regular files
       21703 directories
         164 character device files
          10 block device files
           1 fifo
  4294967292 links
       26014 symbolic links (24132 fast symbolic links)
          12 sockets
------------
      286397 files

/run# e2fsck -f -v -C0 /dev/sda2
e2fsck 1.43-WIP (1-Aug-2012)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

      286777 inodes used (29.24%, out of 980720)
         278 non-contiguous files (0.1%)
         173 non-contiguous directories (0.1%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 262385/65
     3738850 blocks used (38.29%, out of 9765511)
           0 bad blocks
           0 large files

      238864 regular files
       21703 directories
         164 character device files
          10 block device files
           1 fifo
          36 links
       26014 symbolic links (24132 fast symbolic links)
          12 sockets
------------
      286804 files
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George Spelvin Aug. 8, 2012, 10:34 p.m. UTC | #6 
Yay!  It happened again!

[397113.459309] EXT4-fs error (device md0): ext4_iget:3821: inode #100691970: comm updatedb.mlocat: checksum invalid
[397113.459315] Aborting journal on device md0-8.
[397113.496222] EXT4-fs (md0): Remounting filesystem read-only

However, trying to preserve the evidence runs into a problem:

/root# e2image -Q /dev/md0 md0.qcow2
e2image 1.43-WIP (1-Aug-2012)
e2image: Inode checksum does not match inode while getting next inode
/root# ls -l md0.qcow2
-rw------- 1 root root 0 Aug  8 18:30 md0.qcow2

e2image aborts on the checksum error, preventing me from capturing a useful
image.

Can someone find a workaround QUICKLY?  I can't keep this FS read-only
for long.

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George Spelvin Aug. 8, 2012, 11:42 p.m. UTC | #7 
> Can someone find a workaround QUICKLY?  I can't keep this FS read-only
> for long.

I thought I had figured out a great workaround: Use 1.42.4, which doesn't
know how to check checksums.

But then I doscovered that it aborts and delivers a zero-length file
if there are filesystem inconsistencies, too!  So I get

e2image 1.42.4 (12-Jun-2012)
Illegal block number passed to ext2fs_mark_block_bitmap #3571066296 for in-use block map
Illegal block number passed to ext2fs_mark_block_bitmap #2895243190 for in-use block map
Illegal block number passed to ext2fs_mark_block_bitmap #3276895043 for in-use block map
Illegal block number passed to ext2fs_mark_block_bitmap #2488200263 for in-use block map
Illegal block number passed to ext2fs_mark_block_bitmap #2556839855 for in-use block map
... snip... (2671 total "Illegal block number passed" messages)
Illegal block number passed to ext2fs_mark_block_bitmap #3421917394 for in-use block map
Illegal block number passed to ext2fs_mark_block_bitmap #3469830505 for in-use block map
e2image: Illegal indirect block found while iterating over inode 85800474

I'm not sure this is The Right Thing To Do for a debugging tool.


The file system is a RAID-6, and repeated verifications have failed to find
RAID mismatches.

I am starting to suspect motherboard/RAM on this machine.  Already the bad
magic number error patterns looked odd to me, and I was just reminded that
we had to swap the RAM when it was first built so memtest8 would pass.
We ran it for many hours, but it *is* a consumer Intel box with no ECC.

And 8 GiB of RAM, and acting primarily as a file server, so FS metadata can
sit and bit-rot in RAM for a very long time.

I'm going to play with "hdparm -f" and drop_caches to see if I can make
the file system problems go away with no repair other than re-reading
from disk.

If so, That would confirm it as not ext4's problem.  Although it *would* be
a very cool debugging feature to re-check the checksum whenever a metadata
page is discarded from the buffer cache.

If the checksum matched when first read in, and doesn't when a supposedly
clean page is discarded, *something* is corrupting RAM.  (If you
assume that it's a single bit flip, then you can deduce the location
from the error syndrome.)


Anyway, thanks for the help!
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George Spelvin Aug. 9, 2012, 5 a.m. UTC | #8 
I think indeed it was a RAM issue.  After some cache-flushing, e2fsck
on the file system with the checksum errors reported no errors, even
though there wasn't even a reboot in between.

(Then I discovered the "e2fsck -F" flag, which would have been
a lot simpler than what I did instead.  Oh, well.)

Looking in the BIOS, I found some suspicious timing parameters that
might be responsible, and reset them to stock.  I'll know for sure if
the problem stays away for a month, but in the mean time, thank you
everyone for your help and patience, and consider the matter
resolved unless I find more problems.

Sorry for the false alarm.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
George Spelvin Aug. 9, 2012, 11:48 p.m. UTC | #9 
Havimg, I think, fixed the entire memory corruption problem that inspired
me to turn on metadata checksums, I want to revert to the standard Ubuntu
kernel, but first I need to figure out how to turn them off!

The obvious answer is with tune2fs:

> ~# tune2fs -O ^metadata_csum /dev/md0
> tune2fs 1.43-WIP (1-Aug-2012)
> rewrite_extents: Corrupt extent header while rewriting extents
> ~# 

Arrgh!  The file system passes e2fsck fine.  Well, *now* it doesn't; it
bitches about a zillion missing directory checksums and the lack
of lost+found:

> Directory inode 82676842, block #1, offset 3260: directory passes checks but fails checksum
> Fix? yes
> 
> Directory inode 82676842, block #2, offset 3264: directory passes checks but fails checksum
> Fix? yes
> 
> Directory inode 82677333, block #1, offset 3260: directory passes checks but fails checksum
> Fix? yes
> 
> Directory inode 82677333, block #2, offset 1328: directory passes checks but fails checksum
> Fix? yes
> 
> Directory inode 82675733, block #3, offset 124: directory passes checks but fails checksum
> Fix? yes
> 
> Directory inode 82676842, block #3, offset 432: directory passes checks but fails checksum
> Fix? yes
> 
> Pass 3: Checking directory connectivity
> Error while trying to find /lost+found: Directory block checksum does not match directory block
> /lost+found not found.  Create? yes
> 
> Error creating /lost+found directory (ext2fs_link): Directory block checksum does not match directory block
> Pass 3A: Optimizing directories
> Pass 4: Checking reference counts
> Unattached inode 12
> Connect to /lost+found? yes
> 
> Pass 5: Checking group summary information
> Block bitmap differences:  -937953738
> Fix? yes
> 
> Free blocks count wrong for group #28624 (28170, counted=28171).
> Fix? yes
> 
> Free blocks count wrong (841115975, counted=841115976).
> Fix? yes
> 
> 
> /dev/md0: ***** FILE SYSTEM WAS MODIFIED *****

But *after* that, it passes fine.

Unfortunately, I don't know where the problem is or I'd see if I could
just delete the damn problematic file.  (Or move it somewhere else
temporarily.)

I'm adding debugging to tune2fs and trying to track down the source.

Step 1:
eh_magic = dabe != f30a
Problem with extent of inode #85800449
rewrite_extents: Corrupt extent header while rewriting extents

Step 2:
debugfs:  ncheck 85800449
Inode   Pathname
debugfs:  testi <85800449>
Inode 85800449 is not in use

Step 3: ???
The print is in ext2fs_extent_open2, and I'm just printing the "ino"
parameter if ext2fs_extent_header_verify fails.

Why is tune2fs looking at an inode that's not in use?
That *would* explain the magic number error...

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/lib/Makefile.elf-lib b/lib/Makefile.elf-lib
index c66281c..78479d3 100644
--- a/lib/Makefile.elf-lib
+++ b/lib/Makefile.elf-lib
@@ -24,8 +24,9 @@  image:		$(ELF_LIB)
 
 $(ELF_LIB): $(OBJS)
 	$(E) "	GEN_ELF_SOLIB $(ELF_LIB)"
-	$(Q) (cd elfshared; $(CC) --shared -o $(ELF_LIB) $(ELF_OTHER_LIBS) \
-		$(LDFLAGS) -Wl,-soname,$(ELF_SONAME) $(OBJS))
+	$(Q) (cd elfshared; $(CC) --shared -o $(ELF_LIB) \
+		-L$(top_builddir)/../lib $(LDFLAGS) \
+		-Wl,-soname,$(ELF_SONAME) $(OBJS) $(ELF_OTHER_LIBS))
 	$(Q) $(MV) elfshared/$(ELF_LIB) .
 	$(Q) $(RM) -f ../$(ELF_LIB) ../$(ELF_IMAGE).so ../$(ELF_SONAME)
 	$(Q) (cd ..; $(LN) $(LINK_BUILD_FLAGS) \
diff --git a/lib/Makefile.solaris-lib b/lib/Makefile.solaris-lib
index 66f2b4c..5990be8 100644
--- a/lib/Makefile.solaris-lib
+++ b/lib/Makefile.solaris-lib
@@ -24,8 +24,9 @@  image:		$(ELF_LIB)
 
 $(ELF_LIB): $(OBJS)
 	$(E) "	GEN_ELF_SOLIB $(ELF_LIB)"
-	$(Q) (cd elfshared; $(CC) --shared -o $(ELF_LIB) $(ELF_OTHER_LIBS) \
-		$(LDFLAGS) -Wl,-h,$(ELF_SONAME) $(OBJS))
+	$(Q) (cd elfshared; $(CC) --shared -o $(ELF_LIB) \
+		-L$(top_builddir)/../lib $(LDFLAGS) \
+		-Wl,-h,$(ELF_SONAME) $(OBJS) $(ELF_OTHER_LIBS))
 	$(Q) $(MV) elfshared/$(ELF_LIB) .
 	$(Q) $(RM) -f ../$(ELF_LIB) ../$(ELF_IMAGE).so ../$(ELF_SONAME)
 	$(Q) (cd ..; $(LN) $(LINK_BUILD_FLAGS) \
diff --git a/lib/blkid/Makefile.in b/lib/blkid/Makefile.in
index f23a137..0ec8564 100644
--- a/lib/blkid/Makefile.in
+++ b/lib/blkid/Makefile.in
@@ -36,7 +36,7 @@  ELF_SO_VERSION = 1
 ELF_IMAGE = libblkid
 ELF_MYDIR = blkid
 ELF_INSTALL_DIR = $(root_libdir)
-ELF_OTHER_LIBS = -L../.. -luuid
+ELF_OTHER_LIBS = -luuid
 
 BSDLIB_VERSION = 2.0
 BSDLIB_IMAGE = libblkid
diff --git a/lib/ext2fs/Makefile.in b/lib/ext2fs/Makefile.in
index 0d9ac21..fc196fb 100644
--- a/lib/ext2fs/Makefile.in
+++ b/lib/ext2fs/Makefile.in
@@ -180,7 +180,7 @@  ELF_SO_VERSION = 2
 ELF_IMAGE = libext2fs
 ELF_MYDIR = ext2fs
 ELF_INSTALL_DIR = $(root_libdir)
-ELF_OTHER_LIBS = -L../.. -lcom_err
+ELF_OTHER_LIBS = -lcom_err
 
 BSDLIB_VERSION = 2.1
 BSDLIB_IMAGE = libext2fs
diff --git a/lib/quota/Makefile.in b/lib/quota/Makefile.in
index 2851eac..720befd 100644
--- a/lib/quota/Makefile.in
+++ b/lib/quota/Makefile.in
@@ -31,7 +31,7 @@  LIBDIR= quota
 #ELF_IMAGE = libquota
 #ELF_MYDIR = quota
 #ELF_INSTALL_DIR = $(root_libdir)
-#ELF_OTHER_LIBS = -L../.. -lext2fs
+#ELF_OTHER_LIBS = -lext2fs
 
 #BSDLIB_VERSION = 1.0
 #BSDLIB_IMAGE = libquota
diff --git a/lib/ss/Makefile.in b/lib/ss/Makefile.in
index 19413cc..c396f2d 100644
--- a/lib/ss/Makefile.in
+++ b/lib/ss/Makefile.in
@@ -20,7 +20,7 @@  ELF_SO_VERSION = 2
 ELF_IMAGE = libss
 ELF_MYDIR = ss
 ELF_INSTALL_DIR = $(root_libdir)
-ELF_OTHER_LIBS = -L../.. -lcom_err $(DLOPEN_LIB)
+ELF_OTHER_LIBS = -lcom_err $(DLOPEN_LIB)
 
 BSDLIB_VERSION = 1.0
 BSDLIB_IMAGE = libss