Patchwork ext4: Fix max file size of extent format file

login
register
mail settings
Submitter Kazuya Mio
Date May 11, 2011, 8:08 a.m.
Message ID <4DCA4406.4030601@sx.jp.nec.com>
Download mbox | patch
Permalink /patch/95171/
State Superseded
Headers show

Comments

Kazuya Mio - May 11, 2011, 8:08 a.m.
We hit BUG_ON in ext4_ext_put_gap_in_cache() while creating a file
whose size is the max file size of extent format. Because the extent cache
length is 0 when we allocate two blocks to the file offset 2^32-2, and then
the offset 2^32-1. To fix it, we decrease the max file size to
(2^32-2)*blocksize. In this way, we would be able to allocate a block up to
the offset 2^32-2. I think there is no data loss because we can read all files
created before applying this patch.

How to reproduce:
I'm running 2.6.39-rc6. Note that i386 architecture and 4KB blocksize cannot
reproduce this problem.

# dd if=/dev/zero of=/mnt/mp1/file bs=<blocksize> count=1 seek=$((2**32-2))
# sync
# dd if=/dev/zero of=/mnt/mp1/file bs=<blocksize> count=1 seek=$((2**32-1))

Signed-off-by: Kazuya Mio <k-mio@sx.jp.nec.com>
---
 fs/ext4/super.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lukas Czerner - May 13, 2011, 6:34 a.m.
On Wed, 11 May 2011, Kazuya Mio wrote:

> We hit BUG_ON in ext4_ext_put_gap_in_cache() while creating a file
> whose size is the max file size of extent format. Because the extent cache
> length is 0 when we allocate two blocks to the file offset 2^32-2, and then
> the offset 2^32-1. To fix it, we decrease the max file size to
> (2^32-2)*blocksize. In this way, we would be able to allocate a block up to
> the offset 2^32-2. I think there is no data loss because we can read all files
> created before applying this patch.
> 
> How to reproduce:
> I'm running 2.6.39-rc6. Note that i386 architecture and 4KB blocksize cannot
> reproduce this problem.
> 
> # dd if=/dev/zero of=/mnt/mp1/file bs=<blocksize> count=1 seek=$((2**32-2))
> # sync
> # dd if=/dev/zero of=/mnt/mp1/file bs=<blocksize> count=1 seek=$((2**32-1))

Hi Kazuya,

Thanks for the patch, however I think that there is a better solution
than lowering the max file size, which is not necessary. I would rather
fix ext4_ext_put_gap_in_cache() and allow to invalidate the cache by
setting the length to zero.

Please see the following patch, if it works for you.

Thanks!
-Lukas

> 
> Signed-off-by: Kazuya Mio <k-mio@sx.jp.nec.com>
> ---
>  fs/ext4/super.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> diff --git a/fs/ext4/super.c b/fs/ext4/super.c
> index 8553dfb..fce0249 100644
> --- a/fs/ext4/super.c
> +++ b/fs/ext4/super.c
> @@ -2248,8 +2248,8 @@ static loff_t ext4_max_size(int blkbits, int has_huge_files)
>  
>  	/* 32-bit extent-start container, ee_block */
>  	res = 1LL << 32;
> -	res <<= blkbits;
>  	res -= 1;
> +	res <<= blkbits;
>  
>  	/* Sanity check against vm- & vfs- imposed limits */
>  	if (res > upper_limit)
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Andreas Dilger - June 3, 2011, 5:35 p.m.
On 2011-05-13, at 12:34 AM, Lukas Czerner wrote:
> On Wed, 11 May 2011, Kazuya Mio wrote:
>> We hit BUG_ON in ext4_ext_put_gap_in_cache() while creating a file
>> whose size is the max file size of extent format. Because the extent cache
>> length is 0 when we allocate two blocks to the file offset 2^32-2, and then
>> the offset 2^32-1. To fix it, we decrease the max file size to
>> (2^32-2)*blocksize. In this way, we would be able to allocate a block up to
>> the offset 2^32-2. I think there is no data loss because we can read all files
>> created before applying this patch.
>> 
>> How to reproduce:
>> I'm running 2.6.39-rc6. Note that i386 architecture and 4KB blocksize cannot
>> reproduce this problem.
>> 
>> # dd if=/dev/zero of=/mnt/mp1/file bs=<blocksize> count=1 seek=$((2**32-2))
>> # sync
>> # dd if=/dev/zero of=/mnt/mp1/file bs=<blocksize> count=1 seek=$((2**32-1))
> 
> Hi Kazuya,
> 
> Thanks for the patch, however I think that there is a better solution
> than lowering the max file size, which is not necessary. I would rather
> fix ext4_ext_put_gap_in_cache() and allow to invalidate the cache by
> setting the length to zero.
> 
> Please see the following patch, if it works for you.

This thread has stalled.  Until we can come up with a better fix, I think
we should land Kazuya's patch for the current merge window and then worry
about allocating that last 4kB at the end of the 16TB file later (2.3e-6%
of the file :-).

>> Signed-off-by: Kazuya Mio <k-mio@sx.jp.nec.com>
>> ---
>> fs/ext4/super.c |    2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>> diff --git a/fs/ext4/super.c b/fs/ext4/super.c
>> index 8553dfb..fce0249 100644
>> --- a/fs/ext4/super.c
>> +++ b/fs/ext4/super.c
>> @@ -2248,8 +2248,8 @@ static loff_t ext4_max_size(int blkbits, int has_huge_files)
>> 
>> 	/* 32-bit extent-start container, ee_block */
>> 	res = 1LL << 32;
>> -	res <<= blkbits;
>> 	res -= 1;
>> +	res <<= blkbits;
>> 
>> 	/* Sanity check against vm- & vfs- imposed limits */
>> 	if (res > upper_limit)
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers, Andreas





--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lukas Czerner - June 3, 2011, 6:08 p.m.
On Fri, 3 Jun 2011, Andreas Dilger wrote:

> On 2011-05-13, at 12:34 AM, Lukas Czerner wrote:
> > On Wed, 11 May 2011, Kazuya Mio wrote:
> >> We hit BUG_ON in ext4_ext_put_gap_in_cache() while creating a file
> >> whose size is the max file size of extent format. Because the extent cache
> >> length is 0 when we allocate two blocks to the file offset 2^32-2, and then
> >> the offset 2^32-1. To fix it, we decrease the max file size to
> >> (2^32-2)*blocksize. In this way, we would be able to allocate a block up to
> >> the offset 2^32-2. I think there is no data loss because we can read all files
> >> created before applying this patch.
> >> 
> >> How to reproduce:
> >> I'm running 2.6.39-rc6. Note that i386 architecture and 4KB blocksize cannot
> >> reproduce this problem.
> >> 
> >> # dd if=/dev/zero of=/mnt/mp1/file bs=<blocksize> count=1 seek=$((2**32-2))
> >> # sync
> >> # dd if=/dev/zero of=/mnt/mp1/file bs=<blocksize> count=1 seek=$((2**32-1))
> > 
> > Hi Kazuya,
> > 
> > Thanks for the patch, however I think that there is a better solution
> > than lowering the max file size, which is not necessary. I would rather
> > fix ext4_ext_put_gap_in_cache() and allow to invalidate the cache by
> > setting the length to zero.
> > 
> > Please see the following patch, if it works for you.
> 
> This thread has stalled.  Until we can come up with a better fix, I think
> we should land Kazuya's patch for the current merge window and then worry
> about allocating that last 4kB at the end of the 16TB file later (2.3e-6%
> of the file :-).

Wait just a sec, I am testing new version of my patch. I have tried and
failed with my approach. It seems that there is no way to do that
without also changing the struct ext4_ext_cache and other structures,
resulting in different on-disk format.

So I think that Kazuyas fix is good after all, however there is
unconsistency in using EXT_MAX_BLOCK and in comments as well. So I fixed
that as well.

Also I hit another bug, while testing this change so I'll send both as
a series shortly.

Thanks!
-Lukas

> 
> >> Signed-off-by: Kazuya Mio <k-mio@sx.jp.nec.com>
> >> ---
> >> fs/ext4/super.c |    2 +-
> >> 1 file changed, 1 insertion(+), 1 deletion(-)
> >> diff --git a/fs/ext4/super.c b/fs/ext4/super.c
> >> index 8553dfb..fce0249 100644
> >> --- a/fs/ext4/super.c
> >> +++ b/fs/ext4/super.c
> >> @@ -2248,8 +2248,8 @@ static loff_t ext4_max_size(int blkbits, int has_huge_files)
> >> 
> >> 	/* 32-bit extent-start container, ee_block */
> >> 	res = 1LL << 32;
> >> -	res <<= blkbits;
> >> 	res -= 1;
> >> +	res <<= blkbits;
> >> 
> >> 	/* Sanity check against vm- & vfs- imposed limits */
> >> 	if (res > upper_limit)
> >> 
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> Cheers, Andreas
> 
> 
> 
> 
> 
>

Patch

diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 8553dfb..fce0249 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -2248,8 +2248,8 @@  static loff_t ext4_max_size(int blkbits, int has_huge_files)
 
 	/* 32-bit extent-start container, ee_block */
 	res = 1LL << 32;
-	res <<= blkbits;
 	res -= 1;
+	res <<= blkbits;
 
 	/* Sanity check against vm- & vfs- imposed limits */
 	if (res > upper_limit)