diff mbox series

[U-Boot,v3,09/26] fs: fat: refactor write interface for a file offset

Message ID 20180911065922.19141-10-takahiro.akashi@linaro.org
State Accepted
Headers show
Series subject: fs: fat: extend FAT write operations | expand

Commit Message

AKASHI Takahiro Sept. 11, 2018, 6:59 a.m. UTC
From: AKASHI Takahiro <takahiro.akashi@linaro.org>

The current write implementation is quite simple: remove existing clusters
and then allocating new ones and filling them with data. This, inevitably,
enforces always writing from the beginning of a file.

As the first step to lift this restriction, fat_file_write() and
set_contents() are modified to accept an additional parameter, file offset
and further re-factored so that, in the next patch, all the necessary code
will be put into set_contents().

Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
---
 fs/fat/fat_write.c | 179 ++++++++++++++++-----------------------------
 1 file changed, 65 insertions(+), 114 deletions(-)

Comments

Faiz Abbas March 12, 2019, 8:41 a.m. UTC | #1
Hi Akashi,

On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
> From: AKASHI Takahiro <takahiro.akashi@linaro.org>
> 
> The current write implementation is quite simple: remove existing clusters
> and then allocating new ones and filling them with data. This, inevitably,
> enforces always writing from the beginning of a file.
> 
> As the first step to lift this restriction, fat_file_write() and
> set_contents() are modified to accept an additional parameter, file offset
> and further re-factored so that, in the next patch, all the necessary code
> will be put into set_contents().
> 
> Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
> ---

My fatwrite, fatload and compare tests are failing in MMC with this
commit. This is what I see:

=> fatwrite mmc 0 ${loadaddr} test 0x2000000
33554432 bytes written
=> fatload mmc 0 84000000 test
33554432 bytes read in 2149 ms (14.9 MiB/s)
=> cmp.b 82000000 84000000 0x2000000
byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
Total of 806912 byte(s) were the same
=>

Reverting this commit fixes this issue for me.

Thanks,
Faiz
Faiz Abbas March 13, 2019, 5:11 p.m. UTC | #2
Tom,

On 3/12/2019 2:11 PM, Faiz Abbas wrote:
> Hi Akashi,
> 
> On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
>> From: AKASHI Takahiro <takahiro.akashi@linaro.org>
>>
>> The current write implementation is quite simple: remove existing clusters
>> and then allocating new ones and filling them with data. This, inevitably,
>> enforces always writing from the beginning of a file.
>>
>> As the first step to lift this restriction, fat_file_write() and
>> set_contents() are modified to accept an additional parameter, file offset
>> and further re-factored so that, in the next patch, all the necessary code
>> will be put into set_contents().
>>
>> Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
>> ---
> 
> My fatwrite, fatload and compare tests are failing in MMC with this
> commit. This is what I see:
> 
> => fatwrite mmc 0 ${loadaddr} test 0x2000000
> 33554432 bytes written
> => fatload mmc 0 84000000 test
> 33554432 bytes read in 2149 ms (14.9 MiB/s)
> => cmp.b 82000000 84000000 0x2000000
> byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
> Total of 806912 byte(s) were the same
> =>
> 
> Reverting this commit fixes this issue for me.
> 
> Thanks,
> Faiz
> 

We have about 100 boards failing to boot from SD card because of fat
filesystem corruption in our test farm (likely) because of this patch. I
am afraid we will have to revert it until something stable can be
figured out. Will send out reverts tomorrow. Hopefully they can be
reviewed and merged quickly.

Thanks,
Faiz
Tom Rini March 13, 2019, 5:25 p.m. UTC | #3
On Wed, Mar 13, 2019 at 10:41:15PM +0530, Rizvi, Mohammad Faiz Abbas wrote:
> Tom,
> 
> On 3/12/2019 2:11 PM, Faiz Abbas wrote:
> > Hi Akashi,
> > 
> > On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
> >> From: AKASHI Takahiro <takahiro.akashi@linaro.org>
> >>
> >> The current write implementation is quite simple: remove existing clusters
> >> and then allocating new ones and filling them with data. This, inevitably,
> >> enforces always writing from the beginning of a file.
> >>
> >> As the first step to lift this restriction, fat_file_write() and
> >> set_contents() are modified to accept an additional parameter, file offset
> >> and further re-factored so that, in the next patch, all the necessary code
> >> will be put into set_contents().
> >>
> >> Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
> >> ---
> > 
> > My fatwrite, fatload and compare tests are failing in MMC with this
> > commit. This is what I see:
> > 
> > => fatwrite mmc 0 ${loadaddr} test 0x2000000
> > 33554432 bytes written
> > => fatload mmc 0 84000000 test
> > 33554432 bytes read in 2149 ms (14.9 MiB/s)
> > => cmp.b 82000000 84000000 0x2000000
> > byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
> > Total of 806912 byte(s) were the same
> > =>
> > 
> > Reverting this commit fixes this issue for me.
> > 
> > Thanks,
> > Faiz
> > 
> 
> We have about 100 boards failing to boot from SD card because of fat
> filesystem corruption in our test farm (likely) because of this patch. I
> am afraid we will have to revert it until something stable can be
> figured out. Will send out reverts tomorrow. Hopefully they can be
> reviewed and merged quickly.

OK, thanks!
AKASHI Takahiro March 18, 2019, 1:42 a.m. UTC | #4
Hi Faiz,

On Tue, Mar 12, 2019 at 02:11:08PM +0530, Faiz Abbas wrote:
> Hi Akashi,
> 
> On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
> > From: AKASHI Takahiro <takahiro.akashi@linaro.org>
> > 
> > The current write implementation is quite simple: remove existing clusters
> > and then allocating new ones and filling them with data. This, inevitably,
> > enforces always writing from the beginning of a file.
> > 
> > As the first step to lift this restriction, fat_file_write() and
> > set_contents() are modified to accept an additional parameter, file offset
> > and further re-factored so that, in the next patch, all the necessary code
> > will be put into set_contents().
> > 
> > Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
> > ---
> 
> My fatwrite, fatload and compare tests are failing in MMC with this
> commit. This is what I see:
> 
> => fatwrite mmc 0 ${loadaddr} test 0x2000000
> 33554432 bytes written
> => fatload mmc 0 84000000 test
> 33554432 bytes read in 2149 ms (14.9 MiB/s)
> => cmp.b 82000000 84000000 0x2000000
> byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
> Total of 806912 byte(s) were the same
> =>

I tried, but could not reproduce this issue.
(v2019.04-rc2)

What I did was:
- On host, create a vfat file system and a random data file.
     dd of=vfat128M.img bs=1M count=128
     mkfs -t vfat vfat128M.img ; mount ...
     head -c 32m /dev/urandom > 32m.data

- On qemu, try fatwrite
  (try 1)
  => fatload scsi 0:0 50000000 /32m.data 2000000
  33554432 bytes read in 539 ms (59.4 MiB/s)
  => fatwrite scsi 0:0 50000000 /32m_w.data 2000000
  33554432 bytes written
  => fatls scsi 0:0 /
  33554432   32m.data
  33554432   32m_w.data

  2 file(s), 0 dir(s)

  => fatload scsi 0:0 52000000 /32m_w.data
  33554432 bytes read in 537 ms (59.6 MiB/s)
  => cmp.b 50000000 52000000 2000000
  Total of 33554432 byte(s) were the same

  (try 2)
  => fatwrite scsi 0:0 54000000 /32m_w2.data 2000000
  33554432 bytes written
  => fatload scsi 0:0 56000000 /32m_w2.data    
  33554432 bytes read in 537 ms (59.6 MiB/s)
  => cmp.b 54000000 56000000 2000000
  Total of 33554432 byte(s) were the same

- I also confirmed that 32m.data and 32m_w.data are the same
  on the host.

> Reverting this commit fixes this issue for me.

So, first let me ask some questions:
- What is the size of your mmc memory?
- How did you format it?
- How many files are already there in the root directory?

Thanks,
-Takahiro Akashi


> Thanks,
> Faiz
> 
> 
> 
> 
>
Tom Rini March 18, 2019, 1:44 a.m. UTC | #5
On Mon, Mar 18, 2019 at 10:42:31AM +0900, Akashi, Takahiro wrote:
> Hi Faiz,
> 
> On Tue, Mar 12, 2019 at 02:11:08PM +0530, Faiz Abbas wrote:
> > Hi Akashi,
> > 
> > On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
> > > From: AKASHI Takahiro <takahiro.akashi@linaro.org>
> > > 
> > > The current write implementation is quite simple: remove existing clusters
> > > and then allocating new ones and filling them with data. This, inevitably,
> > > enforces always writing from the beginning of a file.
> > > 
> > > As the first step to lift this restriction, fat_file_write() and
> > > set_contents() are modified to accept an additional parameter, file offset
> > > and further re-factored so that, in the next patch, all the necessary code
> > > will be put into set_contents().
> > > 
> > > Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
> > > ---
> > 
> > My fatwrite, fatload and compare tests are failing in MMC with this
> > commit. This is what I see:
> > 
> > => fatwrite mmc 0 ${loadaddr} test 0x2000000
> > 33554432 bytes written
> > => fatload mmc 0 84000000 test
> > 33554432 bytes read in 2149 ms (14.9 MiB/s)
> > => cmp.b 82000000 84000000 0x2000000
> > byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
> > Total of 806912 byte(s) were the same
> > =>
> 
> I tried, but could not reproduce this issue.
> (v2019.04-rc2)
> 
> What I did was:
> - On host, create a vfat file system and a random data file.
>      dd of=vfat128M.img bs=1M count=128
>      mkfs -t vfat vfat128M.img ; mount ...
>      head -c 32m /dev/urandom > 32m.data
> 
> - On qemu, try fatwrite
>   (try 1)
>   => fatload scsi 0:0 50000000 /32m.data 2000000
>   33554432 bytes read in 539 ms (59.4 MiB/s)
>   => fatwrite scsi 0:0 50000000 /32m_w.data 2000000
>   33554432 bytes written
>   => fatls scsi 0:0 /
>   33554432   32m.data
>   33554432   32m_w.data
> 
>   2 file(s), 0 dir(s)
> 
>   => fatload scsi 0:0 52000000 /32m_w.data
>   33554432 bytes read in 537 ms (59.6 MiB/s)
>   => cmp.b 50000000 52000000 2000000
>   Total of 33554432 byte(s) were the same
> 
>   (try 2)
>   => fatwrite scsi 0:0 54000000 /32m_w2.data 2000000
>   33554432 bytes written
>   => fatload scsi 0:0 56000000 /32m_w2.data    
>   33554432 bytes read in 537 ms (59.6 MiB/s)
>   => cmp.b 54000000 56000000 2000000
>   Total of 33554432 byte(s) were the same
> 
> - I also confirmed that 32m.data and 32m_w.data are the same
>   on the host.
> 
> > Reverting this commit fixes this issue for me.
> 
> So, first let me ask some questions:
> - What is the size of your mmc memory?
> - How did you format it?
> - How many files are already there in the root directory?

Since I think there's some timezone mismatches here, can you please try
your test in a loop?  And this is likely showing up on something like
am335x_evm or dra7xx_evm.  Thanks!
AKASHI Takahiro March 18, 2019, 1:57 a.m. UTC | #6
On Sun, Mar 17, 2019 at 09:44:20PM -0400, Tom Rini wrote:
> On Mon, Mar 18, 2019 at 10:42:31AM +0900, Akashi, Takahiro wrote:
> > Hi Faiz,
> > 
> > On Tue, Mar 12, 2019 at 02:11:08PM +0530, Faiz Abbas wrote:
> > > Hi Akashi,
> > > 
> > > On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
> > > > From: AKASHI Takahiro <takahiro.akashi@linaro.org>
> > > > 
> > > > The current write implementation is quite simple: remove existing clusters
> > > > and then allocating new ones and filling them with data. This, inevitably,
> > > > enforces always writing from the beginning of a file.
> > > > 
> > > > As the first step to lift this restriction, fat_file_write() and
> > > > set_contents() are modified to accept an additional parameter, file offset
> > > > and further re-factored so that, in the next patch, all the necessary code
> > > > will be put into set_contents().
> > > > 
> > > > Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
> > > > ---
> > > 
> > > My fatwrite, fatload and compare tests are failing in MMC with this
> > > commit. This is what I see:
> > > 
> > > => fatwrite mmc 0 ${loadaddr} test 0x2000000
> > > 33554432 bytes written
> > > => fatload mmc 0 84000000 test
> > > 33554432 bytes read in 2149 ms (14.9 MiB/s)
> > > => cmp.b 82000000 84000000 0x2000000
> > > byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
> > > Total of 806912 byte(s) were the same
> > > =>
> > 
> > I tried, but could not reproduce this issue.
> > (v2019.04-rc2)
> > 
> > What I did was:
> > - On host, create a vfat file system and a random data file.
> >      dd of=vfat128M.img bs=1M count=128
> >      mkfs -t vfat vfat128M.img ; mount ...
> >      head -c 32m /dev/urandom > 32m.data
> > 
> > - On qemu, try fatwrite
> >   (try 1)
> >   => fatload scsi 0:0 50000000 /32m.data 2000000
> >   33554432 bytes read in 539 ms (59.4 MiB/s)
> >   => fatwrite scsi 0:0 50000000 /32m_w.data 2000000
> >   33554432 bytes written
> >   => fatls scsi 0:0 /
> >   33554432   32m.data
> >   33554432   32m_w.data
> > 
> >   2 file(s), 0 dir(s)
> > 
> >   => fatload scsi 0:0 52000000 /32m_w.data
> >   33554432 bytes read in 537 ms (59.6 MiB/s)
> >   => cmp.b 50000000 52000000 2000000
> >   Total of 33554432 byte(s) were the same
> > 
> >   (try 2)
> >   => fatwrite scsi 0:0 54000000 /32m_w2.data 2000000
> >   33554432 bytes written
> >   => fatload scsi 0:0 56000000 /32m_w2.data    
> >   33554432 bytes read in 537 ms (59.6 MiB/s)
> >   => cmp.b 54000000 56000000 2000000
> >   Total of 33554432 byte(s) were the same
> > 
> > - I also confirmed that 32m.data and 32m_w.data are the same
> >   on the host.
> > 
> > > Reverting this commit fixes this issue for me.
> > 
> > So, first let me ask some questions:
> > - What is the size of your mmc memory?
> > - How did you format it?
> > - How many files are already there in the root directory?
> 
> Since I think there's some timezone mismatches here, can you please try
> your test in a loop?

I'm afraid that I don't get your point. "in a loop"?

> And this is likely showing up on something like
> am335x_evm or dra7xx_evm.  Thanks!

Do you mean that this might have board dependency?

Thanks,
-Takahiro Akashi

> -- 
> Tom
Tom Rini March 18, 2019, 1:59 a.m. UTC | #7
On Mon, Mar 18, 2019 at 10:57:37AM +0900, Akashi, Takahiro wrote:
> On Sun, Mar 17, 2019 at 09:44:20PM -0400, Tom Rini wrote:
> > On Mon, Mar 18, 2019 at 10:42:31AM +0900, Akashi, Takahiro wrote:
> > > Hi Faiz,
> > > 
> > > On Tue, Mar 12, 2019 at 02:11:08PM +0530, Faiz Abbas wrote:
> > > > Hi Akashi,
> > > > 
> > > > On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
> > > > > From: AKASHI Takahiro <takahiro.akashi@linaro.org>
> > > > > 
> > > > > The current write implementation is quite simple: remove existing clusters
> > > > > and then allocating new ones and filling them with data. This, inevitably,
> > > > > enforces always writing from the beginning of a file.
> > > > > 
> > > > > As the first step to lift this restriction, fat_file_write() and
> > > > > set_contents() are modified to accept an additional parameter, file offset
> > > > > and further re-factored so that, in the next patch, all the necessary code
> > > > > will be put into set_contents().
> > > > > 
> > > > > Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
> > > > > ---
> > > > 
> > > > My fatwrite, fatload and compare tests are failing in MMC with this
> > > > commit. This is what I see:
> > > > 
> > > > => fatwrite mmc 0 ${loadaddr} test 0x2000000
> > > > 33554432 bytes written
> > > > => fatload mmc 0 84000000 test
> > > > 33554432 bytes read in 2149 ms (14.9 MiB/s)
> > > > => cmp.b 82000000 84000000 0x2000000
> > > > byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
> > > > Total of 806912 byte(s) were the same
> > > > =>
> > > 
> > > I tried, but could not reproduce this issue.
> > > (v2019.04-rc2)
> > > 
> > > What I did was:
> > > - On host, create a vfat file system and a random data file.
> > >      dd of=vfat128M.img bs=1M count=128
> > >      mkfs -t vfat vfat128M.img ; mount ...
> > >      head -c 32m /dev/urandom > 32m.data
> > > 
> > > - On qemu, try fatwrite
> > >   (try 1)
> > >   => fatload scsi 0:0 50000000 /32m.data 2000000
> > >   33554432 bytes read in 539 ms (59.4 MiB/s)
> > >   => fatwrite scsi 0:0 50000000 /32m_w.data 2000000
> > >   33554432 bytes written
> > >   => fatls scsi 0:0 /
> > >   33554432   32m.data
> > >   33554432   32m_w.data
> > > 
> > >   2 file(s), 0 dir(s)
> > > 
> > >   => fatload scsi 0:0 52000000 /32m_w.data
> > >   33554432 bytes read in 537 ms (59.6 MiB/s)
> > >   => cmp.b 50000000 52000000 2000000
> > >   Total of 33554432 byte(s) were the same
> > > 
> > >   (try 2)
> > >   => fatwrite scsi 0:0 54000000 /32m_w2.data 2000000
> > >   33554432 bytes written
> > >   => fatload scsi 0:0 56000000 /32m_w2.data    
> > >   33554432 bytes read in 537 ms (59.6 MiB/s)
> > >   => cmp.b 54000000 56000000 2000000
> > >   Total of 33554432 byte(s) were the same
> > > 
> > > - I also confirmed that 32m.data and 32m_w.data are the same
> > >   on the host.
> > > 
> > > > Reverting this commit fixes this issue for me.
> > > 
> > > So, first let me ask some questions:
> > > - What is the size of your mmc memory?
> > > - How did you format it?
> > > - How many files are already there in the root directory?
> > 
> > Since I think there's some timezone mismatches here, can you please try
> > your test in a loop?
> 
> I'm afraid that I don't get your point. "in a loop"?

Yes.  The process you describe above, put it in a script and let it run
over and over.

> > And this is likely showing up on something like
> > am335x_evm or dra7xx_evm.  Thanks!
> 
> Do you mean that this might have board dependency?

Well, you were asking about the size of the MMC memory, so those boards
might be a good place to look for clues that may differ from your setup.
Thanks!
Faiz Abbas March 21, 2019, 6:50 a.m. UTC | #8
Tom, Akashi,

On 18/03/19 7:29 AM, Tom Rini wrote:
> On Mon, Mar 18, 2019 at 10:57:37AM +0900, Akashi, Takahiro wrote:
>> On Sun, Mar 17, 2019 at 09:44:20PM -0400, Tom Rini wrote:
>>> On Mon, Mar 18, 2019 at 10:42:31AM +0900, Akashi, Takahiro wrote:
>>>> Hi Faiz,
>>>>
>>>> On Tue, Mar 12, 2019 at 02:11:08PM +0530, Faiz Abbas wrote:
>>>>> Hi Akashi,
>>>>>
>>>>> On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
>>>>>> From: AKASHI Takahiro <takahiro.akashi@linaro.org>
>>>>>>
>>>>>> The current write implementation is quite simple: remove existing clusters
>>>>>> and then allocating new ones and filling them with data. This, inevitably,
>>>>>> enforces always writing from the beginning of a file.
>>>>>>
>>>>>> As the first step to lift this restriction, fat_file_write() and
>>>>>> set_contents() are modified to accept an additional parameter, file offset
>>>>>> and further re-factored so that, in the next patch, all the necessary code
>>>>>> will be put into set_contents().
>>>>>>
>>>>>> Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
>>>>>> ---
>>>>>
>>>>> My fatwrite, fatload and compare tests are failing in MMC with this
>>>>> commit. This is what I see:
>>>>>
>>>>> => fatwrite mmc 0 ${loadaddr} test 0x2000000
>>>>> 33554432 bytes written
>>>>> => fatload mmc 0 84000000 test
>>>>> 33554432 bytes read in 2149 ms (14.9 MiB/s)
>>>>> => cmp.b 82000000 84000000 0x2000000
>>>>> byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
>>>>> Total of 806912 byte(s) were the same
>>>>> =>
>>>>
>>>> I tried, but could not reproduce this issue.
>>>> (v2019.04-rc2)
>>>>
>>>> What I did was:
>>>> - On host, create a vfat file system and a random data file.
>>>>      dd of=vfat128M.img bs=1M count=128
>>>>      mkfs -t vfat vfat128M.img ; mount ...
>>>>      head -c 32m /dev/urandom > 32m.data
>>>>
>>>> - On qemu, try fatwrite
>>>>   (try 1)
>>>>   => fatload scsi 0:0 50000000 /32m.data 2000000
>>>>   33554432 bytes read in 539 ms (59.4 MiB/s)
>>>>   => fatwrite scsi 0:0 50000000 /32m_w.data 2000000
>>>>   33554432 bytes written
>>>>   => fatls scsi 0:0 /
>>>>   33554432   32m.data
>>>>   33554432   32m_w.data
>>>>
>>>>   2 file(s), 0 dir(s)
>>>>
>>>>   => fatload scsi 0:0 52000000 /32m_w.data
>>>>   33554432 bytes read in 537 ms (59.6 MiB/s)
>>>>   => cmp.b 50000000 52000000 2000000
>>>>   Total of 33554432 byte(s) were the same
>>>>
>>>>   (try 2)
>>>>   => fatwrite scsi 0:0 54000000 /32m_w2.data 2000000
>>>>   33554432 bytes written
>>>>   => fatload scsi 0:0 56000000 /32m_w2.data    
>>>>   33554432 bytes read in 537 ms (59.6 MiB/s)
>>>>   => cmp.b 54000000 56000000 2000000
>>>>   Total of 33554432 byte(s) were the same
>>>>
>>>> - I also confirmed that 32m.data and 32m_w.data are the same
>>>>   on the host.
>>>>
>>>>> Reverting this commit fixes this issue for me.
>>>>
>>>> So, first let me ask some questions:
>>>> - What is the size of your mmc memory?
It doesn't seem to be related to MMC size. It killed most of our boards
and all of them have different sized SD cards. The fat partition would
be around 100 MB on each.

>>>> - How did you format it?

Using the default ubuntu disks app.

>>>> - How many files are already there in the root directory?

Only MLO and U-boot.img

>>>
>>> Since I think there's some timezone mismatches here, can you please try
>>> your test in a loop?
>>
>> I'm afraid that I don't get your point. "in a loop"?
> 
> Yes.  The process you describe above, put it in a script and let it run
> over and over.
> 
>>> And this is likely showing up on something like
>>> am335x_evm or dra7xx_evm.  Thanks!>>
>> Do you mean that this might have board dependency?
> 
> Well, you were asking about the size of the MMC memory, so those boards
> might be a good place to look for clues that may differ from your setup.
> Thanks!
>

Unfortunately, I am unable to reproduce it now either. I tried it on
rc4, went back to offending commit and couldn't reproduce it there
either. I tried with different boards and with reformatted partitions
but it seems to have vanished. I will try out the read and write in a
loop that Tom suggested.

It would be a good idea if you could get fat filesystem on an MMC device
to reproduce it. It could be related to MMC only.

Thanks,
Faiz
Faiz Abbas March 22, 2019, 9:13 a.m. UTC | #9
Hi,

On 21/03/19 12:20 PM, Faiz Abbas wrote:
> Tom, Akashi,
> 
> On 18/03/19 7:29 AM, Tom Rini wrote:
>> On Mon, Mar 18, 2019 at 10:57:37AM +0900, Akashi, Takahiro wrote:
>>> On Sun, Mar 17, 2019 at 09:44:20PM -0400, Tom Rini wrote:
>>>> On Mon, Mar 18, 2019 at 10:42:31AM +0900, Akashi, Takahiro wrote:
>>>>> Hi Faiz,
>>>>>
>>>>> On Tue, Mar 12, 2019 at 02:11:08PM +0530, Faiz Abbas wrote:
>>>>>> Hi Akashi,
>>>>>>
>>>>>> On 11/09/18 12:29 PM, Akashi, Takahiro wrote:
>>>>>>> From: AKASHI Takahiro <takahiro.akashi@linaro.org>
>>>>>>>
>>>>>>> The current write implementation is quite simple: remove existing clusters
>>>>>>> and then allocating new ones and filling them with data. This, inevitably,
>>>>>>> enforces always writing from the beginning of a file.
>>>>>>>
>>>>>>> As the first step to lift this restriction, fat_file_write() and
>>>>>>> set_contents() are modified to accept an additional parameter, file offset
>>>>>>> and further re-factored so that, in the next patch, all the necessary code
>>>>>>> will be put into set_contents().
>>>>>>>
>>>>>>> Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
>>>>>>> ---
>>>>>>
>>>>>> My fatwrite, fatload and compare tests are failing in MMC with this
>>>>>> commit. This is what I see:
>>>>>>
>>>>>> => fatwrite mmc 0 ${loadaddr} test 0x2000000
>>>>>> 33554432 bytes written
>>>>>> => fatload mmc 0 84000000 test
>>>>>> 33554432 bytes read in 2149 ms (14.9 MiB/s)
>>>>>> => cmp.b 82000000 84000000 0x2000000
>>>>>> byte at 0x820c5000 (0x85) != byte at 0x840c5000 (0x9d)
>>>>>> Total of 806912 byte(s) were the same
>>>>>> =>
>>>>>
>>>>> I tried, but could not reproduce this issue.
>>>>> (v2019.04-rc2)
>>>>>
>>>>> What I did was:
>>>>> - On host, create a vfat file system and a random data file.
>>>>>      dd of=vfat128M.img bs=1M count=128
>>>>>      mkfs -t vfat vfat128M.img ; mount ...
>>>>>      head -c 32m /dev/urandom > 32m.data
>>>>>
>>>>> - On qemu, try fatwrite
>>>>>   (try 1)
>>>>>   => fatload scsi 0:0 50000000 /32m.data 2000000
>>>>>   33554432 bytes read in 539 ms (59.4 MiB/s)
>>>>>   => fatwrite scsi 0:0 50000000 /32m_w.data 2000000
>>>>>   33554432 bytes written
>>>>>   => fatls scsi 0:0 /
>>>>>   33554432   32m.data
>>>>>   33554432   32m_w.data
>>>>>
>>>>>   2 file(s), 0 dir(s)
>>>>>
>>>>>   => fatload scsi 0:0 52000000 /32m_w.data
>>>>>   33554432 bytes read in 537 ms (59.6 MiB/s)
>>>>>   => cmp.b 50000000 52000000 2000000
>>>>>   Total of 33554432 byte(s) were the same
>>>>>
>>>>>   (try 2)
>>>>>   => fatwrite scsi 0:0 54000000 /32m_w2.data 2000000
>>>>>   33554432 bytes written
>>>>>   => fatload scsi 0:0 56000000 /32m_w2.data    
>>>>>   33554432 bytes read in 537 ms (59.6 MiB/s)
>>>>>   => cmp.b 54000000 56000000 2000000
>>>>>   Total of 33554432 byte(s) were the same
>>>>>
>>>>> - I also confirmed that 32m.data and 32m_w.data are the same
>>>>>   on the host.
>>>>>
>>>>>> Reverting this commit fixes this issue for me.
>>>>>
>>>>> So, first let me ask some questions:
>>>>> - What is the size of your mmc memory?
> It doesn't seem to be related to MMC size. It killed most of our boards
> and all of them have different sized SD cards. The fat partition would
> be around 100 MB on each.
> 
>>>>> - How did you format it?
> 
> Using the default ubuntu disks app.
> 
>>>>> - How many files are already there in the root directory?
> 
> Only MLO and U-boot.img
> 
>>>>
>>>> Since I think there's some timezone mismatches here, can you please try
>>>> your test in a loop?
>>>
>>> I'm afraid that I don't get your point. "in a loop"?
>>
>> Yes.  The process you describe above, put it in a script and let it run
>> over and over.
>>
>>>> And this is likely showing up on something like
>>>> am335x_evm or dra7xx_evm.  Thanks!>>
>>> Do you mean that this might have board dependency?
>>
>> Well, you were asking about the size of the MMC memory, so those boards
>> might be a good place to look for clues that may differ from your setup.
>> Thanks!
>>
> 
> Unfortunately, I am unable to reproduce it now either. I tried it on
> rc4, went back to offending commit and couldn't reproduce it there
> either. I tried with different boards and with reformatted partitions
> but it seems to have vanished. I will try out the read and write in a
> loop that Tom suggested.
> 
> It would be a good idea if you could get fat filesystem on an MMC device
> to reproduce it. It could be related to MMC only.
> 

Update on this:

I saw the issue in rc4 on a dra76x-evm one time but its really hard to
find out the exact steps and successive tries have failed to reproduce
it. I did setup an infinite while loop with tftp, write, read-back and
compare of boot images again and again, but that doesn't reproduce it
either after a few tens of tries. It seems some random writes and
read-backs should be able to reproduce it. Any ideas on a randomized
stress test?

Thanks,
Faiz
diff mbox series

Patch

diff --git a/fs/fat/fat_write.c b/fs/fat/fat_write.c
index 37ef0564eb5e..c22d8c7a46a1 100644
--- a/fs/fat/fat_write.c
+++ b/fs/fat/fat_write.c
@@ -528,6 +528,42 @@  static int clear_fatent(fsdata *mydata, __u32 entry)
 	return 0;
 }
 
+/*
+ * Set start cluster in directory entry
+ */
+static void set_start_cluster(const fsdata *mydata, dir_entry *dentptr,
+			      __u32 start_cluster)
+{
+	if (mydata->fatsize == 32)
+		dentptr->starthi =
+			cpu_to_le16((start_cluster & 0xffff0000) >> 16);
+	dentptr->start = cpu_to_le16(start_cluster & 0xffff);
+}
+
+/*
+ * Check whether adding a file makes the file system to
+ * exceed the size of the block device
+ * Return -1 when overflow occurs, otherwise return 0
+ */
+static int check_overflow(fsdata *mydata, __u32 clustnum, loff_t size)
+{
+	__u32 startsect, sect_num, offset;
+
+	if (clustnum > 0)
+		startsect = clust_to_sect(mydata, clustnum);
+	else
+		startsect = mydata->rootdir_sect;
+
+	sect_num = div_u64_rem(size, mydata->sect_size, &offset);
+
+	if (offset != 0)
+		sect_num++;
+
+	if (startsect + sect_num > total_sector)
+		return -1;
+	return 0;
+}
+
 /*
  * Write at most 'maxsize' bytes from 'buffer' into
  * the file associated with 'dentptr'
@@ -535,29 +571,36 @@  static int clear_fatent(fsdata *mydata, __u32 entry)
  * or return -1 on fatal errors.
  */
 static int
-set_contents(fsdata *mydata, dir_entry *dentptr, __u8 *buffer,
-	      loff_t maxsize, loff_t *gotsize)
+set_contents(fsdata *mydata, dir_entry *dentptr, loff_t pos, __u8 *buffer,
+	     loff_t maxsize, loff_t *gotsize)
 {
-	loff_t filesize = FAT2CPU32(dentptr->size);
+	loff_t filesize;
 	unsigned int bytesperclust = mydata->clust_size * mydata->sect_size;
 	__u32 curclust = START(dentptr);
 	__u32 endclust = 0, newclust = 0;
 	loff_t actsize;
 
 	*gotsize = 0;
-	debug("Filesize: %llu bytes\n", filesize);
-
-	if (maxsize > 0 && filesize > maxsize)
-		filesize = maxsize;
+	filesize = maxsize;
 
 	debug("%llu bytes\n", filesize);
 
-	if (!curclust) {
-		if (filesize) {
-			debug("error: nonempty clusterless file!\n");
+	if (curclust) {
+		/*
+		 * release already-allocated clusters anyway
+		 */
+		if (clear_fatent(mydata, curclust)) {
+			printf("Error: clearing FAT entries\n");
 			return -1;
 		}
-		return 0;
+	}
+
+	curclust = find_empty_cluster(mydata);
+	set_start_cluster(mydata, dentptr, curclust);
+
+	if (check_overflow(mydata, curclust, filesize)) {
+		printf("Error: no space left: %llu\n", filesize);
+		return -1;
 	}
 
 	actsize = bytesperclust;
@@ -568,6 +611,7 @@  set_contents(fsdata *mydata, dir_entry *dentptr, __u8 *buffer,
 			newclust = determine_fatent(mydata, endclust);
 
 			if ((newclust - 1) != endclust)
+				/* write to <curclust..endclust> */
 				goto getit;
 
 			if (CHECK_CLUST(newclust, mydata->fatsize)) {
@@ -614,18 +658,8 @@  getit:
 		actsize = bytesperclust;
 		curclust = endclust = newclust;
 	} while (1);
-}
 
-/*
- * Set start cluster in directory entry
- */
-static void set_start_cluster(const fsdata *mydata, dir_entry *dentptr,
-				__u32 start_cluster)
-{
-	if (mydata->fatsize == 32)
-		dentptr->starthi =
-			cpu_to_le16((start_cluster & 0xffff0000) >> 16);
-	dentptr->start = cpu_to_le16(start_cluster & 0xffff);
+	return 0;
 }
 
 /*
@@ -642,31 +676,6 @@  static void fill_dentry(fsdata *mydata, dir_entry *dentptr,
 	set_name(dentptr, filename);
 }
 
-/*
- * Check whether adding a file makes the file system to
- * exceed the size of the block device
- * Return -1 when overflow occurs, otherwise return 0
- */
-static int check_overflow(fsdata *mydata, __u32 clustnum, loff_t size)
-{
-	__u32 startsect, sect_num, offset;
-
-	if (clustnum > 0) {
-		startsect = clust_to_sect(mydata, clustnum);
-	} else {
-		startsect = mydata->rootdir_sect;
-	}
-
-	sect_num = div_u64_rem(size, mydata->sect_size, &offset);
-
-	if (offset != 0)
-		sect_num++;
-
-	if (startsect + sect_num > total_sector)
-		return -1;
-	return 0;
-}
-
 /*
  * Find a directory entry based on filename or start cluster number
  * If the directory entry is not found,
@@ -784,11 +793,10 @@  static int normalize_longname(char *l_filename, const char *filename)
 	return 0;
 }
 
-static int do_fat_write(const char *filename, void *buffer, loff_t size,
-			loff_t *actwrite)
+int file_fat_write_at(const char *filename, loff_t pos, void *buffer,
+		      loff_t size, loff_t *actwrite)
 {
 	dir_entry *retdent;
-	__u32 start_cluster;
 	fsdata datablock = { .fatbuf = NULL, };
 	fsdata *mydata = &datablock;
 	fat_itr *itr = NULL;
@@ -796,6 +804,8 @@  static int do_fat_write(const char *filename, void *buffer, loff_t size,
 	char *filename_copy, *parent, *basename;
 	char l_filename[VFAT_MAXLEN_BYTES];
 
+	debug("writing %s\n", filename);
+
 	filename_copy = strdup(filename);
 	if (!filename_copy)
 		return -ENOMEM;
@@ -839,47 +849,8 @@  static int do_fat_write(const char *filename, void *buffer, loff_t size,
 			goto exit;
 		}
 
-		/* Update file size and start_cluster in a directory entry */
-		retdent->size = cpu_to_le32(size);
-		start_cluster = START(retdent);
-
-		if (start_cluster) {
-			if (size) {
-				ret = check_overflow(mydata, start_cluster,
-							size);
-				if (ret) {
-					printf("Error: %llu overflow\n", size);
-					ret = -ENOSPC;
-					goto exit;
-				}
-			}
-
-			ret = clear_fatent(mydata, start_cluster);
-			if (ret) {
-				printf("Error: clearing FAT entries\n");
-				ret = -EIO;
-				goto exit;
-			}
-
-			if (!size)
-				set_start_cluster(mydata, retdent, 0);
-		} else if (size) {
-			ret = start_cluster = find_empty_cluster(mydata);
-			if (ret < 0) {
-				printf("Error: finding empty cluster\n");
-				ret = -ENOSPC;
-				goto exit;
-			}
-
-			ret = check_overflow(mydata, start_cluster, size);
-			if (ret) {
-				printf("Error: %llu overflow\n", size);
-				ret = -ENOSPC;
-				goto exit;
-			}
-
-			set_start_cluster(mydata, retdent, start_cluster);
-		}
+		/* Update file size in a directory entry */
+		retdent->size = cpu_to_le32(pos + size);
 	} else {
 		/* Create a new file */
 
@@ -907,32 +878,13 @@  static int do_fat_write(const char *filename, void *buffer, loff_t size,
 			goto exit;
 		}
 
-		if (size) {
-			ret = start_cluster = find_empty_cluster(mydata);
-			if (ret < 0) {
-				printf("Error: finding empty cluster\n");
-				ret = -ENOSPC;
-				goto exit;
-			}
-
-			ret = check_overflow(mydata, start_cluster, size);
-			if (ret) {
-				printf("Error: %llu overflow\n", size);
-				ret = -ENOSPC;
-				goto exit;
-			}
-		} else {
-			start_cluster = 0;
-		}
-
 		/* Set attribute as archive for regular file */
-		fill_dentry(itr->fsdata, itr->dent, filename,
-			    start_cluster, size, 0x20);
+		fill_dentry(itr->fsdata, itr->dent, filename, 0, size, 0x20);
 
 		retdent = itr->dent;
 	}
 
-	ret = set_contents(mydata, retdent, buffer, size, actwrite);
+	ret = set_contents(mydata, retdent, pos, buffer, size, actwrite);
 	if (ret < 0) {
 		printf("Error: writing contents\n");
 		ret = -EIO;
@@ -971,6 +923,5 @@  int file_fat_write(const char *filename, void *buffer, loff_t offset,
 		return -EINVAL;
 	}
 
-	printf("writing %s\n", filename);
-	return do_fat_write(filename, buffer, maxsize, actwrite);
+	return file_fat_write_at(filename, offset, buffer, maxsize, actwrite);
 }