diff mbox series

[v2,1/2] quota: flush quota_release_work upon quota writeback

Message ID 20241121123855.645335-2-ojaswin@linux.ibm.com
State New
Headers show
Series Fix generic/390 failure due to quota release after freeze | expand

Commit Message

Ojaswin Mujoo Nov. 21, 2024, 12:38 p.m. UTC 
One of the paths quota writeback is called from is:

freeze_super()
  sync_filesystem()
    ext4_sync_fs()
      dquot_writeback_dquots()

Since we currently don't always flush the quota_release_work queue in
this path, we can end up with the following race:

 1. dquot are added to releasing_dquots list during regular operations.
 2. FS Freeze starts, however, this does not flush the quota_release_work queue.
 3. Freeze completes.
 4. Kernel eventually tries to flush the workqueue while FS is frozen which
    hits a WARN_ON since transaction gets started during frozen state:

  ext4_journal_check_start+0x28/0x110 [ext4] (unreliable)
  __ext4_journal_start_sb+0x64/0x1c0 [ext4]
  ext4_release_dquot+0x90/0x1d0 [ext4]
  quota_release_workfn+0x43c/0x4d0

Which is the following line:

  WARN_ON(sb->s_writers.frozen == SB_FREEZE_COMPLETE);

Which ultimately results in generic/390 failing due to dmesg
noise. This was detected on powerpc machine 15 cores.

To avoid this, make sure to flush the workqueue during
dquot_writeback_dquots() so we dont have any pending workitems after
freeze.

Reported-by: Disha Goel <disgoel@linux.ibm.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
 fs/quota/dquot.c | 2 ++
 1 file changed, 2 insertions(+)

Comments

Jan Kara Nov. 26, 2024, 8:47 a.m. UTC | #1 
On Thu 21-11-24 18:08:54, Ojaswin Mujoo wrote:
> One of the paths quota writeback is called from is:
> 
> freeze_super()
>   sync_filesystem()
>     ext4_sync_fs()
>       dquot_writeback_dquots()
> 
> Since we currently don't always flush the quota_release_work queue in
> this path, we can end up with the following race:
> 
>  1. dquot are added to releasing_dquots list during regular operations.
>  2. FS Freeze starts, however, this does not flush the quota_release_work queue.
>  3. Freeze completes.
>  4. Kernel eventually tries to flush the workqueue while FS is frozen which
>     hits a WARN_ON since transaction gets started during frozen state:
> 
>   ext4_journal_check_start+0x28/0x110 [ext4] (unreliable)
>   __ext4_journal_start_sb+0x64/0x1c0 [ext4]
>   ext4_release_dquot+0x90/0x1d0 [ext4]
>   quota_release_workfn+0x43c/0x4d0
> 
> Which is the following line:
> 
>   WARN_ON(sb->s_writers.frozen == SB_FREEZE_COMPLETE);
> 
> Which ultimately results in generic/390 failing due to dmesg
> noise. This was detected on powerpc machine 15 cores.
> 
> To avoid this, make sure to flush the workqueue during
> dquot_writeback_dquots() so we dont have any pending workitems after
> freeze.
> 
> Reported-by: Disha Goel <disgoel@linux.ibm.com>
> Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>

Thanks. Since this patch is independent, I've picked it up into my tree
(will push it to Linus for rc2).

								Honza
Baokun Li Nov. 26, 2024, 2:28 p.m. UTC | #2 
On 2024/11/21 20:38, Ojaswin Mujoo wrote:
> One of the paths quota writeback is called from is:
>
> freeze_super()
>    sync_filesystem()
>      ext4_sync_fs()
>        dquot_writeback_dquots()
>
> Since we currently don't always flush the quota_release_work queue in
> this path, we can end up with the following race:
>
>   1. dquot are added to releasing_dquots list during regular operations.
>   2. FS Freeze starts, however, this does not flush the quota_release_work queue.
>   3. Freeze completes.
>   4. Kernel eventually tries to flush the workqueue while FS is frozen which
>      hits a WARN_ON since transaction gets started during frozen state:
>
>    ext4_journal_check_start+0x28/0x110 [ext4] (unreliable)
>    __ext4_journal_start_sb+0x64/0x1c0 [ext4]
>    ext4_release_dquot+0x90/0x1d0 [ext4]
>    quota_release_workfn+0x43c/0x4d0
>
> Which is the following line:
>
>    WARN_ON(sb->s_writers.frozen == SB_FREEZE_COMPLETE);
>
> Which ultimately results in generic/390 failing due to dmesg
> noise. This was detected on powerpc machine 15 cores.
>
> To avoid this, make sure to flush the workqueue during
> dquot_writeback_dquots() so we dont have any pending workitems after
> freeze.
>
> Reported-by: Disha Goel <disgoel@linux.ibm.com>
> Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Looks good, thanks for the patch!

Reviewed-by: Baokun Li <libaokun1@huawei.com>
> ---
>   fs/quota/dquot.c | 2 ++
>   1 file changed, 2 insertions(+)
>
> diff --git a/fs/quota/dquot.c b/fs/quota/dquot.c
> index 3dd8d6f27725..f9578918cfb2 100644
> --- a/fs/quota/dquot.c
> +++ b/fs/quota/dquot.c
> @@ -688,6 +688,8 @@ int dquot_writeback_dquots(struct super_block *sb, int type)
>   
>   	WARN_ON_ONCE(!rwsem_is_locked(&sb->s_umount));
>   
> +	flush_delayed_work(&quota_release_work);
> +
>   	for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
>   		if (type != -1 && cnt != type)
>   			continue;
Disha Goel Nov. 29, 2024, 8 a.m. UTC | #3 
On 21/11/24 6:08 pm, Ojaswin Mujoo wrote:

One of the paths quota writeback is called from is:

freeze_super()
   sync_filesystem()
     ext4_sync_fs()
       dquot_writeback_dquots()

Since we currently don't always flush the quota_release_work queue in
this path, we can end up with the following race:

  1. dquot are added to releasing_dquots list during regular operations.
  2. FS Freeze starts, however, this does not flush the quota_release_work queue.
  3. Freeze completes.
  4. Kernel eventually tries to flush the workqueue while FS is frozen which
     hits a WARN_ON since transaction gets started during frozen state:

   ext4_journal_check_start+0x28/0x110 [ext4] (unreliable)
   __ext4_journal_start_sb+0x64/0x1c0 [ext4]
   ext4_release_dquot+0x90/0x1d0 [ext4]
   quota_release_workfn+0x43c/0x4d0

Which is the following line:

   WARN_ON(sb->s_writers.frozen == SB_FREEZE_COMPLETE);

Which ultimately results in generic/390 failing due to dmesg
noise. This was detected on powerpc machine 15 cores.

To avoid this, make sure to flush the workqueue during
dquot_writeback_dquots() so we dont have any pending workitems after
freeze.

Reported-by: Disha Goel <disgoel@linux.ibm.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>

Thanks for the fix patch Ojaswin.
I have tested the patch on powerpc machine, and it fixes the generic/390 test failure.

Tested-by: Disha Goel <disgoel@linux.ibm.com>

---
  fs/quota/dquot.c | 2 ++
  1 file changed, 2 insertions(+)

diff --git a/fs/quota/dquot.c b/fs/quota/dquot.c
index 3dd8d6f27725..f9578918cfb2 100644
--- a/fs/quota/dquot.c
+++ b/fs/quota/dquot.c
@@ -688,6 +688,8 @@ int dquot_writeback_dquots(struct super_block *sb, int type)
  
  	WARN_ON_ONCE(!rwsem_is_locked(&sb->s_umount));
  
+	flush_delayed_work(&quota_release_work);
+
  	for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
  		if (type != -1 && cnt != type)
  			continue;
diff mbox series

Patch

diff --git a/fs/quota/dquot.c b/fs/quota/dquot.c
index 3dd8d6f27725..f9578918cfb2 100644
--- a/fs/quota/dquot.c
+++ b/fs/quota/dquot.c
@@ -688,6 +688,8 @@  int dquot_writeback_dquots(struct super_block *sb, int type)
 
 	WARN_ON_ONCE(!rwsem_is_locked(&sb->s_umount));
 
+	flush_delayed_work(&quota_release_work);
+
 	for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
 		if (type != -1 && cnt != type)
 			continue;