From patchwork Tue Apr 24 06:25:44 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "jeff.liu" X-Patchwork-Id: 154686 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 74048B6FD3 for ; Wed, 25 Apr 2012 00:28:53 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754777Ab2DXO2u (ORCPT ); Tue, 24 Apr 2012 10:28:50 -0400 Received: from rcsinet15.oracle.com ([148.87.113.117]:47227 "EHLO rcsinet15.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754768Ab2DXO2t (ORCPT ); Tue, 24 Apr 2012 10:28:49 -0400 Received: from ucsinet21.oracle.com (ucsinet21.oracle.com [156.151.31.93]) by rcsinet15.oracle.com (Sentrion-MTA-4.2.2/Sentrion-MTA-4.2.2) with ESMTP id q3OERpiE009070 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 24 Apr 2012 14:27:52 GMT Received: from acsmt357.oracle.com (acsmt357.oracle.com [141.146.40.157]) by ucsinet21.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id q3OERp3D026865 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 24 Apr 2012 14:27:51 GMT Received: from abhmt113.oracle.com (abhmt113.oracle.com [141.146.116.65]) by acsmt357.oracle.com (8.12.11.20060308/8.12.11) with ESMTP id q3OERo0d018651; Tue, 24 Apr 2012 09:27:51 -0500 Received: from [192.168.1.104] (/123.119.102.211) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 24 Apr 2012 07:27:50 -0700 Message-ID: <4F964768.3080302@oracle.com> Date: Tue, 24 Apr 2012 14:25:44 +0800 From: Jeff Liu Reply-To: jeff.liu@oracle.com Organization: Oracle User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.28) Gecko/20120313 Thunderbird/3.1.20 MIME-Version: 1.0 To: "linux-fsdevel@vger.kernel.org" CC: linux-ext4@vger.kernel.org, Jan Kara , tytso@mit.edu Subject: [PATCH] quota: fix a potential dead lock at add_dquot_ref() when performing quotaon against ext4 with files was writing X-Source-IP: ucsinet21.oracle.com [156.151.31.93] Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Hello, I just ran into an issue on vanilla-kernel-3.3.0 when executing quotaon(8) on ext4 with "usrquota,grpquota", and there have files are being writing at the same time. I have a lxc guest running on an ext4 partition, $ mount|grep sda6 /dev/sda6 on /ext4 type ext4 (rw,usrquota,grpquota) Execute quotacheck -cvumg /ext4 to force the quota checking without shutdown that guest, at this stage, everything is fine, # quotacheck -cvgum /ext4 quotacheck: Your kernel probably supports journaled quota but you are not using it. Consider switching to journaled quota to avoid running quotacheck after an unclean shutdown. quotacheck: Scanning /dev/sda6 [/ext4] done quotacheck: Cannot stat old user quota file /ext4/aquota.user: No such file or directory. Usage will not be subtracted. quotacheck: Cannot stat old group quota file /ext4/aquota.group: No such file or directory. Usage will not be subtracted. quotacheck: Cannot stat old user quota file /ext4/aquota.user: No such file or directory. Usage will not be subtracted. quotacheck: Cannot stat old group quota file /ext4/aquota.group: No such file or directory. Usage will not be subtracted. quotacheck: Checked 3357 directories and 39335 files quotacheck: Old file not found. quotacheck: Old file not found. However, the kernel was hang when running quotaon /ext4. I observed the following info via netconsole: [ 423.140177] *** DEADLOCK *** [ 423.140177] [ 423.140177] May be due to missing lock nesting notation [ 423.140177] [ 423.140177] 4 locks held by quotaon/2350: [ 423.140177] #0: (&type->s_umount_key#26){++++++}, at: [] get_super+0xf9/0x1ec [ 423.140177] #1: (&s->s_dquot.dqonoff_mutex){+.+...}, at: [] vfs_load_quota_inode+0x3e5/0xac6 [ 423.140177] #2: (inode_sb_list_lock){+.+...}, at: [] vfs_load_quota_inode+0x64a/0xac6 [ 423.140177] #3: (&sb->s_type->i_lock_key#16){+.+...}, at: [] vfs_load_quota_inode+0x68d/0xac6 [ 423.140177] [ 423.140177] stack backtrace: [ 423.140177] Pid: 2350, comm: quotaon Not tainted 3.3.0 #79 [ 423.140177] Call Trace: [ 423.140177] [] ? printk+0x57/0x6a [ 423.140177] [] __lock_acquire+0x133d/0x1a8a [ 423.140177] [] ? vprintk+0x910/0x93a [ 423.140177] [] ? inode_get_rsv_space+0x45/0x8b [ 423.140177] [] lock_acquire+0x13a/0x176 [ 423.140177] [] ? inode_get_rsv_space+0x45/0x8b [ 423.140177] [] _raw_spin_lock+0x54/0x7d [ 423.140177] [] ? inode_get_rsv_space+0x45/0x8b [ 423.140177] [] inode_get_rsv_space+0x45/0x8b [ 423.140177] [] vfs_load_quota_inode+0x891/0xac6 [ 423.140177] [] dquot_quota_on+0x82/0x97 [ 423.140177] [] ext4_quota_on+0x191/0x219 [ext4] [ 423.140177] [] ? ns_capable+0x71/0xa3 [ 423.140177] [] do_quotactl+0x2f7/0x80f [ 423.140177] [] ? ext4_msg+0x61/0x61 [ext4] [ 423.140177] [] ? get_super+0xf9/0x1ec [ 423.140177] [] ? get_super_thawed+0x33/0x147 [ 423.140177] [] ? iput+0x66/0x320 [ 423.140177] [] sys_quotactl+0x290/0x2fc [ 423.140177] [] syscall_call+0x7/0xb As per my investigation, it occurred due to inode_get_rsv_space(inode) lock acquiring if the kernel was built with QUOTA_DEBUG enabled. At add_dquot_ref(), the lock did not released if the inode.i_writecount is not *ZERO*, inode is not in I_FREEING|I_WILL_FREE|I_NEW state, as well as it need to do quota init. if ((inode->i_state & (I_FREEING|I_WILL_FREE|I_NEW)) || !atomic_read(&inode->i_writecount) || !dqinit_needed(inode, type)) { spin_unlock(&inode->i_lock); continue; } #ifdef CONFIG_QUOTA_DEBUG if (unlikely(inode_get_rsv_space(inode) > 0)) reserved = 1; #endif In my situation, atomic_read(&inode->i_writecount) returns -2, looks that the related file have two deny-writers via mmap at that time. To nail down this issue, I refresh formated an ext4 file system, and try to open a file and keep writing to it(so that the atomic_read(&inode->i_writecount) = 1). then perform quotacheck and quotaon at the same time, and repeat this process for 4 times, the kernel hang for 3 times, 1 time its ok. # python -c "f=open('/ext4/test', 'w'); [(f.seek(x) or f.write(str(x))) for x in range(1, 1000000000, 99)]; f.close()" # quotacheck -cvumg /ext4 quotacheck: Your kernel probably supports journaled quota but you are not using it. Consider switching to journaled quota to avoid running quotacheck after an unclean shutdown. quotacheck: Scanning /dev/sda6 [/ext4] done quotacheck: Cannot stat old user quota file /ext4/aquota.user: No such file or directory. Usage will not be subtracted. quotacheck: Cannot stat old group quota file /ext4/aquota.group: No such file or directory. Usage will not be subtracted. quotacheck: Cannot stat old user quota file /ext4/aquota.user: No such file or directory. Usage will not be subtracted. quotacheck: Cannot stat old group quota file /ext4/aquota.group: No such file or directory. Usage will not be subtracted. quotacheck: Checked 2 directories and 1 files quotacheck: Old file not found. quotacheck: Old file not found. # quotaon /ext4 /* kernel was hang. */ I also verified that this issue can be reproduced against the latest kernel commit(Mon Apr 23 19:52:00 2012 95f714727436836bb46236ce2bcd8ee8f9274aed). Below is a small patch, it looks a bit ugly, but works for me. Thanks, -Jeff Signed-off-by: Jie Liu --- fs/quota/dquot.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/fs/quota/dquot.c b/fs/quota/dquot.c index 4674197..0ae7fc3 100644 --- a/fs/quota/dquot.c +++ b/fs/quota/dquot.c @@ -907,8 +907,10 @@ static void add_dquot_ref(struct super_block *sb, int type) continue; } #ifdef CONFIG_QUOTA_DEBUG + spin_unlock(&inode->i_lock); if (unlikely(inode_get_rsv_space(inode) > 0)) reserved = 1; + spin_lock(&inode->i_lock); #endif __iget(inode); spin_unlock(&inode->i_lock);