From patchwork Tue Feb 15 23:17:46 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiyuki Okajima X-Patchwork-Id: 83322 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 26DA3B7116 for ; Wed, 16 Feb 2011 12:39:42 +1100 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757081Ab1BPAUO (ORCPT ); Tue, 15 Feb 2011 19:20:14 -0500 Received: from fgwmail5.fujitsu.co.jp ([192.51.44.35]:41128 "EHLO fgwmail5.fujitsu.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757062Ab1BPAUL (ORCPT ); Tue, 15 Feb 2011 19:20:11 -0500 Received: from m3.gw.fujitsu.co.jp (unknown [10.0.50.73]) by fgwmail5.fujitsu.co.jp (Postfix) with ESMTP id 149563EE0BC; Wed, 16 Feb 2011 09:20:09 +0900 (JST) Received: from smail (m3 [127.0.0.1]) by outgoing.m3.gw.fujitsu.co.jp (Postfix) with ESMTP id ECE5345DE57; Wed, 16 Feb 2011 09:20:08 +0900 (JST) Received: from s3.gw.fujitsu.co.jp (s3.gw.fujitsu.co.jp [10.0.50.93]) by m3.gw.fujitsu.co.jp (Postfix) with ESMTP id C967645DE59; Wed, 16 Feb 2011 09:20:08 +0900 (JST) Received: from s3.gw.fujitsu.co.jp (localhost.localdomain [127.0.0.1]) by s3.gw.fujitsu.co.jp (Postfix) with ESMTP id ABC7EE08003; Wed, 16 Feb 2011 09:20:08 +0900 (JST) Received: from m106.s.css.fujitsu.com (m106.s.css.fujitsu.com [10.249.87.106]) by s3.gw.fujitsu.co.jp (Postfix) with ESMTP id 43E28E18004; Wed, 16 Feb 2011 09:20:08 +0900 (JST) Received: from m106.css.fujitsu.com (m106 [127.0.0.1]) by m106.s.css.fujitsu.com (Postfix) with ESMTP id 017B45B90F6; Wed, 16 Feb 2011 09:20:08 +0900 (JST) Received: from stratos (stratos.soft.fujitsu.com [10.124.101.28]) by m106.s.css.fujitsu.com (Postfix) with SMTP id 1B3F05B90E8; Wed, 16 Feb 2011 09:20:07 +0900 (JST) Date: Wed, 16 Feb 2011 08:17:46 +0900 From: Toshiyuki Okajima To: Jan Kara Cc: Ted Ts'o , Masayoshi MIZUMA , Andreas Dilger , linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: [BUG] ext4: cannot unfreeze a filesystem due to a deadlock Message-Id: <20110216081746.54d146d1.toshi.okajima@jp.fujitsu.com> In-Reply-To: <20110215172954.GK17313@quack.suse.cz> References: <20110207205325.FB6A.61FB500B@jp.fujitsu.com> <20110215160630.GH17313@quack.suse.cz> <20110215170352.GE4255@thunk.org> <20110215172954.GK17313@quack.suse.cz> Organization: Fujitsu co.,ltd. X-Mailer: Sylpheed 2.7.1 (GTK+ 2.12.8; i686-pc-linux-gnu) Mime-Version: 1.0 Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Hi. On Tue, 15 Feb 2011 18:29:54 +0100 Jan Kara wrote: > On Tue 15-02-11 12:03:52, Ted Ts'o wrote: > > On Tue, Feb 15, 2011 at 05:06:30PM +0100, Jan Kara wrote: > > > Thanks for detailed analysis. Indeed this is a bug. Whenever we do IO > > > under s_umount semaphore, we are prone to deadlock like the one you > > > describe above. > > > > One of the fundamental problems here is that the freeze and thaw > > routines are using down_write(&sb->s_umount) for two purposes. The > > first is to prevent the resume/thaw from racing with a umount (which > > it could do just as well by taking a read lock), but the second is to > > prevent the resume/thaw code from racing with itself. That's the core > > fundamental problem here. > > > > So I think we can solve this by introduce a new mutex, s_freeze, and > > having the the resume/thaw first take the s_freeze mutex and then > > second take a read lock on the s_umount. > Sadly this does not quite work because even down_read(&sb->s_umount) > in thaw_super() can block if there is another process that tries to acquire > s_umount for writing - a situation like: > TASK 1 (e.g. flusher) TASK 2 (e.g. remount) TASK 3 (unfreeze) > down_read(&sb->s_umount) > block on s_frozen > down_write(&sb->s_umount) > -blocked > down_read(&sb->s_umount) > -blocked > behind the write access... > > The only working solution I see is to check for frozen filesystem before > taking s_umount semaphore which seems rather ugly (but might be bearable if > we did so in some well described wrapper). I created the patch that you imagine yesterday. I got a reproducer from Mizuma-san yesterday, and then I executed it on the kernel without a fixed patch. After an hour, I confirmed that this deadlock happened. However, on the kernel with a fixed patch, this deadlock doesn't still happen after 12 hours passed. The patch for linux-2.6.38-rc4 is as follows: --- fs/fs-writeback.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 59c6e49..1c9a05e 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -456,7 +456,7 @@ static bool pin_sb_for_writeback(struct super_block *sb) spin_unlock(&sb_lock); if (down_read_trylock(&sb->s_umount)) { - if (sb->s_root) + if (sb->s_frozen == SB_UNFROZEN && sb->s_root) return true; up_read(&sb->s_umount); }