From patchwork Tue Apr 26 17:20:57 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Theodore Ts'o X-Patchwork-Id: 92943 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 7EBABB6F1B for ; Wed, 27 Apr 2011 03:21:04 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754387Ab1DZRVC (ORCPT ); Tue, 26 Apr 2011 13:21:02 -0400 Received: from li9-11.members.linode.com ([67.18.176.11]:38240 "EHLO test.thunk.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751544Ab1DZRVB (ORCPT ); Tue, 26 Apr 2011 13:21:01 -0400 Received: from root (helo=tytso-glaptop) by test.thunk.org with local-esmtp (Exim 4.69) (envelope-from ) id 1QElwZ-0000oZ-IE; Tue, 26 Apr 2011 17:20:59 +0000 Received: from tytso by tytso-glaptop with local (Exim 4.71) (envelope-from ) id 1QElwY-0007h3-28; Tue, 26 Apr 2011 13:20:58 -0400 Date: Tue, 26 Apr 2011 13:20:57 -0400 From: Ted Ts'o To: Martin_Zielinski@mcafee.com Cc: linux-ext4@vger.kernel.org Subject: Re: 2.6.32 ext3 assertion j_running_transaction != NULL fails in commit.c Message-ID: <20110426172057.GH9486@thunk.org> References: <20110425231454.GB9486@thunk.org> <20110426122338.GE9486@thunk.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: tytso@thunk.org X-SA-Exim-Scanned: No (on test.thunk.org); SAEximRunCond expanded to false Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org On Tue, Apr 26, 2011 at 07:45:33AM -0500, Martin_Zielinski@mcafee.com wrote: > I will port the jbd2 debugging code to jbd an will try to get the > new kernel into production. After a reboot we will have to wait > several weeks. (Strange: all machines failed within 72 hours). Great, thanks. > With sqlite I can currently produce ~10.000.000 commits in one hour > with a program that does nothing else. I doubt that it is possible > to have an overflow in such a short time that we are observing. > Maybe the __log_start_commit commit call comes with a corrupt target > id from elsewhere. But your patch will catch that, too. Agreed; that's why I don't really believe the wraparound theory. For your convenience, this is the revised (cleaned up) patch for the ext3/jbd (it just cleans up how we print the warning). - Ted commit 4ea00445c7f5d3dfa6219262598a2a8319df07c7 Author: Theodore Ts'o Date: Tue Apr 26 13:14:55 2011 -0400 jbd: fix fsync() tid wraparound bug If an application program does not make any changes to the indirect blocks or extent tree, i_datasync_tid will not get updated. If there are enough commits (i.e., 2**31) such that tid_geq()'s calculations wrap, and there isn't a currently active transaction at the time of the fdatasync() call, this can end up triggering a BUG_ON in fs/jbd/commit.c: J_ASSERT(journal->j_running_transaction != NULL); It's pretty rare that this can happen, since it requires the use of fdatasync() plus *very* frequent and excessive use of fsync(). But with the right workload, it can. We fix this by replacing the use of tid_geq() with an equality test, since there's only one valid transaction id that we is valid for us to wait until it is commited: namely, the currently running transaction (if it exists). Signed-off-by: "Theodore Ts'o" --- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/fs/jbd/journal.c b/fs/jbd/journal.c index b3713af..1b71ce6 100644 --- a/fs/jbd/journal.c +++ b/fs/jbd/journal.c @@ -437,9 +437,12 @@ int __log_space_left(journal_t *journal) int __log_start_commit(journal_t *journal, tid_t target) { /* - * Are we already doing a recent enough commit? + * The only transaction we can possibly wait upon is the + * currently running transaction (if it exists). Otherwise, + * the target tid must be an old one. */ - if (!tid_geq(journal->j_commit_request, target)) { + if (journal->j_running_transaction && + journal->j_running_transaction->t_tid == target) { /* * We want a new commit: OK, mark the request and wakeup the * commit thread. We do _not_ do the commit ourselves. @@ -451,7 +454,14 @@ int __log_start_commit(journal_t *journal, tid_t target) journal->j_commit_sequence); wake_up(&journal->j_wait_commit); return 1; - } + } else if (!tid_geq(journal->j_commit_request, target)) + /* This should never happen, but if it does, preserve + the evidence before kjournald goes into a loop and + increments j_commit_sequence beyond all recognition. */ + WARN(1, "jbd: bad log_start_commit: %u %u %u %u\n", + journal->j_commit_request, journal->j_commit_sequence, + target, journal->j_running_transaction ? + journal->j_running_transaction->t_tid : 0); return 0; }