From patchwork Fri Nov 20 19:15:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404061 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=tTz0Crpw; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5r86sCDz9sVH for ; Sat, 21 Nov 2020 06:16:24 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732114AbgKTTQY (ORCPT ); Fri, 20 Nov 2020 14:16:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729453AbgKTTQX (ORCPT ); Fri, 20 Nov 2020 14:16:23 -0500 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6B58C061A04 for ; Fri, 20 Nov 2020 11:16:23 -0800 (PST) Received: by mail-pl1-x643.google.com with SMTP id 5so5346464plj.8 for ; Fri, 20 Nov 2020 11:16:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v8z1HukCSndbU15jqdnFsZ8EBy8uLWxjfAHj0nxUilQ=; b=tTz0CrpwHSGUlwteNQHhk1wnVuKiqaMeLgg1knsVembHaQ5/bmP3a2cz5kCp0g14sW KZpWPVhkeAqHh1VsgzRjlvWXWmIBQjGB/a3czrGhGzugGquhallyhYUOdr/w5JM0zuGo j6cAcMTGvlI7Sx3bJgFj2R8fPqetNutz7UTwu5PMTrjuX3c7DnXYplhVGIlPhTBYn4OQ 2jl3pQCXHIwHPslkL1Fn45kjaeLQMPVdGHGSuXuUDlF0lVAbkkvPrH4Ib0koWem39wMC J/8E0qhTvhSqkgEykPHHRgtmXdgMsfyJI2DS5Y2Mhjp6bHT2u4eKKeK4ZTJNUTnmxxf4 ucdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v8z1HukCSndbU15jqdnFsZ8EBy8uLWxjfAHj0nxUilQ=; b=VfTAewAqo5rrlBAKmxGIVq9l1sNixUh02Ilv70l2Bujxv2iY3OgyXyjDDGRKppfdM/ 0RFtcpO3M+JaiguutkM9WchfQjd1sLeLcKiVLiQ92w4QqS/s8JxFuxuzdG5e22hSsjAF m78KKdDGC3F8B7poSIIh+jT6yt5pLpWM4cntjEPNX1KZGDkKATknkpF8NBUtbWRI67Ab m1veXnuH5HYwjmNjtpmMT/vBKk3ys86RmXG7qYeJ6K34Oe1AEAfSEPFV3noUyYBp95Ru hN8E0fKC7+rQNyBGxaoeR/gdameZIYweJWWDq6rzB7uHLL/xX3nXiMZmMEjsVs7aCqtb 8sng== X-Gm-Message-State: AOAM532M4nmzV3hXMBmefN5pxxNrvsX7l+Seee3TMqhqCRzd/TbQ4Mwb HVKDVAtJJpg/WyTIkw8EaNYV1XueM+I= X-Google-Smtp-Source: ABdhPJw6B7nEvc+zPDPZYeKTP/xlNH3zxJ0YA2+1/dHCDdo6qay3lxAxH3Rygk5VXYfc5yUZPj5W/w== X-Received: by 2002:a17:902:b196:b029:d5:a8fd:9a1c with SMTP id s22-20020a170902b196b02900d5a8fd9a1cmr15165459plr.44.1605899782590; Fri, 20 Nov 2020 11:16:22 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:21 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 01/15] ext2fs: move calculate_summary_stats to ext2fs lib Date: Fri, 20 Nov 2020 11:15:52 -0800 Message-Id: <20201120191606.2224881-2-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org The function calculate_summary_stats sets the global metadata of the file system. Tune2fs had this function defined statically in tune2fs.c. Fast commit replay needs this function to set global metadata at the end of the replay phase. So, move this function to libext2fs. Signed-off-by: Harshad Shirwadkar Reviewed-by: Theodore Ts'o --- lib/ext2fs/ext2fs.h | 1 + lib/ext2fs/initialize.c | 94 +++++++++++++++++++++++++++++++++++++++++ misc/tune2fs.c | 59 +------------------------- resize/resize2fs.c | 6 +-- 4 files changed, 99 insertions(+), 61 deletions(-) diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 69c8a3ff..a8a6e091 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -1484,6 +1484,7 @@ errcode_t ext2fs_write_ind_block(ext2_filsys fs, blk_t blk, void *buf); extern errcode_t ext2fs_initialize(const char *name, int flags, struct ext2_super_block *param, io_manager manager, ext2_filsys *ret_fs); +extern errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs, int super_only); /* icount.c */ extern void ext2fs_free_icount(ext2_icount_t icount); diff --git a/lib/ext2fs/initialize.c b/lib/ext2fs/initialize.c index 96ec1cff..edd692bb 100644 --- a/lib/ext2fs/initialize.c +++ b/lib/ext2fs/initialize.c @@ -575,3 +575,97 @@ cleanup: ext2fs_free(fs); return retval; } + +errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs, int super_only) +{ + blk64_t blk; + ext2_ino_t ino; + unsigned int group = 0; + unsigned int count = 0; + int total_free = 0; + int group_free = 0; + int last_allocated = 0; + int uninit; + + /* + * First calculate the block statistics + */ + uninit = 1; + for (blk = fs->super->s_first_data_block; + blk < ext2fs_blocks_count(fs->super); blk++) { + if (!ext2fs_fast_test_block_bitmap2(fs->block_map, blk)) { + group_free++; + total_free++; + } else { + uninit = 0; + } + count++; + if ((count == fs->super->s_blocks_per_group) || + (blk == ext2fs_blocks_count(fs->super)-1)) { + ext2fs_bg_free_blocks_count_set(fs, group, + group_free); + if (!super_only) { + if (uninit && blk != + ext2fs_blocks_count(fs->super) - 1) + ext2fs_bg_flags_set(fs, group, + EXT2_BG_BLOCK_UNINIT); + else + ext2fs_bg_flags_clear(fs, group, + EXT2_BG_BLOCK_UNINIT); + } + count = 0; + group_free = 0; + uninit = 1; + group++; + } + } + total_free = EXT2FS_C2B(fs, total_free); + ext2fs_free_blocks_count_set(fs->super, total_free); + + /* + * Next, calculate the inode statistics + */ + group_free = 0; + total_free = 0; + last_allocated = 0; + count = 0; + group = 0; + + /* Protect loop from wrap-around if s_inodes_count maxed */ + for (ino = 1; ino <= fs->super->s_inodes_count && ino > 0; ino++) { + if (!ext2fs_test_inode_bitmap2(fs->inode_map, ino)) { + group_free++; + total_free++; + } else { + last_allocated = ino; + } + count++; + if ((count == fs->super->s_inodes_per_group) || + (ino == fs->super->s_inodes_count)) { + if (!super_only) { + if (last_allocated) { + ext2fs_bg_flags_clear(fs, group, + EXT2_BG_INODE_UNINIT); + ext2fs_bg_itable_unused_set(fs, group, + fs->super->s_inodes_per_group - + (last_allocated % + fs->super->s_inodes_per_group)); + } else { + ext2fs_bg_flags_set(fs, group, + EXT2_BG_INODE_UNINIT); + ext2fs_bg_itable_unused_set(fs, group, + 0); + } + ext2fs_bg_free_inodes_count_set(fs, group, + group_free); + } + group++; + count = 0; + group_free = 0; + last_allocated = 0; + } + } + fs->super->s_free_inodes_count = total_free; + ext2fs_mark_super_dirty(fs); + return 0; +} diff --git a/misc/tune2fs.c b/misc/tune2fs.c index f942c698..670ed9e0 100644 --- a/misc/tune2fs.c +++ b/misc/tune2fs.c @@ -2603,63 +2603,6 @@ err_out: return retval; } -static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs) -{ - blk64_t blk; - ext2_ino_t ino; - unsigned int group = 0; - unsigned int count = 0; - int total_free = 0; - int group_free = 0; - - /* - * First calculate the block statistics - */ - for (blk = fs->super->s_first_data_block; - blk < ext2fs_blocks_count(fs->super); blk++) { - if (!ext2fs_fast_test_block_bitmap2(fs->block_map, blk)) { - group_free++; - total_free++; - } - count++; - if ((count == fs->super->s_blocks_per_group) || - (blk == ext2fs_blocks_count(fs->super)-1)) { - ext2fs_bg_free_blocks_count_set(fs, group++, - group_free); - count = 0; - group_free = 0; - } - } - total_free = EXT2FS_C2B(fs, total_free); - ext2fs_free_blocks_count_set(fs->super, total_free); - - /* - * Next, calculate the inode statistics - */ - group_free = 0; - total_free = 0; - count = 0; - group = 0; - - /* Protect loop from wrap-around if s_inodes_count maxed */ - for (ino = 1; ino <= fs->super->s_inodes_count && ino > 0; ino++) { - if (!ext2fs_fast_test_inode_bitmap2(fs->inode_map, ino)) { - group_free++; - total_free++; - } - count++; - if ((count == fs->super->s_inodes_per_group) || - (ino == fs->super->s_inodes_count)) { - ext2fs_bg_free_inodes_count_set(fs, group++, - group_free); - count = 0; - group_free = 0; - } - } - fs->super->s_free_inodes_count = total_free; - ext2fs_mark_super_dirty(fs); - return 0; -} #define list_for_each_safe(pos, pnext, head) \ for (pos = (head)->next, pnext = pos->next; pos != (head); \ @@ -2738,7 +2681,7 @@ static int resize_inode(ext2_filsys fs, unsigned long new_size) if (retval) goto err_out_undo; - ext2fs_calculate_summary_stats(fs); + ext2fs_calculate_summary_stats(fs, 1 /* super only */); fs->super->s_state |= EXT2_VALID_FS; /* mark super block and block bitmap as dirty */ diff --git a/resize/resize2fs.c b/resize/resize2fs.c index 2443ff67..270e4deb 100644 --- a/resize/resize2fs.c +++ b/resize/resize2fs.c @@ -49,7 +49,7 @@ static errcode_t inode_scan_and_fix(ext2_resize_t rfs); static errcode_t inode_ref_fix(ext2_resize_t rfs); static errcode_t move_itables(ext2_resize_t rfs); static errcode_t fix_resize_inode(ext2_filsys fs); -static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs); +static errcode_t resize2fs_calculate_summary_stats(ext2_filsys fs); static errcode_t fix_sb_journal_backup(ext2_filsys fs); static errcode_t mark_table_blocks(ext2_filsys fs, ext2fs_block_bitmap bmap); @@ -211,7 +211,7 @@ errcode_t resize_fs(ext2_filsys fs, blk64_t *new_size, int flags, goto errout; init_resource_track(&rtrack, "calculate_summary_stats", fs->io); - retval = ext2fs_calculate_summary_stats(rfs->new_fs); + retval = resize2fs_calculate_summary_stats(rfs->new_fs); if (retval) goto errout; print_resource_track(rfs, &rtrack, fs->io); @@ -2740,7 +2740,7 @@ errout: /* * Finally, recalculate the summary information */ -static errcode_t ext2fs_calculate_summary_stats(ext2_filsys fs) +static errcode_t resize2fs_calculate_summary_stats(ext2_filsys fs) { blk64_t blk; ext2_ino_t ino; From patchwork Fri Nov 20 19:15:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404062 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=KMFimrcB; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rB0sXbz9sTR for ; Sat, 21 Nov 2020 06:16:26 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732185AbgKTTQZ (ORCPT ); Fri, 20 Nov 2020 14:16:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729453AbgKTTQZ (ORCPT ); Fri, 20 Nov 2020 14:16:25 -0500 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 716E2C0613CF for ; Fri, 20 Nov 2020 11:16:25 -0800 (PST) Received: by mail-pl1-x644.google.com with SMTP id l11so5361175plt.1 for ; Fri, 20 Nov 2020 11:16:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LoaZ+lPWTh2hzvkHLYosf1B+tt8bIDWRafiaipRQw/I=; b=KMFimrcB8O4RhZ2ypYwvrTMteZQxXgA79418hifE0Q2aOEwyM86WnVTGzj8x0mYYKg Oh4OzLRB7dJsHF2PpUzKUUV50+gOHtRpuXQtub26TtghMtPiMwZ3jIC3JKcsQJQSyLVn lDX19a8yeiAhWOCTwdT9bBzg/zNmEiFPHoDPJ3rY1DcdalQZTjXrAGrZqKJ5hHREKwpC Z11wHu+Mbnd7b/xdTWOi0uq48JnfXFWJGkKMBT8We4rlHx+bxJTq4+C5ZMVQKYYvaH7k bTEa+En9hDBly+uL/jP4ODhjtnzBHouwvgEdRzEeFyfhFbpn6+uEJ7SWeJdroHPrd+Tx JB4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LoaZ+lPWTh2hzvkHLYosf1B+tt8bIDWRafiaipRQw/I=; b=WjNu+gWTtsoPtJEKORTyqexwTZVnMJNEST3DWgqTAehjujpvjoJSCZugEUwBant6xw XAGE54Fw861gi4sTx3w9Tm1er1L8TOeUqb+u6fgrXiwp+1QmGY73G7fDvN5E9FDUe8YO WxB2hFwfLoYVa5Jqafj6haTTgb3rm3EW69KHSvD1Prt76ftbd9CSakzliDo2qbLvfQSL OqIaulP23vjBmYpMk3/15+p4rAWBloiWp5NHJhRevcz+0sB2ecJ0TBncFO+dnC1zAnCT Xz9P2TkHH2rqgJroCdZA0iHDpNhx0QrXKejJ7joajOWQMvhaApi31AeNTPBMo6kHpDTt RFQQ== X-Gm-Message-State: AOAM532PC5OKsg5X3i7ag8O4sgv69iI4QG0khKPwxSdek59LNe5i9JSU e+ZyphvkR/6s8QcNePqietRyKIMl5Mg= X-Google-Smtp-Source: ABdhPJxphQ82aRQJ+RyZDlFR3qvRzC9zFdTfM531clLLGZPTyO1goJCtFMdC1sMA1Kc0ol8rkBkoLw== X-Received: by 2002:a17:902:7486:b029:d9:d4aa:e033 with SMTP id h6-20020a1709027486b02900d9d4aae033mr10544327pll.16.1605899784520; Fri, 20 Nov 2020 11:16:24 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:22 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 02/15] ext2fs, e2fsck: add kernel endian-ness conversion macros Date: Fri, 20 Nov 2020 11:15:53 -0800 Message-Id: <20201120191606.2224881-3-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org In order to make recovery.c identical with kernel, we need endianness conversion macros (such as cpu_to_be32 and friends) defined in e2fsprogs. This patch defines these macros and also fixes recovery.c to use these. These macros are also needed for fast commit recovery patches later in this series. Signed-off-by: Harshad Shirwadkar --- e2fsck/recovery.c | 42 ++++++++++-------------------------------- lib/ext2fs/bitops.h | 8 ++++++++ 2 files changed, 18 insertions(+), 32 deletions(-) diff --git a/e2fsck/recovery.c b/e2fsck/recovery.c index 5df690ad..6c3b7bb4 100644 --- a/e2fsck/recovery.c +++ b/e2fsck/recovery.c @@ -121,27 +121,6 @@ failed: #endif /* __KERNEL__ */ -static inline __u32 get_be32(__be32 *p) -{ - unsigned char *cp = (unsigned char *) p; - __u32 ret; - - ret = *cp++; - ret = (ret << 8) + *cp++; - ret = (ret << 8) + *cp++; - ret = (ret << 8) + *cp++; - return ret; -} - -static inline __u16 get_be16(__be16 *p) -{ - unsigned char *cp = (unsigned char *) p; - __u16 ret; - - ret = *cp++; - ret = (ret << 8) + *cp++; - return ret; -} /* * Read a block from the journal @@ -232,10 +211,10 @@ static int count_tags(journal_t *journal, struct buffer_head *bh) nr++; tagp += tag_bytes; - if (!(get_be16(&tag->t_flags) & JBD2_FLAG_SAME_UUID)) + if (!(tag->t_flags & cpu_to_be16(JBD2_FLAG_SAME_UUID))) tagp += 16; - if (get_be16(&tag->t_flags) & JBD2_FLAG_LAST_TAG) + if (tag->t_flags & cpu_to_be16(JBD2_FLAG_LAST_TAG)) break; } @@ -358,9 +337,9 @@ int jbd2_journal_skip_recovery(journal_t *journal) static inline unsigned long long read_tag_block(journal_t *journal, journal_block_tag_t *tag) { - unsigned long long block = get_be32(&tag->t_blocknr); + unsigned long long block = be32_to_cpu(tag->t_blocknr); if (jbd2_has_feature_64bit(journal)) - block |= (u64)get_be32(&tag->t_blocknr_high) << 32; + block |= (u64)be32_to_cpu(tag->t_blocknr_high) << 32; return block; } @@ -429,9 +408,9 @@ static int jbd2_block_tag_csum_verify(journal_t *j, journal_block_tag_t *tag, csum32 = jbd2_chksum(j, csum32, buf, j->j_blocksize); if (jbd2_has_feature_csum3(j)) - return get_be32(&tag3->t_checksum) == csum32; - - return get_be16(&tag->t_checksum) == (csum32 & 0xFFFF); + return tag3->t_checksum == cpu_to_be32(csum32); + else + return tag->t_checksum == cpu_to_be16(csum32); } static int do_one_pass(journal_t *journal, @@ -579,7 +558,7 @@ static int do_one_pass(journal_t *journal, unsigned long io_block; tag = (journal_block_tag_t *) tagp; - flags = get_be16(&tag->t_flags); + flags = be16_to_cpu(tag->t_flags); io_block = next_log_block++; wrap(journal, next_log_block); @@ -643,9 +622,8 @@ static int do_one_pass(journal_t *journal, memcpy(nbh->b_data, obh->b_data, journal->j_blocksize); if (flags & JBD2_FLAG_ESCAPE) { - __be32 magic = cpu_to_be32(JBD2_MAGIC_NUMBER); - memcpy(nbh->b_data, &magic, - sizeof(magic)); + *((__be32 *)nbh->b_data) = + cpu_to_be32(JBD2_MAGIC_NUMBER); } BUFFER_TRACE(nbh, "marking dirty"); diff --git a/lib/ext2fs/bitops.h b/lib/ext2fs/bitops.h index 505b3c9c..3c7b2496 100644 --- a/lib/ext2fs/bitops.h +++ b/lib/ext2fs/bitops.h @@ -247,6 +247,14 @@ extern errcode_t ext2fs_find_first_set_generic_bmap(ext2fs_generic_bitmap bitmap #endif /* __STDC_VERSION__ >= 199901L */ #endif /* INCLUDE_INLINE_FUNCS */ +/* Macros for kernel compatibility */ +#define be32_to_cpu(x) ext2fs_be32_to_cpu(x) +#define le32_to_cpu(x) ext2fs_le32_to_cpu(x) +#define le16_to_cpu(x) ext2fs_le16_to_cpu(x) + +#define cpu_to_be32(x) ext2fs_cpu_to_be32(x) +#define cpu_to_be16(x) ext2fs_cpu_to_be16(x) +#define cpu_to_le16(x) ext2fs_cpu_to_le16(x) /* * Fast bit set/clear functions that doesn't need to return the * previous bit value. From patchwork Fri Nov 20 19:15:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404063 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=drUWcdp8; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rD0HJKz9sTR for ; Sat, 21 Nov 2020 06:16:28 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732197AbgKTTQ1 (ORCPT ); Fri, 20 Nov 2020 14:16:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729453AbgKTTQ1 (ORCPT ); Fri, 20 Nov 2020 14:16:27 -0500 Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B642C0613CF for ; Fri, 20 Nov 2020 11:16:27 -0800 (PST) Received: by mail-pg1-x543.google.com with SMTP id 81so8133653pgf.0 for ; Fri, 20 Nov 2020 11:16:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=29shLS0fRrysv9ri8Z80C/AhILC50U976CeLSjjHo5s=; b=drUWcdp8IYSBsVKcTqfIuKAb/4rRJmYlMLPvUGbUrvKYLQ+jLiW7IO2EuXqZ3pmA28 02AIVw/vVQF8fG5WKhljiwQuIvkej0iDgaf+K51jvFxuFdjqV5pQ7VjBvRLGo1Utdlq6 YMywOaWGSfA5KJzUHRfGXANo6FbaLVfJ2+iqCj47qlToEkE7a5w6o5QubGNPPODPGSBm MvjECBwz7GfHg5/HZ1/OZAklp4m0B2r3Nz1DeaMuHvvOXnNctkFwX4+hutUJNp9mC3iZ 0AQ1WdKav3QbSSKTlkgEsMo8AFtp5xKvBXvxoK5eBRktREdih+IFa4m0thq/y0vcGRvp 1LkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=29shLS0fRrysv9ri8Z80C/AhILC50U976CeLSjjHo5s=; b=BVmAxhINZLwDXbKwigEEFLEBErL0IK2Du3J85s5OXI6hJKRy2WkKouecFgmgqiByTl pHSBpwM9vKmVdA5Q8phLG5WFW7C7nNt84IGb9I1SlhkeKyGE3MZJzt0gUHqZvNW7eNuw USaDOQaCq1oZkjfrdROfguNO2bLWXHVBVNk03BiRqemFG+rustJHWdFVojlD4/TA0fV9 ZhIhN6e3lEMlPLOWdxFfCD+3wwgGaH5x21kzXuXNmhKl85A6ShHeIXLO8NHgSQtamZwa Asv4qpYD4iaO1Rxu/G5MrUsIdT8Je5RybvM50DZoHXSY4D3FxVe+A6UWXx5eENa7G+3t gV5Q== X-Gm-Message-State: AOAM533AZ5Su4qDowGNcQ85OThL39pJ0UL8ncCHXQBYUlnX1BoKa66w7 O3YrsFnymFma03by7dsGL9+DqaTkaVY= X-Google-Smtp-Source: ABdhPJxRdkYNfG795nOURBMXqCPgsRiOxRxbMIS6kL9sJUV88/qjpqLz2yn+Gf95DfvBy/1LxRZvqA== X-Received: by 2002:aa7:8ed0:0:b029:18a:e177:7bce with SMTP id b16-20020aa78ed00000b029018ae1777bcemr15777323pfr.0.1605899786075; Fri, 20 Nov 2020 11:16:26 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:24 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 03/15] e2fsck: port fc changes from kernel's recovery.c to e2fsck Date: Fri, 20 Nov 2020 11:15:54 -0800 Message-Id: <20201120191606.2224881-4-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org This patch makes recovery.c identical with fast commit kernel changes. Signed-off-by: Harshad Shirwadkar Reviewed-by: Theodore Ts'o --- debugfs/journal.c | 10 +-- e2fsck/journal.c | 28 ++++-- e2fsck/recovery.c | 190 +++++++++++++++++++++++++++++----------- lib/ext2fs/jfs_compat.h | 19 +++- lib/ext2fs/kernel-jbd.h | 19 +++- 5 files changed, 197 insertions(+), 69 deletions(-) diff --git a/debugfs/journal.c b/debugfs/journal.c index fa72ec57..e8872f05 100644 --- a/debugfs/journal.c +++ b/debugfs/journal.c @@ -378,7 +378,7 @@ try_backup_journal: goto errout; } - journal->j_maxlen = EXT2_I_SIZE(&j_inode->i_ext2) / + journal->j_total_len = EXT2_I_SIZE(&j_inode->i_ext2) / journal->j_blocksize; #ifdef USE_INODE_IO @@ -493,7 +493,7 @@ try_backup_journal: brelse(bh); maxlen = ext2fs_blocks_count(&jsuper); - journal->j_maxlen = (maxlen < 1ULL << 32) ? maxlen : + journal->j_total_len = (maxlen < 1ULL << 32) ? maxlen : (1ULL << 32) - 1; start++; } @@ -629,9 +629,9 @@ static errcode_t ext2fs_journal_load(journal_t *journal) if (jsb->s_blocksize != htonl(journal->j_blocksize)) return EXT2_ET_CORRUPT_JOURNAL_SB; - if (ntohl(jsb->s_maxlen) < journal->j_maxlen) - journal->j_maxlen = ntohl(jsb->s_maxlen); - else if (ntohl(jsb->s_maxlen) > journal->j_maxlen) + if (ntohl(jsb->s_maxlen) < journal->j_total_len) + journal->j_total_len = ntohl(jsb->s_maxlen); + else if (ntohl(jsb->s_maxlen) > journal->j_total_len) return EXT2_ET_CORRUPT_JOURNAL_SB; journal->j_tail_sequence = ntohl(jsb->s_sequence); diff --git a/e2fsck/journal.c b/e2fsck/journal.c index 7d9f1b40..75fefcde 100644 --- a/e2fsck/journal.c +++ b/e2fsck/journal.c @@ -379,7 +379,7 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal) goto errout; } - journal->j_maxlen = EXT2_I_SIZE(&j_inode->i_ext2) / + journal->j_total_len = EXT2_I_SIZE(&j_inode->i_ext2) / journal->j_blocksize; #ifdef USE_INODE_IO @@ -503,7 +503,7 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal) brelse(bh); maxlen = ext2fs_blocks_count(&jsuper); - journal->j_maxlen = (maxlen < 1ULL << 32) ? maxlen : (1ULL << 32) - 1; + journal->j_total_len = (maxlen < 1ULL << 32) ? maxlen : (1ULL << 32) - 1; start++; } @@ -675,9 +675,9 @@ static errcode_t e2fsck_journal_load(journal_t *journal) return EXT2_ET_CORRUPT_JOURNAL_SB; } - if (ntohl(jsb->s_maxlen) < journal->j_maxlen) - journal->j_maxlen = ntohl(jsb->s_maxlen); - else if (ntohl(jsb->s_maxlen) > journal->j_maxlen) { + if (ntohl(jsb->s_maxlen) < journal->j_total_len) + journal->j_total_len = ntohl(jsb->s_maxlen); + else if (ntohl(jsb->s_maxlen) > journal->j_total_len) { com_err(ctx->program_name, EXT2_ET_CORRUPT_JOURNAL_SB, _("%s: journal too short\n"), ctx->device_name); @@ -688,7 +688,21 @@ static errcode_t e2fsck_journal_load(journal_t *journal) journal->j_transaction_sequence = journal->j_tail_sequence; journal->j_tail = ntohl(jsb->s_start); journal->j_first = ntohl(jsb->s_first); - journal->j_last = ntohl(jsb->s_maxlen); + if (jbd2_has_feature_fast_commit(journal)) { + if (ntohl(jsb->s_maxlen) - jbd2_journal_get_num_fc_blks(jsb) + < JBD2_MIN_JOURNAL_BLOCKS) { + com_err(ctx->program_name, EXT2_ET_CORRUPT_JOURNAL_SB, + _("%s: incorrect fast commit blocks\n"), + ctx->device_name); + return EXT2_ET_CORRUPT_JOURNAL_SB; + } + journal->j_fc_last = ntohl(jsb->s_maxlen); + journal->j_last = journal->j_fc_last - + jbd2_journal_get_num_fc_blks(jsb); + journal->j_fc_first = journal->j_last + 1; + } else { + journal->j_last = ntohl(jsb->s_maxlen); + } return 0; } @@ -720,7 +734,7 @@ static void e2fsck_journal_reset_super(e2fsck_t ctx, journal_superblock_t *jsb, memset (p, 0, ctx->fs->blocksize-sizeof(journal_header_t)); jsb->s_blocksize = htonl(ctx->fs->blocksize); - jsb->s_maxlen = htonl(journal->j_maxlen); + jsb->s_maxlen = htonl(journal->j_total_len); jsb->s_first = htonl(1); /* Initialize the journal sequence number so that there is "no" diff --git a/e2fsck/recovery.c b/e2fsck/recovery.c index 6c3b7bb4..dc0694fc 100644 --- a/e2fsck/recovery.c +++ b/e2fsck/recovery.c @@ -35,7 +35,6 @@ struct recovery_info int nr_revoke_hits; }; -enum passtype {PASS_SCAN, PASS_REVOKE, PASS_REPLAY}; static int do_one_pass(journal_t *journal, struct recovery_info *info, enum passtype pass); static int scan_revoke_records(journal_t *, struct buffer_head *, @@ -75,8 +74,8 @@ static int do_readahead(journal_t *journal, unsigned int start) /* Do up to 128K of readahead */ max = start + (128 * 1024 / journal->j_blocksize); - if (max > journal->j_maxlen) - max = journal->j_maxlen; + if (max > journal->j_total_len) + max = journal->j_total_len; /* Do the readahead itself. We'll submit MAXBUF buffer_heads at * a time to the block device IO layer. */ @@ -135,7 +134,7 @@ static int jread(struct buffer_head **bhp, journal_t *journal, *bhp = NULL; - if (offset >= journal->j_maxlen) { + if (offset >= journal->j_total_len) { printk(KERN_ERR "JBD2: corrupted journal superblock\n"); return -EFSCORRUPTED; } @@ -180,7 +179,7 @@ static int jbd2_descriptor_block_csum_verify(journal_t *j, void *buf) if (!jbd2_journal_has_csum_v2or3(j)) return 1; - tail = (struct jbd2_journal_block_tail *)((char *)buf + j->j_blocksize - + tail = (struct jbd2_journal_block_tail *)(buf + j->j_blocksize - sizeof(struct jbd2_journal_block_tail)); provided = tail->t_checksum; tail->t_checksum = 0; @@ -225,10 +224,51 @@ static int count_tags(journal_t *journal, struct buffer_head *bh) /* Make sure we wrap around the log correctly! */ #define wrap(journal, var) \ do { \ - if (var >= (journal)->j_last) \ - var -= ((journal)->j_last - (journal)->j_first); \ + unsigned long _wrap_last = \ + jbd2_has_feature_fast_commit(journal) ? \ + (journal)->j_fc_last : (journal)->j_last; \ + \ + if (var >= _wrap_last) \ + var -= (_wrap_last - (journal)->j_first); \ } while (0) +static int fc_do_one_pass(journal_t *journal, + struct recovery_info *info, enum passtype pass) +{ + unsigned int expected_commit_id = info->end_transaction; + unsigned long next_fc_block; + struct buffer_head *bh; + int err = 0; + + next_fc_block = journal->j_fc_first; + if (!journal->j_fc_replay_callback) + return 0; + + while (next_fc_block <= journal->j_fc_last) { + jbd_debug(3, "Fast commit replay: next block %ld", + next_fc_block); + err = jread(&bh, journal, next_fc_block); + if (err) { + jbd_debug(3, "Fast commit replay: read error"); + break; + } + + jbd_debug(3, "Processing fast commit blk with seq %d"); + err = journal->j_fc_replay_callback(journal, bh, pass, + next_fc_block - journal->j_fc_first, + expected_commit_id); + next_fc_block++; + if (err < 0 || err == JBD2_FC_REPLAY_STOP) + break; + err = 0; + } + + if (err) + jbd_debug(3, "Fast commit replay failed, err = %d\n", err); + + return err; +} + /** * jbd2_journal_recover - recovers a on-disk journal * @journal: the journal to recover @@ -286,7 +326,7 @@ int jbd2_journal_recover(journal_t *journal) err = err2; /* Make sure all replayed data is on permanent storage */ if (journal->j_flags & JBD2_BARRIER) { - err2 = blkdev_issue_flush(journal->j_fs_dev, GFP_KERNEL, NULL); + err2 = blkdev_issue_flush(journal->j_fs_dev, GFP_KERNEL); if (!err) err = err2; } @@ -428,6 +468,8 @@ static int do_one_pass(journal_t *journal, __u32 crc32_sum = ~0; /* Transactional Checksums */ int descr_csum_size = 0; int block_error = 0; + bool need_check_commit_time = false; + __u64 last_trans_commit_time = 0, commit_time; /* * First thing is to establish what we expect to find in the log @@ -470,7 +512,9 @@ static int do_one_pass(journal_t *journal, break; jbd_debug(2, "Scanning for sequence ID %u at %lu/%lu\n", - next_commit_ID, next_log_block, journal->j_last); + next_commit_ID, next_log_block, + jbd2_has_feature_fast_commit(journal) ? + journal->j_fc_last : journal->j_last); /* Skip over each chunk of the transaction looking * either the next descriptor block or the final commit @@ -520,9 +564,21 @@ static int do_one_pass(journal_t *journal, if (descr_csum_size > 0 && !jbd2_descriptor_block_csum_verify(journal, bh->b_data)) { - err = -EFSBADCRC; - brelse(bh); - goto failed; + /* + * PASS_SCAN can see stale blocks due to lazy + * journal init. Don't error out on those yet. + */ + if (pass != PASS_SCAN) { + pr_err("JBD2: Invalid checksum recovering block %lu in log\n", + next_log_block); + err = -EFSBADCRC; + brelse(bh); + goto failed; + } + need_check_commit_time = true; + jbd_debug(1, + "invalid descriptor block found in %lu\n", + next_log_block); } /* If it is a valid descriptor block, replay it @@ -532,6 +588,7 @@ static int do_one_pass(journal_t *journal, if (pass != PASS_REPLAY) { if (pass == PASS_SCAN && jbd2_has_feature_checksum(journal) && + !need_check_commit_time && !info->end_transaction) { if (calc_chksums(journal, bh, &next_log_block, @@ -663,7 +720,7 @@ static int do_one_pass(journal_t *journal, * | GO TO NEXT "Journal Corruption" * | TRANSACTION * | - * {(n+1)th transaction} + * {(n+1)th transanction} * | * _______|______________ * | | @@ -680,21 +737,48 @@ static int do_one_pass(journal_t *journal, * mentioned conditions. Hence assume * "Interrupted Commit".) */ + commit_time = be64_to_cpu( + ((struct commit_header *)bh->b_data)->h_commit_sec); + /* + * If need_check_commit_time is set, it means we are in + * PASS_SCAN and csum verify failed before. If + * commit_time is increasing, it's the same journal, + * otherwise it is stale journal block, just end this + * recovery. + */ + if (need_check_commit_time) { + if (commit_time >= last_trans_commit_time) { + pr_err("JBD2: Invalid checksum found in transaction %u\n", + next_commit_ID); + err = -EFSBADCRC; + brelse(bh); + goto failed; + } + ignore_crc_mismatch: + /* + * It likely does not belong to same journal, + * just end this recovery with success. + */ + jbd_debug(1, "JBD2: Invalid checksum ignored in transaction %u, likely stale data\n", + next_commit_ID); + err = 0; + brelse(bh); + goto done; + } - /* Found an expected commit block: if checksums - * are present verify them in PASS_SCAN; else not + /* + * Found an expected commit block: if checksums + * are present, verify them in PASS_SCAN; else not * much to do other than move on to the next sequence - * number. */ + * number. + */ if (pass == PASS_SCAN && jbd2_has_feature_checksum(journal)) { - int chksum_err, chksum_seen; struct commit_header *cbh = (struct commit_header *)bh->b_data; unsigned found_chksum = be32_to_cpu(cbh->h_chksum[0]); - chksum_err = chksum_seen = 0; - if (info->end_transaction) { journal->j_failed_commit = info->end_transaction; @@ -702,42 +786,25 @@ static int do_one_pass(journal_t *journal, break; } - if (crc32_sum == found_chksum && - cbh->h_chksum_type == JBD2_CRC32_CHKSUM && - cbh->h_chksum_size == - JBD2_CRC32_CHKSUM_SIZE) - chksum_seen = 1; - else if (!(cbh->h_chksum_type == 0 && - cbh->h_chksum_size == 0 && - found_chksum == 0 && - !chksum_seen)) - /* - * If fs is mounted using an old kernel and then - * kernel with journal_chksum is used then we - * get a situation where the journal flag has - * checksum flag set but checksums are not - * present i.e chksum = 0, in the individual - * commit blocks. - * Hence to avoid checksum failures, in this - * situation, this extra check is added. - */ - chksum_err = 1; - - if (chksum_err) { - info->end_transaction = next_commit_ID; + /* Neither checksum match nor unused? */ + if (!((crc32_sum == found_chksum && + cbh->h_chksum_type == + JBD2_CRC32_CHKSUM && + cbh->h_chksum_size == + JBD2_CRC32_CHKSUM_SIZE) || + (cbh->h_chksum_type == 0 && + cbh->h_chksum_size == 0 && + found_chksum == 0))) + goto chksum_error; - if (!jbd2_has_feature_async_commit(journal)) { - journal->j_failed_commit = - next_commit_ID; - brelse(bh); - break; - } - } crc32_sum = ~0; } if (pass == PASS_SCAN && !jbd2_commit_block_csum_verify(journal, bh->b_data)) { + chksum_error: + if (commit_time < last_trans_commit_time) + goto ignore_crc_mismatch; info->end_transaction = next_commit_ID; if (!jbd2_has_feature_async_commit(journal)) { @@ -747,11 +814,24 @@ static int do_one_pass(journal_t *journal, break; } } + if (pass == PASS_SCAN) + last_trans_commit_time = commit_time; brelse(bh); next_commit_ID++; continue; case JBD2_REVOKE_BLOCK: + /* + * Check revoke block crc in pass_scan, if csum verify + * failed, check commit block time later. + */ + if (pass == PASS_SCAN && + !jbd2_descriptor_block_csum_verify(journal, + bh->b_data)) { + jbd_debug(1, "JBD2: invalid revoke block found in %lu\n", + next_log_block); + need_check_commit_time = true; + } /* If we aren't in the REVOKE pass, then we can * just skip over this block. */ if (pass != PASS_REVOKE) { @@ -796,6 +876,13 @@ static int do_one_pass(journal_t *journal, success = -EIO; } } + + if (jbd2_has_feature_fast_commit(journal) && pass != PASS_REVOKE) { + err = fc_do_one_pass(journal, info, pass); + if (err) + success = err; + } + if (block_error && success == 0) success = -EIO; return success; @@ -811,7 +898,7 @@ static int scan_revoke_records(journal_t *journal, struct buffer_head *bh, { jbd2_journal_revoke_header_t *header; int offset, max; - unsigned csum_size = 0; + int csum_size = 0; __u32 rcount; int record_len = 4; @@ -819,9 +906,6 @@ static int scan_revoke_records(journal_t *journal, struct buffer_head *bh, offset = sizeof(jbd2_journal_revoke_header_t); rcount = be32_to_cpu(header->r_count); - if (!jbd2_descriptor_block_csum_verify(journal, header)) - return -EFSBADCRC; - if (jbd2_journal_has_csum_v2or3(journal)) csum_size = sizeof(struct jbd2_journal_block_tail); if (rcount > journal->j_blocksize - csum_size) diff --git a/lib/ext2fs/jfs_compat.h b/lib/ext2fs/jfs_compat.h index 2bda521d..d6da6bb1 100644 --- a/lib/ext2fs/jfs_compat.h +++ b/lib/ext2fs/jfs_compat.h @@ -12,6 +12,7 @@ #else #include #endif +#include #define printk printf #define KERN_ERR "" @@ -60,9 +61,15 @@ static inline __u32 jbd2_chksum(journal_t *j EXT2FS_ATTR((unused)), sizeof(struct __struct), __alignof__(struct __struct),\ (__flags), NULL) -#define blkdev_issue_flush(kdev, a, b) sync_blockdev(kdev) +#define blkdev_issue_flush(kdev, a) sync_blockdev(kdev) #define is_power_of_2(x) ((x) != 0 && (((x) & ((x) - 1)) == 0)) #define pr_emerg(fmt) +#define pr_err(...) + +enum passtype {PASS_SCAN, PASS_REVOKE, PASS_REPLAY}; + +#define JBD2_FC_REPLAY_STOP 0 +#define JBD2_FC_REPLAY_CONTINUE 1 struct journal_s { @@ -73,13 +80,16 @@ struct journal_s int j_format_version; unsigned long j_head; unsigned long j_tail; + unsigned long j_fc_first; + unsigned long j_fc_off; + unsigned long j_fc_last; unsigned long j_free; unsigned long j_first, j_last; kdev_t j_dev; kdev_t j_fs_dev; int j_blocksize; unsigned int j_blk_offset; - unsigned int j_maxlen; + unsigned int j_total_len; struct inode * j_inode; tid_t j_tail_sequence; tid_t j_transaction_sequence; @@ -88,6 +98,11 @@ struct journal_s struct jbd2_revoke_table_s *j_revoke_table[2]; tid_t j_failed_commit; __u32 j_csum_seed; + int (*j_fc_replay_callback)(struct journal_s *journal, + struct buffer_head *bh, + enum passtype pass, int off, + tid_t expected_tid); + }; #define is_journal_abort(x) 0 diff --git a/lib/ext2fs/kernel-jbd.h b/lib/ext2fs/kernel-jbd.h index cb1bc308..2978ccb6 100644 --- a/lib/ext2fs/kernel-jbd.h +++ b/lib/ext2fs/kernel-jbd.h @@ -74,6 +74,7 @@ extern void * __jbd_kmalloc (char *where, size_t size, int flags, int retry); __jbd_kmalloc(__FUNCTION__, (size), (flags), 1) #define JBD2_MIN_JOURNAL_BLOCKS 1024 +#define JBD2_DEFAULT_FAST_COMMIT_BLOCKS 256 /* * Internal structures used by the logging mechanism: @@ -94,6 +95,7 @@ extern void * __jbd_kmalloc (char *where, size_t size, int flags, int retry); #define JBD2_SUPERBLOCK_V1 3 #define JBD2_SUPERBLOCK_V2 4 #define JBD2_REVOKE_BLOCK 5 +#define JBD2_FC_BLOCK 6 /* * Standard header for all descriptor blocks: @@ -233,7 +235,10 @@ typedef struct journal_superblock_s /* 0x0050 */ __u8 s_checksum_type; /* checksum type */ __u8 s_padding2[3]; - __be32 s_padding[42]; +/* 0x0054 */ + __be32 s_num_fc_blks; /* Number of fast commit blocks */ +/* 0x0058 */ + __be32 s_padding[41]; __be32 s_checksum; /* crc32c(superblock) */ /* 0x0100 */ @@ -259,6 +264,7 @@ typedef struct journal_superblock_s #define JBD2_FEATURE_INCOMPAT_ASYNC_COMMIT 0x00000004 #define JBD2_FEATURE_INCOMPAT_CSUM_V2 0x00000008 #define JBD2_FEATURE_INCOMPAT_CSUM_V3 0x00000010 +#define JBD2_FEATURE_INCOMPAT_FAST_COMMIT 0x00000020 /* Features known to this kernel version: */ #define JBD2_KNOWN_COMPAT_FEATURES 0 @@ -267,7 +273,8 @@ typedef struct journal_superblock_s JBD2_FEATURE_INCOMPAT_ASYNC_COMMIT| \ JBD2_FEATURE_INCOMPAT_64BIT|\ JBD2_FEATURE_INCOMPAT_CSUM_V2| \ - JBD2_FEATURE_INCOMPAT_CSUM_V3) + JBD2_FEATURE_INCOMPAT_CSUM_V3 | \ + JBD2_FEATURE_INCOMPAT_FAST_COMMIT) #ifdef NO_INLINE_FUNCS extern size_t journal_tag_bytes(journal_t *journal); @@ -384,6 +391,7 @@ JBD2_FEATURE_INCOMPAT_FUNCS(64bit, 64BIT) JBD2_FEATURE_INCOMPAT_FUNCS(async_commit, ASYNC_COMMIT) JBD2_FEATURE_INCOMPAT_FUNCS(csum2, CSUM_V2) JBD2_FEATURE_INCOMPAT_FUNCS(csum3, CSUM_V3) +JBD2_FEATURE_INCOMPAT_FUNCS(fast_commit, FAST_COMMIT) #if (defined(E2FSCK_INCLUDE_INLINE_FUNCS) || !defined(NO_INLINE_FUNCS)) /* @@ -415,6 +423,13 @@ _INLINE_ int jbd2_journal_has_csum_v2or3(journal_t *journal) return 0; } +_INLINE_ int jbd2_journal_get_num_fc_blks(journal_superblock_t *jsb) +{ + int num_fc_blocks = be32_to_cpu(jsb->s_num_fc_blks); + + return num_fc_blocks ? num_fc_blocks : JBD2_DEFAULT_FAST_COMMIT_BLOCKS; +} + /* Comparison functions for transaction IDs: perform comparisons using * modulo arithmetic so that they work over sequence number wraps. */ From patchwork Fri Nov 20 19:15:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404064 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=lhqyaOt+; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rG0BQbz9sTR for ; Sat, 21 Nov 2020 06:16:30 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729571AbgKTTQ3 (ORCPT ); Fri, 20 Nov 2020 14:16:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732198AbgKTTQ3 (ORCPT ); Fri, 20 Nov 2020 14:16:29 -0500 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F02CC0613CF for ; Fri, 20 Nov 2020 11:16:29 -0800 (PST) Received: by mail-pg1-x536.google.com with SMTP id i13so8102800pgm.9 for ; Fri, 20 Nov 2020 11:16:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VcMOaeyz0ktxpU6MKQrj/GLeTsJJb5U72B60NctIIZw=; b=lhqyaOt+QCiTWZUBqQVRWetD1e0CJtt5kCGcLAC+7XLNsRTP64PS8sgfv4nj1d51zt YaRBcMWXzfkHBqXMevxaEvUJ6R0rMMceZbdnw1K/W9QaBRr0OShG6tbVu6pLHECO0DB5 HD8HU2oiwL96piAR2/jCP/UIElwY3AKZ17Wf+UCi0DPXJtWoi3f50KfypiX91b8O2eVH RvnwbjwhWbYHArWL448dCPtyuzMdtAX49e/7+eQNRYWEYLLsYI42xhLWVNDmGau4zIPu 28ZUwQkY9+mkjn5WWj1T00eHcbhpp3tLnKzt3yp2O3mzQr8sepsXNce1mdQ2gH/BCPvu KdBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VcMOaeyz0ktxpU6MKQrj/GLeTsJJb5U72B60NctIIZw=; b=QC9XsZytDXH0ymOp+BUNwcd2+B3ecZX1ncJuYjNkkepjrDxMm/qGIK0ZzIG/X5LcfX OkLcvaZs4Yfu+qECR87arDGQYtNXywF7BCWfNxAQllt26mKxGsMPjzUWi6BebH7AQlh6 jkobuj/WaK4zg7LYkH+59Z0Vz0+g7hG07O4RmVuBRP5KAxTH/+93rIQPghP5TrEuGwG0 6DPHSBzcoJZxYOcvk978qqhahnUmQeI51ubg0EjuOI6mZ1BXWhV2uH6Gq7lyQS8ExSTC 0kRh1lnGxhdoTJGRqY5BbKAmrZT6y96tKUx0uZukZK8i8bKEtcgz+PdznrcU15ntRPh/ E1Xg== X-Gm-Message-State: AOAM5328YulYZhVPmXwM30IiP2C/5yO0oy6xnHlvngA9ABiSF5T3ygUQ nkBzFSTTohaRN7ZL9WrJmHVrR3I9j68= X-Google-Smtp-Source: ABdhPJway0sgv7+CFgj82MdU4hhtmL5Zajt1NGeetwzYQ8sQj+QydfIjw5ZKFFA84fa6U0pV57xcAA== X-Received: by 2002:a17:90a:458e:: with SMTP id v14mr11844064pjg.40.1605899787969; Fri, 20 Nov 2020 11:16:27 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:26 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 04/15] mke2fs, dumpe2fs: make fast commit blocks configurable Date: Fri, 20 Nov 2020 11:15:55 -0800 Message-Id: <20201120191606.2224881-5-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org This patch makes number of fast commit blocks configurable. Also, the number of fast commit blocks can now be seen in dumpe2fs output. $ ./misc/mke2fs -O fast_commit -t ext4 image mke2fs 1.46-WIP (20-Mar-2020) Discarding device blocks: done Creating filesystem with 5120 1k blocks and 1280 inodes Allocating group tables: done Writing inode tables: done Creating journal (1040 blocks): done Writing superblocks and filesystem accounting information: done $ ./misc/dumpe2fs image dumpe2fs 1.46-WIP (20-Mar-2020) ... Journal features: (none) Total journal size: 1040k Total journal blocks: 1040 Max transaction length: 1024 Fast commit length: 16 Journal sequence: 0x00000001 Journal start: 0 $ ./misc/mke2fs -O fast_commit -t ext4 image -J fast_commit_size=256,size=1 mke2fs 1.46-WIP (20-Mar-2020) Creating filesystem with 5120 1k blocks and 1280 inodes Allocating group tables: done Writing inode tables: done Creating journal (1280 blocks): done Writing superblocks and filesystem accounting information: done $ ./misc/dumpe2fs image dumpe2fs 1.46-WIP (20-Mar-2020) ... Journal features: (none) Total journal size: 1280k Total journal blocks: 1280 Max transaction length: 1024 Fast commit length: 256 Journal sequence: 0x00000001 Journal start: 0 Signed-off-by: Harshad Shirwadkar --- e2fsck/unix.c | 8 ++-- lib/e2p/e2p.h | 1 + lib/e2p/ljs.c | 16 +++++-- lib/ext2fs/ext2fs.h | 21 ++++++--- lib/ext2fs/mkjournal.c | 99 +++++++++++++++++++++++++++++------------- misc/dumpe2fs.c | 10 ++++- misc/mke2fs.c | 24 +++++++--- misc/tune2fs.c | 8 ++-- misc/util.c | 63 ++++++++++++++++++++------- misc/util.h | 4 +- 10 files changed, 185 insertions(+), 69 deletions(-) diff --git a/e2fsck/unix.c b/e2fsck/unix.c index 1cb51672..3162896a 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -1388,7 +1388,7 @@ int main (int argc, char *argv[]) blk64_t orig_superblock = ~(blk64_t)0; struct problem_context pctx; int flags, run_result, was_changed; - int journal_size; + int journal_size, journal_fc_size; int sysval, sys_page_size = 4096; int old_bitmaps; __u32 features[3]; @@ -1912,7 +1912,7 @@ print_unsupp_features: (ctx->flags & E2F_FLAG_JOURNAL_INODE)) { if (fix_problem(ctx, PR_6_RECREATE_JOURNAL, &pctx)) { if (journal_size < 1024) - journal_size = ext2fs_default_journal_size(ext2fs_blocks_count(fs->super)); + ext2fs_default_journal_size(&journal_size, &journal_fc_size, fs); if (journal_size < 0) { ext2fs_clear_feature_journal(fs->super); fs->flags &= ~EXT2_FLAG_MASTER_SB_ONLY; @@ -1923,8 +1923,8 @@ print_unsupp_features: log_out(ctx, _("Creating journal (%d blocks): "), journal_size); fflush(stdout); - retval = ext2fs_add_journal_inode(fs, - journal_size, 0); + retval = ext2fs_add_journal_inode(fs, journal_size, + journal_fc_size, 0); if (retval) { log_out(ctx, "%s: while trying to create " "journal\n", error_message(retval)); diff --git a/lib/e2p/e2p.h b/lib/e2p/e2p.h index 90efb624..65702a7e 100644 --- a/lib/e2p/e2p.h +++ b/lib/e2p/e2p.h @@ -47,6 +47,7 @@ void print_fs_state (FILE * f, unsigned short state); int setflags (int fd, unsigned long flags); int setversion (int fd, unsigned long version); +#define E2P_LIST_JOURNAL_FLAG_FC 0x1 void e2p_list_journal_super(FILE *f, char *journal_sb_buf, int exp_block_size, int flags); diff --git a/lib/e2p/ljs.c b/lib/e2p/ljs.c index 4ffe9b61..9f866c7e 100644 --- a/lib/e2p/ljs.c +++ b/lib/e2p/ljs.c @@ -54,7 +54,12 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf, unsigned int size; int j, printed = 0; unsigned int i, nr_users; + int num_fc_blks = 0; + int journal_blks = 0; + if (flags & E2P_LIST_JOURNAL_FLAG_FC) + num_fc_blks = jbd2_journal_get_num_fc_blks((journal_superblock_t *)journal_sb_buf); + journal_blks = ntohl(jsb->s_maxlen) - num_fc_blks; fprintf(f, "%s", "Journal features: "); for (i=0, mask_ptr=&jsb->s_feature_compat; i <3; i++,mask_ptr++) { mask = e2p_be32(*mask_ptr); @@ -68,7 +73,7 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf, if (printed == 0) fprintf(f, " (none)"); fputc('\n', f); - fputs("Journal size: ", f); + fputs("Total journal size: ", f); size = (ntohl(jsb->s_blocksize) / 1024) * ntohl(jsb->s_maxlen); if (size < 8192) fprintf(f, "%uk\n", size); @@ -78,8 +83,13 @@ void e2p_list_journal_super(FILE *f, char *journal_sb_buf, if (exp_block_size != (int) ntohl(jsb->s_blocksize)) fprintf(f, "Journal block size: %u\n", (unsigned int)ntohl(jsb->s_blocksize)); - fprintf(f, "Journal length: %u\n", - (unsigned int)ntohl(jsb->s_maxlen)); + fprintf(f, "Total journal blocks: %u\n", + (unsigned int)(journal_blks + num_fc_blks)); + fprintf(f, "Max transaction length: %u\n", + (unsigned int)journal_blks); + fprintf(f, "Fast commit length: %u\n", + (unsigned int)num_fc_blks); + if (ntohl(jsb->s_first) != 1) fprintf(f, "Journal first block: %u\n", (unsigned int)ntohl(jsb->s_first)); diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index a8a6e091..01132245 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -220,6 +220,12 @@ typedef struct ext2_file *ext2_file_t; #define EXT2_MKJOURNAL_LAZYINIT 0x0000002 /* don't zero journal inode before use*/ #define EXT2_MKJOURNAL_NO_MNT_CHECK 0x0000004 /* don't check mount status */ +/* + * Normal journal area size to fast commit area size ratio. This is used to + * set default size of fast commit area. + */ +#define EXT2_JOURNAL_TO_FC_BLKS_RATIO 64 + struct blk_alloc_ctx; struct opaque_ext2_group_desc; @@ -1625,15 +1631,18 @@ extern errcode_t ext2fs_zero_blocks(ext2_filsys fs, blk_t blk, int num, extern errcode_t ext2fs_zero_blocks2(ext2_filsys fs, blk64_t blk, int num, blk64_t *ret_blk, int *ret_count); extern errcode_t ext2fs_create_journal_superblock(ext2_filsys fs, - __u32 num_blocks, int flags, - char **ret_jsb); + __u32 num_blocks, __u32 num_fc_blks, + int flags, char **ret_jsb); +extern errcode_t ext2fs_split_journal_size(ext2_filsys fs, blk_t *journal_blks, + blk_t *fc_blks, blk_t total_blks); extern errcode_t ext2fs_add_journal_device(ext2_filsys fs, ext2_filsys journal_dev); extern errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks, - int flags); + blk_t num_fc_blocks, int flags); extern errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks, - blk64_t goal, int flags); -extern int ext2fs_default_journal_size(__u64 num_blocks); + blk_t num_fc_blocks, + blk64_t goal, int flags); +extern errcode_t ext2fs_default_journal_size(int *journal_size, int *fc_size, ext2_filsys fs); extern int ext2fs_journal_sb_start(int blocksize); /* openfs.c */ @@ -2122,6 +2131,8 @@ static inline unsigned int ext2_dir_htree_level(ext2_filsys fs) return EXT4_HTREE_LEVEL_COMPAT; } +#define max(a, b) ((a) > (b) ? (a) : (b)) + #ifdef __cplusplus } #endif diff --git a/lib/ext2fs/mkjournal.c b/lib/ext2fs/mkjournal.c index f47f71e6..74d0c7fc 100644 --- a/lib/ext2fs/mkjournal.c +++ b/lib/ext2fs/mkjournal.c @@ -12,6 +12,7 @@ #include "config.h" #include #include +#include #if HAVE_UNISTD_H #include #endif @@ -39,13 +40,32 @@ #include "kernel-jbd.h" +errcode_t ext2fs_split_journal_size(ext2_filsys fs, blk_t *journal_blks, + blk_t *fc_blks, blk_t total_blks) +{ + if (total_blks < JBD2_MIN_JOURNAL_BLOCKS) + return EXT2_ET_JOURNAL_TOO_SMALL; + + if (!ext2fs_has_feature_fast_commit(fs->super)) { + *journal_blks = total_blks; + *fc_blks = 0; + return 0; + } + *journal_blks = ext2fs_blocks_count(fs->super) * + EXT2_JOURNAL_TO_FC_BLKS_RATIO / + (EXT2_JOURNAL_TO_FC_BLKS_RATIO + 1); + *journal_blks = max(JBD2_MIN_JOURNAL_BLOCKS, *journal_blks); + *fc_blks = total_blks - *journal_blks; + return 0; +} + /* * This function automatically sets up the journal superblock and * returns it as an allocated block. */ errcode_t ext2fs_create_journal_superblock(ext2_filsys fs, - __u32 num_blocks, int flags, - char **ret_jsb) + __u32 num_blocks, __u32 num_fc_blks, + int flags, char **ret_jsb) { errcode_t retval; journal_superblock_t *jsb; @@ -64,10 +84,11 @@ errcode_t ext2fs_create_journal_superblock(ext2_filsys fs, else jsb->s_header.h_blocktype = htonl(JBD2_SUPERBLOCK_V2); jsb->s_blocksize = htonl(fs->blocksize); - jsb->s_maxlen = htonl(num_blocks); + jsb->s_maxlen = htonl(num_blocks + num_fc_blks); jsb->s_nr_users = htonl(1); jsb->s_first = htonl(1); jsb->s_sequence = htonl(1); + jsb->s_num_fc_blks = htonl(num_fc_blks); memcpy(jsb->s_uuid, fs->super->s_uuid, sizeof(fs->super->s_uuid)); /* * If we're creating an external journal device, we need to @@ -88,14 +109,16 @@ errcode_t ext2fs_create_journal_superblock(ext2_filsys fs, * filesystems. */ static errcode_t write_journal_file(ext2_filsys fs, char *filename, - blk_t num_blocks, int flags) + blk_t num_blocks, blk_t num_fc_blocks, + int flags) { errcode_t retval; char *buf = 0; int fd, ret_size; blk_t i; - if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags, + if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, + num_fc_blocks, flags, &buf))) return retval; @@ -119,7 +142,7 @@ static errcode_t write_journal_file(ext2_filsys fs, char *filename, if (flags & EXT2_MKJOURNAL_LAZYINIT) goto success; - for (i = 1; i < num_blocks; i++) { + for (i = 1; i < num_blocks + num_fc_blocks; i++) { ret_size = write(fd, buf, fs->blocksize); if (ret_size < 0) { retval = errno; @@ -262,7 +285,8 @@ static blk64_t get_midpoint_journal_block(ext2_filsys fs) * This function creates a journal using direct I/O routines. */ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino, - blk_t num_blocks, blk64_t goal, int flags) + blk_t num_blocks, blk_t num_fc_blocks, + blk64_t goal, int flags) { char *buf; errcode_t retval; @@ -271,7 +295,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino, int falloc_flags = EXT2_FALLOCATE_FORCE_INIT; blk64_t zblk; - if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, flags, + if ((retval = ext2fs_create_journal_superblock(fs, num_blocks, + num_fc_blocks, flags, &buf))) return retval; @@ -295,7 +320,8 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino, if (!(flags & EXT2_MKJOURNAL_LAZYINIT)) falloc_flags |= EXT2_FALLOCATE_ZERO_BLOCKS; - inode_size = (unsigned long long)fs->blocksize * num_blocks; + inode_size = (unsigned long long)fs->blocksize * + (num_blocks + num_fc_blocks); inode.i_mtime = inode.i_ctime = fs->now ? fs->now : time(0); inode.i_links_count = 1; inode.i_mode = LINUX_S_IFREG | 0600; @@ -304,7 +330,7 @@ static errcode_t write_journal_inode(ext2_filsys fs, ext2_ino_t journal_ino, goto out2; retval = ext2fs_fallocate(fs, falloc_flags, journal_ino, - &inode, goal, 0, num_blocks); + &inode, goal, 0, num_blocks + num_fc_blocks); if (retval) goto out2; @@ -337,25 +363,34 @@ out2: * * n.b. comments assume 4k blocks */ -int ext2fs_default_journal_size(__u64 num_blocks) +errcode_t ext2fs_default_journal_size(int *journal_size, int *fc_size, ext2_filsys fs) { + __u64 num_blocks = ext2fs_blocks_count(fs->super); + if (num_blocks < 2048) - return -1; + return EXT2_ET_TOOSMALL; if (num_blocks < 32768) /* 128 MB */ - return (1024); /* 4 MB */ - if (num_blocks < 256*1024) /* 1 GB */ - return (4096); /* 16 MB */ - if (num_blocks < 512*1024) /* 2 GB */ - return (8192); /* 32 MB */ - if (num_blocks < 4096*1024) /* 16 GB */ - return (16384); /* 64 MB */ - if (num_blocks < 8192*1024) /* 32 GB */ - return (32768); /* 128 MB */ - if (num_blocks < 16384*1024) /* 64 GB */ - return (65536); /* 256 MB */ - if (num_blocks < 32768*1024) /* 128 GB */ - return (131072); /* 512 MB */ - return 262144; /* 1 GB */ + *journal_size = 1024; /* 4 MB */ + else if (num_blocks < 256*1024) /* 1 GB */ + *journal_size = 4096; /* 16 MB */ + else if (num_blocks < 512*1024) /* 2 GB */ + *journal_size = 8192; /* 32 MB */ + else if (num_blocks < 4096*1024) /* 16 GB */ + *journal_size = 16384; /* 64 MB */ + else if (num_blocks < 8192*1024) /* 32 GB */ + *journal_size = 32768; /* 128 MB */ + else if (num_blocks < 16384*1024) /* 64 GB */ + *journal_size = 65536; /* 256 MB */ + else if (num_blocks < 32768*1024) /* 128 GB */ + *journal_size = 131072; /* 512 MB */ + else + *journal_size = 262144; /* 1 GB */ + + *fc_size = ext2fs_has_feature_fast_commit(fs->super) ? + *journal_size / EXT2_JOURNAL_TO_FC_BLKS_RATIO : 0; + assert(*journal_size + *fc_size < num_blocks); + + return 0; } int ext2fs_journal_sb_start(int blocksize) @@ -435,6 +470,7 @@ errcode_t ext2fs_add_journal_device(ext2_filsys fs, ext2_filsys journal_dev) * functions if it is not. */ errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks, + blk_t num_fc_blocks, blk64_t goal, int flags) { errcode_t retval; @@ -486,7 +522,8 @@ errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks, * filesystems is extremely rare these days... Ignore it. */ flags &= ~EXT2_MKJOURNAL_LAZYINIT; - if ((retval = write_journal_file(fs, jfile, num_blocks, flags))) + if ((retval = write_journal_file(fs, jfile, + num_blocks, num_fc_blocks, flags))) goto errout; /* Get inode number of the journal file */ @@ -528,7 +565,8 @@ errcode_t ext2fs_add_journal_inode2(ext2_filsys fs, blk_t num_blocks, } journal_ino = EXT2_JOURNAL_INO; if ((retval = write_journal_inode(fs, journal_ino, - num_blocks, goal, flags))) + num_blocks, num_fc_blocks, + goal, flags))) return retval; } @@ -546,9 +584,10 @@ errout: return retval; } -errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks, int flags) +errcode_t ext2fs_add_journal_inode(ext2_filsys fs, blk_t num_blocks, + blk_t num_fc_blocks, int flags) { - return ext2fs_add_journal_inode2(fs, num_blocks, ~0ULL, flags); + return ext2fs_add_journal_inode2(fs, num_blocks, num_fc_blocks, ~0ULL, flags); } diff --git a/misc/dumpe2fs.c b/misc/dumpe2fs.c index d295ba4d..e24dc4e6 100644 --- a/misc/dumpe2fs.c +++ b/misc/dumpe2fs.c @@ -364,6 +364,7 @@ static void print_inline_journal_information(ext2_filsys fs) errcode_t retval; ext2_ino_t ino = fs->super->s_journal_inum; char buf[1024]; + int flags; if (fs->flags & EXT2_FLAG_IMAGE_FILE) return; @@ -392,7 +393,9 @@ static void print_inline_journal_information(ext2_filsys fs) _("Journal superblock magic number invalid!\n")); exit(1); } - e2p_list_journal_super(stdout, buf, fs->blocksize, 0); + flags = ext2fs_has_feature_fast_commit(fs->super) ? + E2P_LIST_JOURNAL_FLAG_FC : 0; + e2p_list_journal_super(stdout, buf, fs->blocksize, flags); } static void print_journal_information(ext2_filsys fs) @@ -400,6 +403,7 @@ static void print_journal_information(ext2_filsys fs) errcode_t retval; char buf[1024]; journal_superblock_t *jsb; + int flags; /* Get the journal superblock */ if ((retval = io_channel_read_blk64(fs->io, @@ -417,7 +421,9 @@ static void print_journal_information(ext2_filsys fs) _("Couldn't find journal superblock magic numbers")); exit(1); } - e2p_list_journal_super(stdout, buf, fs->blocksize, 0); + flags = ext2fs_has_feature_fast_commit(fs->super) ? + E2P_LIST_JOURNAL_FLAG_FC : 0; + e2p_list_journal_super(stdout, buf, fs->blocksize, flags); } static int check_mmp(ext2_filsys fs) diff --git a/misc/mke2fs.c b/misc/mke2fs.c index 8c8f5ea4..2d9a5449 100644 --- a/misc/mke2fs.c +++ b/misc/mke2fs.c @@ -93,6 +93,7 @@ static uid_t root_uid; static gid_t root_gid; int journal_size; int journal_flags; +int journal_fc_size; static int lazy_itable_init; static int packed_meta_blocks; int no_copy_xattrs; @@ -604,9 +605,18 @@ static void create_journal_dev(ext2_filsys fs) char *buf; blk64_t blk, err_blk; int c, count, err_count; + int num_journal_blks = 0, num_fc_blks = 0; + + retval = ext2fs_split_journal_size(fs, &num_journal_blks, &num_fc_blks, + ext2fs_blocks_count(fs->super)); + if (retval) { + com_err("create_journal_dev", retval, "%s", + _("while splitting the journal size")); + exit(1); + } retval = ext2fs_create_journal_superblock(fs, - ext2fs_blocks_count(fs->super), 0, &buf); + num_journal_blks, num_fc_blks, 0, &buf); if (retval) { com_err("create_journal_dev", retval, "%s", _("while initializing journal superblock")); @@ -1753,6 +1763,8 @@ profile_error: case 'j': if (!journal_size) journal_size = -1; + if (!journal_fc_size) + journal_fc_size = -1; break; case 'J': parse_journal_opts(optarg); @@ -2937,7 +2949,7 @@ int main (int argc, char *argv[]) badblocks_list bb_list = 0; badblocks_iterate bb_iter; blk_t blk; - unsigned int journal_blocks = 0; + blk_t journal_blocks = 0, journal_fc_blocks = 0; unsigned int i, checkinterval; int max_mnt_count; int val, hash_alg; @@ -3047,7 +3059,8 @@ int main (int argc, char *argv[]) /* Calculate journal blocks */ if (!journal_device && ((journal_size) || ext2fs_has_feature_journal(&fs_param))) - journal_blocks = figure_journal_size(journal_size, fs); + figure_journal_size(&journal_blocks, &journal_fc_blocks, + journal_size, journal_fc_size, fs); sprintf(opt_string, "tdb_data_size=%d", fs->blocksize <= 4096 ? 32768 : fs->blocksize * 8); @@ -3382,7 +3395,7 @@ int main (int argc, char *argv[]) free(journal_device); } else if ((journal_size) || ext2fs_has_feature_journal(&fs_param)) { - overhead += EXT2FS_NUM_B2C(fs, journal_blocks); + overhead += EXT2FS_NUM_B2C(fs, journal_blocks + journal_fc_blocks); if (super_only) { printf("%s", _("Skipping journal creation in super-only mode\n")); fs->super->s_journal_inum = EXT2_JOURNAL_INO; @@ -3395,10 +3408,11 @@ int main (int argc, char *argv[]) } if (!quiet) { printf(_("Creating journal (%u blocks): "), - journal_blocks); + journal_blocks + journal_fc_blocks); fflush(stdout); } retval = ext2fs_add_journal_inode2(fs, journal_blocks, + journal_fc_blocks, journal_location, journal_flags); if (retval) { diff --git a/misc/tune2fs.c b/misc/tune2fs.c index 670ed9e0..daee154d 100644 --- a/misc/tune2fs.c +++ b/misc/tune2fs.c @@ -102,7 +102,7 @@ static int feature_64bit; static int fsck_requested; static char *undo_file; -int journal_size, journal_flags; +int journal_size, journal_fc_size, journal_flags; char *journal_device; static blk64_t journal_location = ~0LL; @@ -1543,7 +1543,7 @@ mmp_error: */ static int add_journal(ext2_filsys fs) { - unsigned long journal_blocks; + blk_t journal_blocks, journal_fc_blocks; errcode_t retval; ext2_filsys jfs; io_manager io_ptr; @@ -1589,13 +1589,15 @@ static int add_journal(ext2_filsys fs) } else if (journal_size) { fputs(_("Creating journal inode: "), stdout); fflush(stdout); - journal_blocks = figure_journal_size(journal_size, fs); + figure_journal_size(&journal_blocks, &journal_fc_blocks, + journal_size, journal_fc_size, fs); if (journal_location_string) journal_location = parse_num_blocks2(journal_location_string, fs->super->s_log_block_size); retval = ext2fs_add_journal_inode2(fs, journal_blocks, + journal_fc_blocks, journal_location, journal_flags); if (retval) { diff --git a/misc/util.c b/misc/util.c index dcd2f0a7..85f10a70 100644 --- a/misc/util.c +++ b/misc/util.c @@ -200,6 +200,14 @@ void parse_journal_opts(const char *opts) journal_size = strtoul(arg, &p, 0); if (*p) journal_usage++; + } else if (strcmp(token, "fast_commit_size") == 0) { + if (!arg) { + journal_usage++; + continue; + } + journal_fc_size = strtoul(arg, &p, 0); + if (*p) + journal_usage++; } else if (!strcmp(token, "location")) { if (!arg) { journal_usage++; @@ -229,42 +237,65 @@ void parse_journal_opts(const char *opts) free(buf); } +static inline int jsize_to_blks(ext2_filsys fs, int size) +{ + return (size * 1024) / (fs->blocksize / 1024); +} + +/* Fast commit size is in KBs */ +static inline int fcsize_to_blks(ext2_filsys fs, int size) +{ + return (size * 1024) / (fs->blocksize); +} + /* * Determine the number of journal blocks to use, either via * user-specified # of megabytes, or via some intelligently selected * defaults. * - * Find a reasonable journal file size (in blocks) given the number of blocks - * in the filesystem. For very small filesystems, it is not reasonable to - * have a journal that fills more than half of the filesystem. + * Find a reasonable journal file size (in blocks) given the number of blocks in + * the filesystem. For very small filesystems, it is not reasonable to have a + * journal that fills more than half of the filesystem. */ -unsigned int figure_journal_size(int size, ext2_filsys fs) +void figure_journal_size(blk_t *j_blocks, blk_t *fc_blocks, + int requested_j_size, int requested_fc_size, ext2_filsys fs) { - int j_blocks; + int def_j_blocks, def_fc_blocks, total_blocks; + int ret; - j_blocks = ext2fs_default_journal_size(ext2fs_blocks_count(fs->super)); - if (j_blocks < 0) { + ret = ext2fs_default_journal_size(&def_j_blocks, &def_fc_blocks, fs); + if (ret) { fputs(_("\nFilesystem too small for a journal\n"), stderr); - return 0; + return; } - if (size > 0) { - j_blocks = size * 1024 / (fs->blocksize / 1024); - if (j_blocks < 1024 || j_blocks > 10240000) { - fprintf(stderr, _("\nThe requested journal " + *j_blocks = def_j_blocks; + *fc_blocks = def_fc_blocks; + + if (requested_j_size > 0 || + (ext2fs_has_feature_fast_commit(fs->super) && requested_fc_size > 0)) { + *j_blocks = requested_j_size > 0 ? + jsize_to_blks(fs, requested_j_size) : def_j_blocks; + + *fc_blocks = 0; + if (ext2fs_has_feature_fast_commit(fs->super)) + *fc_blocks = requested_fc_size > 0 ? + fcsize_to_blks(fs, requested_fc_size) : def_fc_blocks; + total_blocks = *j_blocks + *fc_blocks; + if (total_blocks < 1024 || total_blocks > 10240000) { + fprintf(stderr, _("\nThe total requested journal " "size is %d blocks; it must be\n" "between 1024 and 10240000 blocks. " "Aborting.\n"), - j_blocks); + total_blocks); exit(1); } - if ((unsigned) j_blocks > ext2fs_free_blocks_count(fs->super) / 2) { - fputs(_("\nJournal size too big for filesystem.\n"), + if ((unsigned int) total_blocks > ext2fs_free_blocks_count(fs->super) / 2) { + fputs(_("\nTotal journal size too big for filesystem.\n"), stderr); exit(1); } } - return j_blocks; } void print_check_message(int mnt, unsigned int check) diff --git a/misc/util.h b/misc/util.h index 49b4b9c1..cee812b4 100644 --- a/misc/util.h +++ b/misc/util.h @@ -11,6 +11,7 @@ */ extern int journal_size; +extern int journal_fc_size; extern int journal_flags; extern char *journal_device; extern char *journal_location_string; @@ -22,6 +23,7 @@ extern char *get_progname(char *argv_zero); extern void proceed_question(int delay); extern void parse_journal_opts(const char *opts); extern void check_mount(const char *device, int force, const char *type); -extern unsigned int figure_journal_size(int size, ext2_filsys fs); +extern void figure_journal_size(blk_t *j_blocks, blk_t *fc_blocks, + int requested_j_size, int requested_fc_size, ext2_filsys fs); extern void print_check_message(int, unsigned int); extern void dump_mmp_msg(struct mmp_struct *mmp, const char *msg); From patchwork Fri Nov 20 19:15:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404065 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=cVNW2fRH; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rG5Dkpz9sVH for ; Sat, 21 Nov 2020 06:16:30 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729454AbgKTTQa (ORCPT ); Fri, 20 Nov 2020 14:16:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729444AbgKTTQa (ORCPT ); Fri, 20 Nov 2020 14:16:30 -0500 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0360CC0613CF for ; Fri, 20 Nov 2020 11:16:30 -0800 (PST) Received: by mail-pf1-x441.google.com with SMTP id t8so8777894pfg.8 for ; Fri, 20 Nov 2020 11:16:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7sMkFu3WbDnH80Glnn82gYd+1da6NUT/aw21v/s39Mc=; b=cVNW2fRH9HCrImPkadX9huYk8LpaLTVzTe/UzGNK2PHCs84/NizEp0cUhQEtsojI5J iv1Ey5YBc8XTx5VxOco8oGtYQO1GWXtg5RCoQF4AG1tdQOjxRQsL6FgRSZZop0H2ixos EB9jACqW1bN5sqeIL9hEd4GDJpb42LHpLO65dQ++AA/CGn3YwoB3z7hsecEQylsp677v 3s9Qnprm+gqAeERDpAQ39NGBcB9F3Q0sE93d0JI+SSTcuLvU7pDHdTLqiLpoWw4/ypPL Ot+fbXSoZ+qbQ7EK0jJqH6j2DB+JmyqzyrGSMocWm+khhiihkNZsGNF4wmGz537u8GCR 99Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7sMkFu3WbDnH80Glnn82gYd+1da6NUT/aw21v/s39Mc=; b=Ru/VxUrZsLFQ2eP+N+amG4vHEyDMghhKlfJ1ep1qqhQPy6V7IJX5E+JF8teIS5lH5V GnDLXV8+5/eClTo06uZI2YVtaUa+bT7eq9gpuCAWvqCZlM0qwW8DBFvLoGwF1jgliphw bZ//+XqHssEChG5G1ep5dlcxYvFM2ZSvtRd2VLml8AefTlOU+WIH517feteUwYFU/J59 teGvV5OlqINL4NxeIL9RlGNScD1Zuk9NMn2zppT0rDuH6rDjkZFnPwVxGX2WiuBiThSx PLEvKB0/ic0J1lSwQW/05dNaiq18q7Q4yxoekTDcOKG9DChU23sdLedhcgLOxXEvZKFG HmUA== X-Gm-Message-State: AOAM531eU+ZpgAdW9IvkMCMoE2nNS6nO+emWGRBFe6NkRGC9pJP5WmaI Kt2Pj2tYNqyUm6hrD1HE9a9uu1YkVeg= X-Google-Smtp-Source: ABdhPJwJ4m6CyOwUTqLu/xYCWIz2om87Mw+75Hx2QApnfd02w1lV3L5Nvg7UUeTNy9Wu/TTHToia4Q== X-Received: by 2002:a17:90b:fd1:: with SMTP id gd17mr7419124pjb.148.1605899789086; Fri, 20 Nov 2020 11:16:29 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:28 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 05/15] mke2fs, tune2fs: update man page with fast commit info Date: Fri, 20 Nov 2020 11:15:56 -0800 Message-Id: <20201120191606.2224881-6-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org This patch adds information about fast commit feature in mke2fs and tune2fs man pages. Signed-off-by: Harshad Shirwadkar --- misc/mke2fs.8.in | 21 +++++++++++++++++++++ misc/tune2fs.8.in | 25 +++++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/misc/mke2fs.8.in b/misc/mke2fs.8.in index e6bfc6d6..2833b408 100644 --- a/misc/mke2fs.8.in +++ b/misc/mke2fs.8.in @@ -521,6 +521,27 @@ The size of the journal must be at least 1024 filesystem blocks and may be no more than 10,240,000 filesystem blocks or half the total file system size (whichever is smaller) .TP +.BI fast_commit_size= fast-commit-size +Create an additional fast commit journal area of size +.I fast-commit-size +kilobytes. +This option is only valid if +.B fast_commit +feature is enabled +on the file system. If this option is not specified and if +.B fast_commit +feature is turned on, fast commit area size defaults to +.I journal-size +/ 64 megabytes. The total size of the journal with +.B fast_commit +feature set is +.I journal-size ++ ( +.I fast-commit-size +* 1024) megabytes. The total journal size may be no more than +10,240,000 filesystem blocks or half the total file system size +(whichever is smaller). +.TP .BI location =journal-location Specify the location of the journal. The argument .I journal-location diff --git a/misc/tune2fs.8.in b/misc/tune2fs.8.in index 582d1da5..2114c623 100644 --- a/misc/tune2fs.8.in +++ b/misc/tune2fs.8.in @@ -357,6 +357,27 @@ and may be no more than 10,240,000 filesystem blocks. There must be enough free space in the filesystem to create a journal of that size. .TP +.BI fast_commit_size= fast-commit-size +Create an additional fast commit journal area of size +.I fast-commit-size +kilobytes. +This option is only valid if +.B fast_commit +feature is enabled +on the file system. If this option is not specified and if +.B fast_commit +feature is turned on, fast commit area size defaults to +.I journal-size +/ 64 megabytes. The total size of the journal with +.B fast_commit +feature set is +.I journal-size ++ ( +.I fast-commit-size +* 1024) megabytes. The total journal size may be no more than +10,240,000 filesystem blocks or half the total file system size +(whichever is smaller). +.TP .BI location =journal-location Specify the location of the journal. The argument .I journal-location @@ -586,6 +607,10 @@ Setting the filesystem feature is equivalent to using the .B \-j option. .TP +.TP +.B fast_commit +Enable fast commit journaling feature to improve fsync latency. +.TP .B large_dir Increase the limit on the number of files per directory. .B Tune2fs From patchwork Fri Nov 20 19:15:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404066 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=KSs3yC7/; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rK0sJwz9sTR for ; Sat, 21 Nov 2020 06:16:33 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729779AbgKTTQc (ORCPT ); Fri, 20 Nov 2020 14:16:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729200AbgKTTQb (ORCPT ); Fri, 20 Nov 2020 14:16:31 -0500 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61BBFC0613CF for ; Fri, 20 Nov 2020 11:16:31 -0800 (PST) Received: by mail-pf1-x443.google.com with SMTP id t8so8777947pfg.8 for ; Fri, 20 Nov 2020 11:16:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IzPvEcwen9tCzLMhzmMye/H24Zg7QoAC4q5q+4mVfC8=; b=KSs3yC7/3o9o6PF8qS6wd8wUjlbB+n6LQoFXCg5+6NaT88Sx8CR7HrjehOUsepnS6L lF5LVIVmO18hB+X5PN9bbmj+HEMQ0qQkNSphJckrQ4cPI59G/OpZj+apUeTb1G3nQX5C L+oqkab2fblmgazpJqZ9HKbiBZnh1fKHwHcf62UQaDDiIrnqkANiqRkUVTCJxZVPZDE+ /jOdg0k8niRG1vp0nnbcpReLb4cBixTP8D0w0FB46uwOVhnws8MaYv9vPK4CuxH5R2xh jWh05GXRIKEL1XYskQWpMtU/Z61ibpB3+NDTvKEsUcuA26tuKLyIvmgxcLnAuC/1096l lBFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IzPvEcwen9tCzLMhzmMye/H24Zg7QoAC4q5q+4mVfC8=; b=bcShqQHPKYcKVVOIojGS4v9wwjZvSetvJP9BYrnUe+DY8gG30vMJ0UcaKpgzRihgC1 CrXmM5nSRv/BJV+7K/6WP0lAwkLCXaOsWbI0RhNHYG0Ia6wNXNpKqlGsLD5d6n2MSE3D 1X1uXtvSdyzm8wfjoebCA57XlAd5RmZdcBQNB44EUt9OlpmAJwYmDHL7UFu2ARZ6b0q6 LeQrQUOlguDrlG+p7g/UNaMlQIldTFKCaaRP6ixAFGC4V5/+S99/8ltD0InYmBb95jyH d2joQZjRFPEKRmcpkipdT1nE8jyLxqojfgd0TCiIe3oBODwujrECaBtgPGRqC1nR2xux P1HQ== X-Gm-Message-State: AOAM533xSEBOg5lpn5tjdgfXcNBPWFo4mx8V6ibCkfmE8gxHY6p8PJnI myOVqolK1Qo8EDXIhUm9L94EitA8Umw= X-Google-Smtp-Source: ABdhPJzNDy5bHimAwHRirmAychzu5eVTPRI9H8NNTDiBNd2v+Vyrkczk38UNjm8gKLir7gLQmtT0CQ== X-Received: by 2002:a17:90a:14e5:: with SMTP id k92mr11691739pja.169.1605899790478; Fri, 20 Nov 2020 11:16:30 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:29 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 06/15] ext2fs: add new APIs needed for fast commits Date: Fri, 20 Nov 2020 11:15:57 -0800 Message-Id: <20201120191606.2224881-7-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org This patch adds the following new APIs: Count the total number of blocks occupied by inode including intermediate extent tree nodes. extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino, struct ext2_inode *inode); Convert ext3_extent to ext2fs_extent. extern void ext2fs_convert_extent(struct ext2fs_extent *to, struct ext3_extent *from); Signed-off-by: Harshad Shirwadkar --- lib/ext2fs/ext2fs.h | 4 ++++ lib/ext2fs/extent.c | 56 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 01132245..afa9c5e4 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -1332,6 +1332,10 @@ extern errcode_t ext2fs_extent_fix_parents(ext2_extent_handle_t handle); extern size_t ext2fs_max_extent_depth(ext2_extent_handle_t handle); extern errcode_t ext2fs_fix_extents_checksums(ext2_filsys fs, ext2_ino_t ino, struct ext2_inode *inode); +extern blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino, + struct ext2_inode *inode); +extern void ext2fs_convert_extent(struct ext2fs_extent *to, + struct ext3_extent *from); /* fallocate.c */ #define EXT2_FALLOCATE_ZERO_BLOCKS (0x1) diff --git a/lib/ext2fs/extent.c b/lib/ext2fs/extent.c index ac3dbfec..43feea0a 100644 --- a/lib/ext2fs/extent.c +++ b/lib/ext2fs/extent.c @@ -1785,6 +1785,62 @@ out: return errcode; } +void ext2fs_convert_extent(struct ext2fs_extent *to, struct ext3_extent *from) +{ + to->e_pblk = ext2fs_le32_to_cpu(from->ee_start) + + ((__u64) ext2fs_le16_to_cpu(from->ee_start_hi) + << 32); + to->e_lblk = ext2fs_le32_to_cpu(from->ee_block); + to->e_len = ext2fs_le16_to_cpu(from->ee_len); + to->e_flags |= EXT2_EXTENT_FLAGS_LEAF; + if (to->e_len > EXT_INIT_MAX_LEN) { + to->e_len -= EXT_INIT_MAX_LEN; + to->e_flags |= EXT2_EXTENT_FLAGS_UNINIT; + } +} + +blk64_t ext2fs_count_blocks(ext2_filsys fs, ext2_ino_t ino, + struct ext2_inode *inode) +{ + ext2_extent_handle_t handle; + struct ext2fs_extent extent; + errcode_t errcode; + int i; + blk64_t blkcount = 0; + blk64_t *intermediate_nodes; + + errcode = ext2fs_extent_open2(fs, ino, inode, &handle); + if (errcode) + goto out; + + errcode = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent); + if (errcode) + goto out; + + ext2fs_get_array(handle->max_depth, sizeof(blk64_t), + &intermediate_nodes); + blkcount = handle->level; + while (!errcode) { + if (extent.e_flags & EXT2_EXTENT_FLAGS_LEAF) { + blkcount += extent.e_len; + for (i = 0; i < handle->level; i++) { + if (intermediate_nodes[i] != + handle->path[i].end_blk) { + blkcount++; + intermediate_nodes[i] = + handle->path[i].end_blk; + } + } + } + errcode = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent); + } + ext2fs_free_mem(&intermediate_nodes); +out: + ext2fs_extent_free(handle); + + return blkcount; +} + #ifdef DEBUG /* * Override debugfs's prompt From patchwork Fri Nov 20 19:15:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404067 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=etyolcxi; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rL4vbhz9sTR for ; Sat, 21 Nov 2020 06:16:34 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729863AbgKTTQe (ORCPT ); Fri, 20 Nov 2020 14:16:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729200AbgKTTQd (ORCPT ); Fri, 20 Nov 2020 14:16:33 -0500 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0934C0613CF for ; Fri, 20 Nov 2020 11:16:32 -0800 (PST) Received: by mail-pf1-x441.google.com with SMTP id 10so8803719pfp.5 for ; Fri, 20 Nov 2020 11:16:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=imVvebcLHZpzkqJeJwhdXd08sKDydHyQ8z8yAZ6eL24=; b=etyolcxi+ZIX7Xk0nlQGVks6BhNAhOU1Tz70XQciYvvKSWqZB3FLf5M63Yf/wY3WQ4 Wel2495YGXhLNTy/QqwC8SynpB4I0//awMDEY/Mtf0CmxVUrxzQYqn+uBfNg4dRZJ+FT Xa6fqF1dXvkAvlmsUB4RVfrkbHZLqbZKLi0zqglCqqlNB0uYKdZH45VP6bmnLJSGDwmv kB2SCnz7hFbi8qfoL4JEtQbMH4YjFrR1kTDP2mw9mlFpYF+/9fJ9/DPRrEt21si2JHi3 PJkERXbYMNs6BysWRF++D38JSdfH/Xvfnj0w/M+ePC+LAKrLswYaCrjw22ftU+PS/0Ht O4yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=imVvebcLHZpzkqJeJwhdXd08sKDydHyQ8z8yAZ6eL24=; b=DFRHNSOBoIQalMrtSBxvo+21lfJkMl///NfyamD8Tc3tg51miezMlM7JiUUmFBaLpU 4iDbbDAnUGWWLrvTKcTVnP0B2/PnYt9LZry+MHookLLbNtqzKyNsBUk6PvqKc+5xqkGv 6OYwft6jdEn5GHiROkE33rqarLIekiZFGw1XlEBxjidTpDYLh3r5//oQecFzcJO9uBM/ /llGmM5m55p3cCEERbJ7ZhAtv4ikQUYwkK5KZ5tHfrEG5Ou0h4UgLy7PFPqq5+2hyO63 4Asz6G28ZipFTsBe24YXCPxjh5nx/PXnjF8SFeA5M8UfjIC3uruEFT4tXoTvZW3OINar uCfQ== X-Gm-Message-State: AOAM533qFcD9tDEtgMMIWi1VIbsCeBzK7+ghC1h2W8KfFV1N6FSD43Jb hjiwSGVxrl+mCgYuqXNDmEOK9D8aMWw= X-Google-Smtp-Source: ABdhPJxDkIMu7WX+ewxLjoFeJOLQ4+lGX9rbunLLFs5c+vmBJHrwjFaPhSBJe0mPrkLyBwZkKK1OZg== X-Received: by 2002:a17:90b:e86:: with SMTP id fv6mr12396396pjb.196.1605899791920; Fri, 20 Nov 2020 11:16:31 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:30 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 07/15] e2fsck: add function to rewrite extent tree Date: Fri, 20 Nov 2020 11:15:58 -0800 Message-Id: <20201120191606.2224881-8-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Fast commit replay needs to rewrite the entire extent tree for inodes found in fast commit area. This patch makes e2fsck's rewrite extent tree path visible. Signed-off-by: Harshad Shirwadkar Reviewed-by: Theodore Ts'o --- e2fsck/e2fsck.h | 16 +++++ e2fsck/extents.c | 168 ++++++++++++++++++++++++++++++----------------- 2 files changed, 124 insertions(+), 60 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 85f953b2..3b9c1874 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -226,6 +226,19 @@ typedef struct e2fsck_struct *e2fsck_t; #define MAX_EXTENT_DEPTH_COUNT 5 +/* + * This strucutre is used to manage the list of extents in a file. Placing + * it here since this is used by fast_commit.h. + */ +struct extent_list { + blk64_t blocks_freed; + struct ext2fs_extent *extents; + unsigned int count; + unsigned int size; + unsigned int ext_read; + errcode_t retval; + ext2_ino_t ino; +}; struct e2fsck_struct { ext2_filsys fs; const char *program_name; @@ -536,6 +549,9 @@ errcode_t e2fsck_should_rebuild_extents(e2fsck_t ctx, struct problem_context *pctx, struct extent_tree_info *eti, struct ext2_extent_info *info); +errcode_t e2fsck_read_extents(e2fsck_t ctx, struct extent_list *extents); +errcode_t e2fsck_rewrite_extent_tree(e2fsck_t ctx, + struct extent_list *extents); /* journal.c */ extern errcode_t e2fsck_check_ext3_journal(e2fsck_t ctx); diff --git a/e2fsck/extents.c b/e2fsck/extents.c index e9139326..d6c74834 100644 --- a/e2fsck/extents.c +++ b/e2fsck/extents.c @@ -58,16 +58,6 @@ int e2fsck_ino_will_be_rebuilt(e2fsck_t ctx, ext2_ino_t ino) return ext2fs_test_inode_bitmap2(ctx->inodes_to_rebuild, ino); } -struct extent_list { - blk64_t blocks_freed; - struct ext2fs_extent *extents; - unsigned int count; - unsigned int size; - unsigned int ext_read; - errcode_t retval; - ext2_ino_t ino; -}; - static errcode_t load_extents(e2fsck_t ctx, struct extent_list *list) { ext2_filsys fs = ctx->fs; @@ -206,66 +196,40 @@ static int find_blocks(ext2_filsys fs, blk64_t *blocknr, e2_blkcnt_t blockcnt, return 0; } -static errcode_t rebuild_extent_tree(e2fsck_t ctx, struct extent_list *list, - ext2_ino_t ino) +errcode_t rewrite_extent_replay(e2fsck_t ctx, struct extent_list *list, + struct ext2_inode_large *inode) { - struct ext2_inode_large inode; errcode_t retval; ext2_extent_handle_t handle; unsigned int i, ext_written; struct ext2fs_extent *ex, extent; - blk64_t start_val, delta; - - list->count = 0; - list->blocks_freed = 0; - list->ino = ino; - list->ext_read = 0; - e2fsck_read_inode_full(ctx, ino, EXT2_INODE(&inode), sizeof(inode), - "rebuild_extents"); + blk64_t start_val, delta, blkcount; - /* Skip deleted inodes and inline data files */ - if (inode.i_links_count == 0 || - inode.i_flags & EXT4_INLINE_DATA_FL) - return 0; - - /* Collect lblk->pblk mappings */ - if (inode.i_flags & EXT4_EXTENTS_FL) { - retval = load_extents(ctx, list); - if (retval) - goto err; - goto extents_loaded; - } - - retval = ext2fs_block_iterate3(ctx->fs, ino, BLOCK_FLAG_READ_ONLY, 0, - find_blocks, list); - if (retval) - goto err; - if (list->retval) { - retval = list->retval; - goto err; - } - -extents_loaded: /* Reset extent tree */ - inode.i_flags &= ~EXT4_EXTENTS_FL; - memset(inode.i_block, 0, sizeof(inode.i_block)); + inode->i_flags &= ~EXT4_EXTENTS_FL; + memset(inode->i_block, 0, sizeof(inode->i_block)); /* Make a note of freed blocks */ - quota_data_sub(ctx->qctx, &inode, ino, + quota_data_sub(ctx->qctx, inode, list->ino, list->blocks_freed * ctx->fs->blocksize); - retval = ext2fs_iblk_sub_blocks(ctx->fs, EXT2_INODE(&inode), + retval = ext2fs_iblk_sub_blocks(ctx->fs, EXT2_INODE(inode), list->blocks_freed); if (retval) - goto err; + return retval; /* Now stuff extents into the file */ - retval = ext2fs_extent_open2(ctx->fs, ino, EXT2_INODE(&inode), &handle); + retval = ext2fs_extent_open2(ctx->fs, list->ino, EXT2_INODE(inode), + &handle); if (retval) - goto err; + return retval; ext_written = 0; - start_val = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(&inode)); + + start_val = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(inode)); + for (i = 0, ex = list->extents; i < list->count; i++, ex++) { + if (ex->e_len == 0) + continue; memcpy(&extent, ex, sizeof(struct ext2fs_extent)); extent.e_flags &= EXT2_EXTENT_FLAGS_UNINIT; if (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) { @@ -289,36 +253,120 @@ extents_loaded: } #ifdef DEBUG - printf("W: ino=%d pblk=%llu lblk=%llu len=%u\n", ino, + printf("W: ino=%d pblk=%llu lblk=%llu len=%u\n", list->ino, extent.e_pblk, extent.e_lblk, extent.e_len); #endif retval = ext2fs_extent_insert(handle, EXT2_EXTENT_INSERT_AFTER, &extent); if (retval) - goto err2; + goto err; retval = ext2fs_extent_fix_parents(handle); if (retval) - goto err2; + goto err; ext_written++; } - delta = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(&inode)) - + delta = ext2fs_get_stat_i_blocks(ctx->fs, EXT2_INODE(inode)) - start_val; if (delta) - quota_data_add(ctx->qctx, &inode, ino, delta << 9); + quota_data_add(ctx->qctx, inode, list->ino, delta << 9); #if defined(DEBUG) || defined(DEBUG_SUMMARY) printf("rebuild: ino=%d extents=%d->%d\n", ino, list->ext_read, ext_written); #endif - e2fsck_write_inode(ctx, ino, EXT2_INODE(&inode), "rebuild_extents"); + e2fsck_write_inode(ctx, list->ino, EXT2_INODE(inode), + "rebuild_extents"); -err2: - ext2fs_extent_free(handle); err: + ext2fs_extent_free(handle); + return retval; +} + +errcode_t e2fsck_rewrite_extent_tree(e2fsck_t ctx, struct extent_list *list) +{ + struct ext2_inode_large inode; + int ret; + + memset(&inode, 0, sizeof(inode)); + ext2fs_read_inode_full(ctx->fs, list->ino, EXT2_INODE(&inode), + sizeof(inode)); + + /* Skip deleted inodes and inline data files */ + if (inode.i_flags & EXT4_INLINE_DATA_FL) + return 0; + + ret = rewrite_extent_replay(ctx, list, &inode); + ext2fs_iblk_set(ctx->fs, EXT2_INODE(&inode), + ext2fs_count_blocks(ctx->fs, list->ino, EXT2_INODE(&inode))); + ext2fs_write_inode_full(ctx->fs, list->ino, EXT2_INODE(&inode), + sizeof(inode)); + + return ret; +} + +errcode_t e2fsck_read_extents(e2fsck_t ctx, struct extent_list *extents) +{ + struct ext2_inode_large inode; + errcode_t retval; + + extents->extents = NULL; + extents->count = 0; + extents->blocks_freed = 0; + extents->ext_read = 0; + extents->size = NUM_EXTENTS; + retval = ext2fs_get_array(NUM_EXTENTS, sizeof(struct ext2fs_extent), + &extents->extents); + if (retval) + return -ENOMEM; + + retval = ext2fs_read_inode(ctx->fs, extents->ino, EXT2_INODE(&inode)); + if (retval) + goto err_out; + + retval = load_extents(ctx, extents); + if (!retval) + return 0; +err_out: + ext2fs_free_mem(&extents->extents); + extents->size = 0; + extents->count = 0; return retval; } +static errcode_t rebuild_extent_tree(e2fsck_t ctx, struct extent_list *list, + ext2_ino_t ino) +{ + struct ext2_inode_large inode; + errcode_t retval; + + list->count = 0; + list->blocks_freed = 0; + list->ino = ino; + list->ext_read = 0; + e2fsck_read_inode_full(ctx, ino, EXT2_INODE(&inode), sizeof(inode), + "rebuild_extents"); + + /* Skip deleted inodes and inline data files */ + if (inode.i_links_count == 0 || + inode.i_flags & EXT4_INLINE_DATA_FL) + return 0; + + /* Collect lblk->pblk mappings */ + if (inode.i_flags & EXT4_EXTENTS_FL) { + retval = load_extents(ctx, list); + if (retval) + return retval; + return rewrite_extent_replay(ctx, list, &inode); + } + + retval = ext2fs_block_iterate3(ctx->fs, ino, BLOCK_FLAG_READ_ONLY, 0, + find_blocks, list); + + return retval || list->retval || + rewrite_extent_replay(ctx, list, &inode); +} + /* Rebuild the extents immediately */ static errcode_t e2fsck_rebuild_extents(e2fsck_t ctx, ext2_ino_t ino) { From patchwork Fri Nov 20 19:15:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404068 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=PR8k1FYS; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rM3Rpwz9sVH for ; Sat, 21 Nov 2020 06:16:35 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732198AbgKTTQf (ORCPT ); Fri, 20 Nov 2020 14:16:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729200AbgKTTQf (ORCPT ); Fri, 20 Nov 2020 14:16:35 -0500 Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D32B2C0613CF for ; Fri, 20 Nov 2020 11:16:34 -0800 (PST) Received: by mail-pg1-x544.google.com with SMTP id 81so8133916pgf.0 for ; Fri, 20 Nov 2020 11:16:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=97YwAeoYnnUjwl0pFMK4MUgqEapRbp6EZYxfTuGWjNo=; b=PR8k1FYSVxhS0ESk5VQ4NiVlcoQrjWK6Lbocjfc3rTnVPRInps4G+qIoJo8y8a9Q3o itUdVJ4Xr+lC7dyWqhWd4XvrIPrPYe7E/qgVXAhP0r5HBpZ+P0Q21aQcr0JkNNGnGVwy K6iaihHUZoYRevIFK6m115lMkJwPWRzc5UMFO9dGwS9a21s++YifDFdQMibpmZdJ079S +XhlU4keYe8tDPnrN8gEljl17HUgBVRIKQ1bVDDY6LdUlY14Rp2YH4qU0rMIoCcXWIC6 RX4sdUX1DEGNIHPz2divtb2GdlcJcUmWGUO6iuozuUhyG4On048lqmmKZ4JNyxPHOgb4 SZzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=97YwAeoYnnUjwl0pFMK4MUgqEapRbp6EZYxfTuGWjNo=; b=c0W5N7fLGmJRi3cF7lr+HJBCEW2zuMYr874IfkmCHG4+SZjaMWBh1YqWX4Fz25QY3I whqfDWX2nJR4o0n5LzTDtBD4N7xvYA1XZrehOeZFgrHcDVNtCt472qPpBfUgXdjwTejd JtGbiNWhvwVSO8Td6wVpJdrau7nQ4CzHz/cvDCfKq4ovo37ViTRYUV51QwR2yYeXvgvM B53BMF10xRvxMrGd5Lvc8vcvrsR3Ik3+lLuHEZrh08ZBJ2ziRT885rWlXlY0XCxO95ih yQKsR3nDjSKOXmKPrYEc+QsihKQ07Q2NCJyMDeCCtVm5SAlNk4rlyM95+2pbVbQJsc3s pw8w== X-Gm-Message-State: AOAM532CSlc3+VOyRBGPcpsLTJ+8IlC7kG7OhN9JW34Ul47XDnS2M0Bg +4xFSakbqdUCCcAONCmdNJtKjIh6tgI= X-Google-Smtp-Source: ABdhPJyYqRoqw734x5NM2o8tn3qFSdGioik3xcha7hnMgEMjB5DKVTsHDPjlGWkEca8Bnej54l7cuw== X-Received: by 2002:a63:194d:: with SMTP id 13mr18899849pgz.317.1605899793290; Fri, 20 Nov 2020 11:16:33 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:32 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 08/15] e2fsck: add fast commit setup code Date: Fri, 20 Nov 2020 11:15:59 -0800 Message-Id: <20201120191606.2224881-9-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Add fast_commit.h that contains the necessary helpers needed for fast commit replay. Note that this file is also byte by byte identical with kernel's fast_commit.h. Also, we introduce the "e2fsck_fc_replay_state" structure which is needed for ext4 fast commit replay. Signed-off-by: Harshad Shirwadkar Reviewed-by: Theodore Ts'o --- e2fsck/e2fsck.h | 16 ++++ e2fsck/journal.c | 15 +++ lib/ext2fs/ext2_fs.h | 1 + lib/ext2fs/ext2fs.h | 3 + lib/ext2fs/fast_commit.h | 201 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 236 insertions(+) create mode 100644 lib/ext2fs/fast_commit.h diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 3b9c1874..f75cc343 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -68,6 +68,7 @@ #endif #include "support/quotaio.h" +#include "ext2fs/fast_commit.h" /* * Exit codes used by fsck-type programs @@ -239,6 +240,18 @@ struct extent_list { errcode_t retval; ext2_ino_t ino; }; + +/* State structure for fast commit replay */ +struct e2fsck_fc_replay_state { + struct extent_list fc_extent_list; + int fc_replay_num_tags; + int fc_replay_expected_off; + int fc_current_pass; + int fc_cur_tag; + int fc_crc; + __u16 fc_super_state; +}; + struct e2fsck_struct { ext2_filsys fs; const char *program_name; @@ -431,6 +444,9 @@ struct e2fsck_struct { /* Undo file */ char *undo_file; + + /* Fast commit replay state */ + struct e2fsck_fc_replay_state fc_replay_state; }; /* Data structures to evaluate whether an extent tree needs rebuilding. */ diff --git a/e2fsck/journal.c b/e2fsck/journal.c index 75fefcde..2c8e3441 100644 --- a/e2fsck/journal.c +++ b/e2fsck/journal.c @@ -278,6 +278,17 @@ static int process_journal_block(ext2_filsys fs, return 0; } +/* + * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE + * to indicate that it is expecting more fast commit blocks. It returns + * JBD2_FC_REPLAY_STOP to indicate that replay is done. + */ +static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh, + enum passtype pass, int off, tid_t expected_tid) +{ + return JBD2_FC_REPLAY_STOP; +} + static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal) { struct process_block_struct pb; @@ -514,6 +525,10 @@ static errcode_t e2fsck_get_journal(e2fsck_t ctx, journal_t **ret_journal) journal->j_sb_buffer = bh; journal->j_superblock = (journal_superblock_t *)bh->b_data; + if (ext2fs_has_feature_fast_commit(ctx->fs->super)) + journal->j_fc_replay_callback = ext4_fc_replay; + else + journal->j_fc_replay_callback = NULL; #ifdef USE_INODE_IO if (j_inode) diff --git a/lib/ext2fs/ext2_fs.h b/lib/ext2fs/ext2_fs.h index bfc30c29..b1e4329c 100644 --- a/lib/ext2fs/ext2_fs.h +++ b/lib/ext2fs/ext2_fs.h @@ -543,6 +543,7 @@ struct ext2_inode *EXT2_INODE(struct ext2_inode_large *large_inode) #define EXT2_VALID_FS 0x0001 /* Unmounted cleanly */ #define EXT2_ERROR_FS 0x0002 /* Errors detected */ #define EXT3_ORPHAN_FS 0x0004 /* Orphans being recovered */ +#define EXT4_FC_REPLAY 0x0020 /* Ext4 fast commit replay ongoing */ /* * Misc. filesystem flags diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index afa9c5e4..bde369ef 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -2141,4 +2141,7 @@ static inline unsigned int ext2_dir_htree_level(ext2_filsys fs) } #endif +/* Commonly used helpers */ +#define max(a, b) ((a) > (b) ? (a) : (b)) + #endif /* _EXT2FS_EXT2FS_H */ diff --git a/lib/ext2fs/fast_commit.h b/lib/ext2fs/fast_commit.h new file mode 100644 index 00000000..b77f70f5 --- /dev/null +++ b/lib/ext2fs/fast_commit.h @@ -0,0 +1,201 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __FAST_COMMIT_H__ +#define __FAST_COMMIT_H__ + +/* + * Note this file is present in e2fsprogs/lib/ext2fs/fast_commit.h and + * linux/fs/ext4/fast_commit.h. These file should always be byte identical. + */ + +/* Fast commit tags */ +#define EXT4_FC_TAG_ADD_RANGE 0x0001 +#define EXT4_FC_TAG_DEL_RANGE 0x0002 +#define EXT4_FC_TAG_CREAT 0x0003 +#define EXT4_FC_TAG_LINK 0x0004 +#define EXT4_FC_TAG_UNLINK 0x0005 +#define EXT4_FC_TAG_INODE 0x0006 +#define EXT4_FC_TAG_PAD 0x0007 +#define EXT4_FC_TAG_TAIL 0x0008 +#define EXT4_FC_TAG_HEAD 0x0009 + +#define EXT4_FC_SUPPORTED_FEATURES 0x0 + +/* On disk fast commit tlv value structures */ + +/* Fast commit on disk tag length structure */ +struct ext4_fc_tl { + __le16 fc_tag; + __le16 fc_len; +}; + +/* Value structure for tag EXT4_FC_TAG_HEAD. */ +struct ext4_fc_head { + __le32 fc_features; + __le32 fc_tid; +}; + +/* Value structure for EXT4_FC_TAG_ADD_RANGE. */ +struct ext4_fc_add_range { + __le32 fc_ino; + __u8 fc_ex[12]; +}; + +/* Value structure for tag EXT4_FC_TAG_DEL_RANGE. */ +struct ext4_fc_del_range { + __le32 fc_ino; + __le32 fc_lblk; + __le32 fc_len; +}; + +/* + * This is the value structure for tags EXT4_FC_TAG_CREAT, EXT4_FC_TAG_LINK + * and EXT4_FC_TAG_UNLINK. + */ +struct ext4_fc_dentry_info { + __le32 fc_parent_ino; + __le32 fc_ino; + __u8 fc_dname[0]; +}; + +/* Value structure for EXT4_FC_TAG_INODE and EXT4_FC_TAG_INODE_PARTIAL. */ +struct ext4_fc_inode { + __le32 fc_ino; + __u8 fc_raw_inode[0]; +}; + +/* Value structure for tag EXT4_FC_TAG_TAIL. */ +struct ext4_fc_tail { + __le32 fc_tid; + __le32 fc_crc; +}; + +/* + * Fast commit reason codes + */ +enum { + /* + * Commit status codes: + */ + EXT4_FC_REASON_OK = 0, + EXT4_FC_REASON_INELIGIBLE, + EXT4_FC_REASON_ALREADY_COMMITTED, + EXT4_FC_REASON_FC_START_FAILED, + EXT4_FC_REASON_FC_FAILED, + + /* + * Fast commit ineligiblity reasons: + */ + EXT4_FC_REASON_XATTR = 0, + EXT4_FC_REASON_CROSS_RENAME, + EXT4_FC_REASON_JOURNAL_FLAG_CHANGE, + EXT4_FC_REASON_NOMEM, + EXT4_FC_REASON_SWAP_BOOT, + EXT4_FC_REASON_RESIZE, + EXT4_FC_REASON_RENAME_DIR, + EXT4_FC_REASON_FALLOC_RANGE, + EXT4_FC_REASON_INODE_JOURNAL_DATA, + EXT4_FC_COMMIT_FAILED, + EXT4_FC_REASON_MAX +}; + +#ifdef __KERNEL__ +/* + * In memory list of dentry updates that are performed on the file + * system used by fast commit code. + */ +struct ext4_fc_dentry_update { + int fcd_op; /* Type of update create / unlink / link */ + int fcd_parent; /* Parent inode number */ + int fcd_ino; /* Inode number */ + struct qstr fcd_name; /* Dirent name */ + unsigned char fcd_iname[DNAME_INLINE_LEN]; /* Dirent name string */ + struct list_head fcd_list; +}; + +struct ext4_fc_stats { + unsigned int fc_ineligible_reason_count[EXT4_FC_REASON_MAX]; + unsigned long fc_num_commits; + unsigned long fc_ineligible_commits; + unsigned long fc_numblks; +}; + +#define EXT4_FC_REPLAY_REALLOC_INCREMENT 4 + +/* + * Physical block regions added to different inodes due to fast commit + * recovery. These are set during the SCAN phase. During the replay phase, + * our allocator excludes these from its allocation. This ensures that + * we don't accidentally allocating a block that is going to be used by + * another inode. + */ +struct ext4_fc_alloc_region { + ext4_lblk_t lblk; + ext4_fsblk_t pblk; + int ino, len; +}; + +/* + * Fast commit replay state. + */ +struct ext4_fc_replay_state { + int fc_replay_num_tags; + int fc_replay_expected_off; + int fc_current_pass; + int fc_cur_tag; + int fc_crc; + struct ext4_fc_alloc_region *fc_regions; + int fc_regions_size, fc_regions_used, fc_regions_valid; + int *fc_modified_inodes; + int fc_modified_inodes_used, fc_modified_inodes_size; +}; + +#define region_last(__region) (((__region)->lblk) + ((__region)->len) - 1) +#endif + +#define fc_for_each_tl(__start, __end, __tl) \ + for (tl = (struct ext4_fc_tl *)(__start); \ + (__u8 *)tl < (__u8 *)(__end); \ + tl = (struct ext4_fc_tl *)((__u8 *)tl + \ + sizeof(struct ext4_fc_tl) + \ + + le16_to_cpu(tl->fc_len))) + +static inline const char *tag2str(__u16 tag) +{ + switch (tag) { + case EXT4_FC_TAG_LINK: + return "ADD_ENTRY"; + case EXT4_FC_TAG_UNLINK: + return "DEL_ENTRY"; + case EXT4_FC_TAG_ADD_RANGE: + return "ADD_RANGE"; + case EXT4_FC_TAG_CREAT: + return "CREAT_DENTRY"; + case EXT4_FC_TAG_DEL_RANGE: + return "DEL_RANGE"; + case EXT4_FC_TAG_INODE: + return "INODE"; + case EXT4_FC_TAG_PAD: + return "PAD"; + case EXT4_FC_TAG_TAIL: + return "TAIL"; + case EXT4_FC_TAG_HEAD: + return "HEAD"; + default: + return "ERROR"; + } +} + +/* Get length of a particular tlv */ +static inline int ext4_fc_tag_len(struct ext4_fc_tl *tl) +{ + return le16_to_cpu(tl->fc_len); +} + +/* Get a pointer to "value" of a tlv */ +static inline __u8 *ext4_fc_tag_val(struct ext4_fc_tl *tl) +{ + return (__u8 *)tl + sizeof(*tl); +} + +#endif /* __FAST_COMMIT_H__ */ From patchwork Fri Nov 20 19:16:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404070 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=pmn2iyhw; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rP6RjVz9sVH for ; Sat, 21 Nov 2020 06:16:37 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732241AbgKTTQh (ORCPT ); Fri, 20 Nov 2020 14:16:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732200AbgKTTQg (ORCPT ); Fri, 20 Nov 2020 14:16:36 -0500 Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56B84C0613CF for ; Fri, 20 Nov 2020 11:16:36 -0800 (PST) Received: by mail-pf1-x442.google.com with SMTP id q5so8795100pfk.6 for ; Fri, 20 Nov 2020 11:16:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Q8YFlbQ7XWrwAN2Ryud5370lyN2Z5xoaPuyNNSOvt/c=; b=pmn2iyhwp7pIKK5JeMJKkfpJ4/nmwFweQpO4LQs+uLb99O9qiANMXPZtNDLGcGgy4B Zo6fBhWijhrYVsOETHZjSngI8X3jJ0zKsA/IJx++aPBmq5xXBO+/1krjh8hd0z8Q1GS/ tc4RUt/1NDUdC17SK1b7plz9K7pP4uZtddZpudTDETtF6ApMDVF+Tytke63BrOtV8S5J M9ATyxKl8al7dtggle/+ISKxPhWRciXIVSlC3kE3YqiOTa0hLoBOPBMothcj+j0HAQlr RF+i5XpFEbnmMkXmU/y0T6USvmSoKUL3llQCAZz+TDFGbh3YKD1hpvPHQeQxyEGXstBy nxvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q8YFlbQ7XWrwAN2Ryud5370lyN2Z5xoaPuyNNSOvt/c=; b=qGZRIDQxMCoDw0xeQa6huMdyZeOe4vVuEeN3XwQuyMIE/i/MPCUdAaTe/yaN61BiL7 PCCbPCTwiYxwaUqi04mlI255Z6CJFj3K7054lSSbki4wBaR+OOq+gS6J+3Wpx+vG2ara 8jGJaOHU2AZ4jLMZ8PV7FDBVNdgC4x2G3QKh1D9fcdjkcAjaUq8peAbZ/CbvrUtHkvaA G1MG+ZfdfEKSKkQVCPHWk67rpu/OitM595baoKvoIOccdYz6+OVAspOwGKTfhHEcHXxx 6+NbcTtn8SRRI1XC4T2yyaLqJ9RGH0aZwUmM6EzEukrnXCC9nsfkGCzlyLvvT8tQD1Gg Xwbw== X-Gm-Message-State: AOAM531OqtQKAR/L60+1eY1ureFrRw9BGuLbMesprx/S7QflPGffO8Nr oXIq5N1bj59hdCklBlnRib87awiUvrk= X-Google-Smtp-Source: ABdhPJwPBhwfDbX5+iLcuK3P5DKTGXGcPZNp/mAJIHUU3MtsIBezcBTLgE0eBu5SFXniQ8C4WOrHeQ== X-Received: by 2002:a17:90a:5518:: with SMTP id b24mr6063098pji.138.1605899794461; Fri, 20 Nov 2020 11:16:34 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:33 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 09/15] e2fsck: add fast commit scan pass Date: Fri, 20 Nov 2020 11:16:00 -0800 Message-Id: <20201120191606.2224881-10-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Add fast commit scan pass. Scan pass is responsible for following things: * Count total number of fast commit tags that need to be replayed during the replay phase. * Validate whether the fast commit area is valid for a given transaction ID. * Verify the CRC of fast commit area. Signed-off-by: Harshad Shirwadkar --- e2fsck/journal.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/e2fsck/journal.c b/e2fsck/journal.c index 2c8e3441..1072dfe8 100644 --- a/e2fsck/journal.c +++ b/e2fsck/journal.c @@ -278,6 +278,106 @@ static int process_journal_block(ext2_filsys fs, return 0; } +static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh, + int off, tid_t expected_tid) +{ + e2fsck_t ctx = j->j_fs_dev->k_ctx; + struct e2fsck_fc_replay_state *state; + int ret = JBD2_FC_REPLAY_CONTINUE; + struct ext4_fc_add_range *ext; + struct ext4_fc_tl *tl; + struct ext4_fc_tail *tail; + __u8 *start, *end; + struct ext4_fc_head *head; + struct ext3_extent *ex; + struct ext2fs_extent ext2fs_ex; + + state = &ctx->fc_replay_state; + + start = (__u8 *)bh->b_data; + end = (__u8 *)bh->b_data + j->j_blocksize - 1; + + jbd_debug(1, "Scan phase starting, expected %d", expected_tid); + if (state->fc_replay_expected_off == 0) { + memset(state, 0, sizeof(*state)); + /* Check if we can stop early */ + if (le16_to_cpu(((struct ext4_fc_tl *)start)->fc_tag) + != EXT4_FC_TAG_HEAD) { + jbd_debug(1, "Ending early!, not a head tag"); + return 0; + } + } + + if (off != state->fc_replay_expected_off) { + ret = -EFSCORRUPTED; + goto out_err; + } + + state->fc_replay_expected_off++; + fc_for_each_tl(start, end, tl) { + jbd_debug(3, "Scan phase, tag:%s, blk %lld\n", + tag2str(le16_to_cpu(tl->fc_tag)), bh->b_blocknr); + switch (le16_to_cpu(tl->fc_tag)) { + case EXT4_FC_TAG_ADD_RANGE: + ext = (struct ext4_fc_add_range *)ext4_fc_tag_val(tl); + ex = (struct ext3_extent *)&ext->fc_ex; + ext2fs_convert_extent(&ext2fs_ex, ex); + ret = JBD2_FC_REPLAY_CONTINUE; + case EXT4_FC_TAG_DEL_RANGE: + case EXT4_FC_TAG_LINK: + case EXT4_FC_TAG_UNLINK: + case EXT4_FC_TAG_CREAT: + case EXT4_FC_TAG_INODE: + case EXT4_FC_TAG_PAD: + state->fc_cur_tag++; + state->fc_crc = jbd2_chksum(j, state->fc_crc, tl, + sizeof(*tl) + ext4_fc_tag_len(tl)); + break; + case EXT4_FC_TAG_TAIL: + state->fc_cur_tag++; + tail = (struct ext4_fc_tail *)ext4_fc_tag_val(tl); + state->fc_crc = jbd2_chksum(j, state->fc_crc, tl, + sizeof(*tl) + + offsetof(struct ext4_fc_tail, + fc_crc)); + jbd_debug(1, "tail tid %d, expected %d\n", + le32_to_cpu(tail->fc_tid), + expected_tid); + if (le32_to_cpu(tail->fc_tid) == expected_tid && + le32_to_cpu(tail->fc_crc) == state->fc_crc) { + state->fc_replay_num_tags = state->fc_cur_tag; + } else { + ret = state->fc_replay_num_tags ? + JBD2_FC_REPLAY_STOP : -EFSBADCRC; + } + state->fc_crc = 0; + break; + case EXT4_FC_TAG_HEAD: + head = (struct ext4_fc_head *)ext4_fc_tag_val(tl); + if (le32_to_cpu(head->fc_features) & + ~EXT4_FC_SUPPORTED_FEATURES) { + ret = -EOPNOTSUPP; + break; + } + if (le32_to_cpu(head->fc_tid) != expected_tid) { + ret = -EINVAL; + break; + } + state->fc_cur_tag++; + state->fc_crc = jbd2_chksum(j, state->fc_crc, tl, + sizeof(*tl) + ext4_fc_tag_len(tl)); + break; + default: + ret = state->fc_replay_num_tags ? + JBD2_FC_REPLAY_STOP : -ECANCELED; + } + if (ret < 0 || ret == JBD2_FC_REPLAY_STOP) + break; + } + +out_err: + return ret; +} /* * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE * to indicate that it is expecting more fast commit blocks. It returns @@ -286,6 +386,13 @@ static int process_journal_block(ext2_filsys fs, static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh, enum passtype pass, int off, tid_t expected_tid) { + e2fsck_t ctx = journal->j_fs_dev->k_ctx; + struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state; + + if (pass == PASS_SCAN) { + state->fc_current_pass = PASS_SCAN; + return ext4_fc_replay_scan(journal, bh, off, expected_tid); + } return JBD2_FC_REPLAY_STOP; } From patchwork Fri Nov 20 19:16:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404069 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=mG7lmfYF; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rP267Hz9sTR for ; Sat, 21 Nov 2020 06:16:37 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732229AbgKTTQh (ORCPT ); Fri, 20 Nov 2020 14:16:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732220AbgKTTQg (ORCPT ); Fri, 20 Nov 2020 14:16:36 -0500 Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8343FC061A04 for ; Fri, 20 Nov 2020 11:16:36 -0800 (PST) Received: by mail-pf1-x444.google.com with SMTP id c66so8802498pfa.4 for ; Fri, 20 Nov 2020 11:16:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=w6NifX4iyXWjBWv9C/Mie0NhiSo4YnyHhwT58ryktRw=; b=mG7lmfYF3r1Kw95GFmVthu51HD10i1wRo5UeVpKxdfInbo4nphujsSFLdr2VTkdZtw 1k06H+CWjXs7XPymW9ZyBK9La7DQ2vwmhdUxCrnWjS30OsNNdagTVesxpVW+blBiG5uW uaRRacAUy60eiqCYY6yU4vdmReO++1CFTaviSMu3opKCsvp/pIBNSI4MDANaPSgW11aL UPV+l1yy4OKcFR7rp+9RVJ7gsMUr+YB81Y2K0eioOXkUXacblRtdYGXmOUBAtq/leN79 3U6Wr8wiOVnA97yCM6NAujSojTj+h/ZjfUVQKh3CtmWE5c9Qnxp/rqUxHBhg5wAX2qkl +TnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=w6NifX4iyXWjBWv9C/Mie0NhiSo4YnyHhwT58ryktRw=; b=HG3XuIA5Luo/QSizPF2rE+G8jQKwcCykHNY7txrc2XgPPmMDCG42Lx58jACNvnjlAB K020foZr3Xk9bc1rG1T3IQVPQllcjQR2dviEaGmNWlaeXmSAGk+W4/VTb4Otd6+FeVHK NSE6fhLR8RJXY28z26j/2lf0oqMKGdd4uzkz9pg2j50WD1I940wat77yJ2ZV5OUuAthE oxb1/devJpvgaODmFqApxcUPNj1mxes3OuubRXaYQyn6YbwOInqDRSyDsE6AzNo1z5sb 4HSFK8oIZoTniwUGNkM92a5it5C17MloKII0SkS1iU5wgPIml3SAl7A66HQpWzhWTL1S wL8Q== X-Gm-Message-State: AOAM532GJsPNnnoTG0TqYKpL0aVYDcxIquF6YN8it6vKTDrfsam0BjjD 1fMz8oqL2S+W81bfEHbBj2Ka9m/u+hk= X-Google-Smtp-Source: ABdhPJyi/cmfZzcxhK39U7zBg4bJx39V1lcSERRFtEDXg67Rm5PqQablGTS9JejrVB49bYvNa/t3cA== X-Received: by 2002:a63:1541:: with SMTP id 1mr18403000pgv.429.1605899795559; Fri, 20 Nov 2020 11:16:35 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:34 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 10/15] e2fsck: add fast commit replay skeleton Date: Fri, 20 Nov 2020 11:16:01 -0800 Message-Id: <20201120191606.2224881-11-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org This function adds the skeleton for the replay path. Following patches in the series implement the handling for individual tags. Signed-off-by: Harshad Shirwadkar --- e2fsck/journal.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/e2fsck/journal.c b/e2fsck/journal.c index 1072dfe8..d44f777b 100644 --- a/e2fsck/journal.c +++ b/e2fsck/journal.c @@ -388,11 +388,83 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh, { e2fsck_t ctx = journal->j_fs_dev->k_ctx; struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state; + int ret = JBD2_FC_REPLAY_CONTINUE; + struct ext4_fc_tl *tl; + __u8 *start, *end; if (pass == PASS_SCAN) { state->fc_current_pass = PASS_SCAN; return ext4_fc_replay_scan(journal, bh, off, expected_tid); } + + if (state->fc_replay_num_tags == 0) + goto replay_done; + + if (state->fc_current_pass != pass) { + /* Starting replay phase */ + state->fc_current_pass = pass; + /* We will reset checksums */ + ctx->fs->flags |= EXT2_FLAG_IGNORE_CSUM_ERRORS; + ret = ext2fs_read_bitmaps(ctx->fs); + if (ret) { + jbd_debug(1, "Error %d while reading bitmaps\n", ret); + return ret; + } + state->fc_super_state = ctx->fs->super->s_state; + /* + * Mark the file system to indicate it contains errors. That's + * because the updates performed by fast commit replay code are + * not atomic and may result in incosistent file system if it + * crashes before the replay is complete. + */ + ctx->fs->super->s_state |= EXT2_ERROR_FS; + ctx->fs->super->s_state |= EXT4_FC_REPLAY; + ext2fs_mark_super_dirty(ctx->fs); + ext2fs_flush(ctx->fs); + } + + start = (__u8 *)bh->b_data; + end = (__u8 *)bh->b_data + journal->j_blocksize - 1; + + fc_for_each_tl(start, end, tl) { + if (state->fc_replay_num_tags == 0) + goto replay_done; + jbd_debug(3, "Replay phase processing %s tag\n", + tag2str(le16_to_cpu(tl->fc_tag))); + state->fc_replay_num_tags--; + switch (le16_to_cpu(tl->fc_tag)) { + case EXT4_FC_TAG_CREAT: + case EXT4_FC_TAG_LINK: + case EXT4_FC_TAG_UNLINK: + case EXT4_FC_TAG_ADD_RANGE: + case EXT4_FC_TAG_DEL_RANGE: + case EXT4_FC_TAG_INODE: + case EXT4_FC_TAG_TAIL: + case EXT4_FC_TAG_PAD: + case EXT4_FC_TAG_HEAD: + break; + default: + ret = -ECANCELED; + break; + } + if (ret < 0) + break; + ret = JBD2_FC_REPLAY_CONTINUE; + } + return ret; +replay_done: + jbd_debug(1, "End of fast commit replay\n"); + if (state->fc_current_pass != pass) + return JBD2_FC_REPLAY_STOP; + + ext2fs_calculate_summary_stats(ctx->fs, 0 /* update bg also */); + ext2fs_write_block_bitmap(ctx->fs); + ext2fs_write_inode_bitmap(ctx->fs); + ext2fs_mark_super_dirty(ctx->fs); + ext2fs_set_gdt_csum(ctx->fs); + ctx->fs->super->s_state = state->fc_super_state; + ext2fs_flush(ctx->fs); + return JBD2_FC_REPLAY_STOP; } From patchwork Fri Nov 20 19:16:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404071 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=gQT6LIFp; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rQ5Ld9z9sTR for ; Sat, 21 Nov 2020 06:16:38 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732250AbgKTTQi (ORCPT ); Fri, 20 Nov 2020 14:16:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732200AbgKTTQi (ORCPT ); Fri, 20 Nov 2020 14:16:38 -0500 Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 303A8C0613CF for ; Fri, 20 Nov 2020 11:16:38 -0800 (PST) Received: by mail-pf1-x443.google.com with SMTP id v5so4750169pff.10 for ; Fri, 20 Nov 2020 11:16:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FqBQi+BRR4OmhyWRF4r/7FDyhTPx00SkRhU/boYxRzY=; b=gQT6LIFpFI9lDAEJkXeDdUWs1dUPkuzhY5FKCzMlou4D3FHRXGAQa0Gn8UJPZiAtdK fHkLgZbSRtxVw4Ju+/8b8eYPVbNlpWULkXoSnjMZJN4VUnkr8u4dX40j9nO/GDyb6tcp m44S3oZBTv/FSscVaX1KAUkNjTUKwRUHYMcF0SQ/eRDJGVqRGg8fA7IMPqnKEjKiHy/s JhCM2ozeENjtQYampLSgG5C0kYhJg4vY9PSfe1JsAH+MDRGgheDaPPa3UvygRXkvQ8zN fqov7/yvGJMdYurafMxxvO8FKakb5YIjSnPNp4q9NhcDkF/exSh0jCjeYk8lONV8wXYk 8uvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FqBQi+BRR4OmhyWRF4r/7FDyhTPx00SkRhU/boYxRzY=; b=XFayKkUfZHQbJA9mgKeIc7WMoFLZsGPW9aF7FHCiIQUVIUTGfkLnxzLzJObz5tBgc+ W2cS6nLawS1JXr4B8s5LtGtjVMdufEeYd02aG5Eh6z7nIWEFeBDBrkIm31Ktv+0lxqIu X/v4uwDj0NbpdjjY38ZOafw43S7YEkEZNsyRbDWPrWKCwIhtnQb7G9D338plPzqXvxGr R3nx7VEQBOOmTgvAw8WQWBs3ZtYZfG0KoGDVFLIlYYeFWBTQ76QtKKXL4CR6epm9zjpU XNzwkQwqe+uk37mxqHg/TlXaQReAL62qgicye98cxdj6tt6kUksrPb7aii4Hb/jSyhFr Q7mw== X-Gm-Message-State: AOAM5324nc9SzOGMrocYTcBH0FKUb7CCW5KiobBR/YScGA/FcRVgBQW/ T5w/72yheB8lAmHjN8vDtWUlYDpkFnw= X-Google-Smtp-Source: ABdhPJyntt0oVxY591Eskksx5DUfjje38fam2iG4YoinpYJ//4lLMb6v4NiPEsDNr7QNCyM1zpCUQQ== X-Received: by 2002:a17:90b:4b11:: with SMTP id lx17mr12521952pjb.154.1605899797076; Fri, 20 Nov 2020 11:16:37 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:35 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 11/15] e2fsck: add fc replay for link, unlink, creat tags Date: Fri, 20 Nov 2020 11:16:02 -0800 Message-Id: <20201120191606.2224881-12-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Add fast commit replay for directory entry updates. Signed-off-by: Harshad Shirwadkar --- e2fsck/journal.c | 93 +++++++++++++++++++++++++++++++++++++++++++++ lib/ext2fs/ext2fs.h | 4 ++ lib/ext2fs/unlink.c | 6 +-- 3 files changed, 100 insertions(+), 3 deletions(-) diff --git a/e2fsck/journal.c b/e2fsck/journal.c index d44f777b..9f4c4a70 100644 --- a/e2fsck/journal.c +++ b/e2fsck/journal.c @@ -378,6 +378,95 @@ static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh, out_err: return ret; } +/* Helper struct for dentry replay routines */ +struct dentry_info_args { + int parent_ino, dname_len, ino, inode_len; + char *dname; +}; + +static inline void tl_to_darg(struct dentry_info_args *darg, + struct ext4_fc_tl *tl) +{ + struct ext4_fc_dentry_info *fcd; + int tag = le16_to_cpu(tl->fc_tag); + + fcd = (struct ext4_fc_dentry_info *)ext4_fc_tag_val(tl); + + darg->parent_ino = le32_to_cpu(fcd->fc_parent_ino); + darg->ino = le32_to_cpu(fcd->fc_ino); + darg->dname = fcd->fc_dname; + darg->dname_len = ext4_fc_tag_len(tl) - + sizeof(struct ext4_fc_dentry_info); + darg->dname = malloc(darg->dname_len + 1); + memcpy(darg->dname, fcd->fc_dname, darg->dname_len); + darg->dname[darg->dname_len] = 0; + jbd_debug(1, "%s: %s, ino %d, parent %d\n", + tag == EXT4_FC_TAG_CREAT ? "create" : + (tag == EXT4_FC_TAG_LINK ? "link" : + (tag == EXT4_FC_TAG_UNLINK ? "unlink" : "error")), + darg->dname, darg->ino, darg->parent_ino); +} + +static int ext4_fc_handle_unlink(e2fsck_t ctx, struct ext4_fc_tl *tl) +{ + struct ext2_inode inode; + struct dentry_info_args darg; + ext2_filsys fs = ctx->fs; + + tl_to_darg(&darg, tl); + ext2fs_unlink(ctx->fs, darg.parent_ino, darg.dname, darg.ino, 0); + /* It's okay if the above call fails */ + free(darg.dname); +} + +static int ext4_fc_handle_link_and_create(e2fsck_t ctx, struct ext4_fc_tl *tl) +{ + struct dentry_info_args darg; + ext2_filsys fs = ctx->fs; + struct ext2_inode_large inode_large; + int ret, filetype, mode; + + tl_to_darg(&darg, tl); + ret = ext2fs_read_inode(fs, darg.ino, + (struct ext2_inode *)&inode_large); + if (ret) + goto out; + + mode = inode_large.i_mode; + + if (LINUX_S_ISREG(mode)) + filetype = EXT2_FT_REG_FILE; + else if (LINUX_S_ISDIR(mode)) + filetype = EXT2_FT_DIR; + else if (LINUX_S_ISCHR(mode)) + filetype = EXT2_FT_CHRDEV; + else if (LINUX_S_ISBLK(mode)) + filetype = EXT2_FT_BLKDEV; + else if (LINUX_S_ISLNK(mode)) + return EXT2_FT_SYMLINK; + else if (LINUX_S_ISFIFO(mode)) + filetype = EXT2_FT_FIFO; + else if (LINUX_S_ISSOCK(mode)) + filetype = EXT2_FT_SOCK; + else { + ret = -EINVAL; + goto out; + } + + /* + * Forcefully unlink if the same name is present and ignore the error + * if any, since this dirent might not exist + */ + ext2fs_unlink(fs, darg.parent_ino, darg.dname, darg.ino, + EXT2FS_UNLINK_FORCE); + + ret = ext2fs_link(fs, darg.parent_ino, darg.dname, darg.ino, + filetype); +out: + free(darg.dname); + return ret; + +} /* * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE * to indicate that it is expecting more fast commit blocks. It returns @@ -435,7 +524,11 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh, switch (le16_to_cpu(tl->fc_tag)) { case EXT4_FC_TAG_CREAT: case EXT4_FC_TAG_LINK: + ret = ext4_fc_handle_link_and_create(ctx, tl); + break; case EXT4_FC_TAG_UNLINK: + ret = ext4_fc_handle_unlink(ctx, tl); + break; case EXT4_FC_TAG_ADD_RANGE: case EXT4_FC_TAG_DEL_RANGE: case EXT4_FC_TAG_INODE: diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index bde369ef..7d9430fd 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -1675,6 +1675,10 @@ extern errcode_t ext2fs_get_pathname(ext2_filsys fs, ext2_ino_t dir, ext2_ino_t char **name); /* link.c */ +#define EXT2FS_UNLINK_FORCE 0x1 /* Forcefully unlink even if + * the inode number doesn't + * match the dirent + */ errcode_t ext2fs_link(ext2_filsys fs, ext2_ino_t dir, const char *name, ext2_ino_t ino, int flags); errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir, const char *name, diff --git a/lib/ext2fs/unlink.c b/lib/ext2fs/unlink.c index 8ab27ee2..3ec04cfb 100644 --- a/lib/ext2fs/unlink.c +++ b/lib/ext2fs/unlink.c @@ -49,7 +49,7 @@ static int unlink_proc(struct ext2_dir_entry *dirent, if (strncmp(ls->name, dirent->name, ext2fs_dirent_name_len(dirent))) return 0; } - if (ls->inode) { + if (!(ls->flags & EXT2FS_UNLINK_FORCE) && ls->inode) { if (dirent->inode != ls->inode) return 0; } else { @@ -70,7 +70,7 @@ static int unlink_proc(struct ext2_dir_entry *dirent, #endif errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir, const char *name, ext2_ino_t ino, - int flags EXT2FS_ATTR((unused))) + int flags) { errcode_t retval; struct link_struct ls; @@ -86,7 +86,7 @@ errcode_t ext2fs_unlink(ext2_filsys fs, ext2_ino_t dir, ls.name = name; ls.namelen = name ? strlen(name) : 0; ls.inode = ino; - ls.flags = 0; + ls.flags = flags; ls.done = 0; ls.prev = 0; From patchwork Fri Nov 20 19:16:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404072 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Z+qYsOTw; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rS61t7z9sTR for ; Sat, 21 Nov 2020 06:16:40 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732255AbgKTTQk (ORCPT ); Fri, 20 Nov 2020 14:16:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732200AbgKTTQj (ORCPT ); Fri, 20 Nov 2020 14:16:39 -0500 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92B7EC0613CF for ; Fri, 20 Nov 2020 11:16:39 -0800 (PST) Received: by mail-pg1-x52a.google.com with SMTP id t37so8031409pga.7 for ; Fri, 20 Nov 2020 11:16:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wt3jsceFNkLIQh4mBUjcPZa+ba6zcRcGDtdcaOK1uP0=; b=Z+qYsOTwYIXF2a4v+jDahjBf5GcHACIHJ6zKhQg7LK1eoHRh1RA2FOYlEq+50wkZT6 B2KxbUADKMsWPejlofK2jeckE7pej+8x76poHkhVjV7YqXD4iUogD6K7LN499YQk+tro 4mFbUbBgM0Bpj3pbVYT6Ee9YRbwyMMn3BAqUR4oFnSiy3dgrYagBoTlMPWOjfqA1ysvC uiA+VDYRQBnzm1LU5pspLFgBybG4HRbu8nUXkJV8kg6WWWly0fSjpLAazVWypR4wDP4e KljDFugaxOSwwPo3J6wEGND3Ok+CM+5dDtM7tMWr98DSRbPPRdtTa+Q61h+LgqRIjWLz SWnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wt3jsceFNkLIQh4mBUjcPZa+ba6zcRcGDtdcaOK1uP0=; b=YdIrBsu6nbfE31JZdWn7RGEraSxQOuNqxaln0ib4d9h6AYuG+E+axFrESFYfWbzj6Q ah1SZbJMhIgbQVq3MH6d51Y5EEmh67MjoO73za2I9dcFssHe8geKImqOslTEKGKQzPUI SufBVozx29pvM34TVKHvIzrR7qK78Cz/vFpkcwRocMmys9RCI1BNQ7qvVlxS73RaWMvk 7WMI8f9N0wzT7g56YgIlsIPBVYIo8PGpCMwCdy0nNl0Ct3b0zzMUgGaXpt1C5kN/91JE /GX5hhRHRVhsNw81puSUdgBfjI0CMurr6YiwBkqH5lpnv7LfW+bI6GhGGe+kAUoHfLVb qvUg== X-Gm-Message-State: AOAM532rj+x1L0iHweBc2JOjq1jgbzV3WbM6gBuQtFLz7EYAQHfIyYLY Hd/5MCVPq9xNEdE++YDjuy2lz0EE3jI= X-Google-Smtp-Source: ABdhPJxfQTExM4nTdn0J/IzJeUbIt4TJ49hfHMmOPqqjB1jDv+tD7rjyWrRACxyKO+Xr60PGLTBvyw== X-Received: by 2002:a63:1918:: with SMTP id z24mr18211314pgl.367.1605899798538; Fri, 20 Nov 2020 11:16:38 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:37 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 12/15] e2fsck: add replay for add_range, del_range, and inode tags Date: Fri, 20 Nov 2020 11:16:03 -0800 Message-Id: <20201120191606.2224881-13-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Add replay for inode's extent trees and inode itself. Signed-off-by: Harshad Shirwadkar --- e2fsck/journal.c | 341 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 341 insertions(+) diff --git a/e2fsck/journal.c b/e2fsck/journal.c index 9f4c4a70..ccd70186 100644 --- a/e2fsck/journal.c +++ b/e2fsck/journal.c @@ -378,6 +378,219 @@ static int ext4_fc_replay_scan(journal_t *j, struct buffer_head *bh, out_err: return ret; } + +#define ex_end(__ex) ((__ex)->e_lblk + (__ex)->e_len - 1) +#define ex_pend(__ex) ((__ex)->e_pblk + (__ex)->e_len - 1) + +static int make_room(struct extent_list *list, int i) +{ + int ret; + + if (list->count == list->size) { + unsigned int new_size = (list->size + 341) * + sizeof(struct ext2fs_extent); + ret = ext2fs_resize_mem(0, new_size, &list->extents); + if (ret) + return ret; + list->size += 341; + } + + memmove(&list->extents[i + 1], &list->extents[i], + sizeof(list->extents[0]) * (list->count - i)); + list->count++; + return 0; +} + +static int ex_compar(const void *arg1, const void *arg2) +{ + struct ext2fs_extent *ex1 = (struct ext2fs_extent *)arg1; + struct ext2fs_extent *ex2 = (struct ext2fs_extent *)arg2; + + if (ex1->e_lblk < ex2->e_lblk) + return -1; + if (ex1->e_lblk > ex2->e_lblk) + return 1; + return ex1->e_len - ex2->e_len; +} + +static int ex_len_compar(const void *arg1, const void *arg2) +{ + struct ext2fs_extent *ex1 = (struct ext2fs_extent *)arg1; + struct ext2fs_extent *ex2 = (struct ext2fs_extent *)arg2; + + if (ex1->e_len < ex2->e_len) + return 1; + + if (ex1->e_lblk > ex2->e_lblk) + return -1; + + return 0; +} + +static void ex_sort_and_merge(e2fsck_t ctx, struct extent_list *list) +{ + blk64_t ex_end; + int i, j; + + if (list->count < 2) + return; + + /* + * Reverse sort by length, that way we strip off all the 0 length + * extents + */ + qsort(list->extents, list->count, sizeof(struct ext2fs_extent), + ex_len_compar); + + for (i = 0; i < list->count; i++) { + if (list->extents[i].e_len == 0) { + list->count = i; + break; + } + } + + /* Now sort by logical offset */ + qsort(list->extents, list->count, sizeof(list->extents[0]), + ex_compar); + + /* Merge adjacent extents if they are logically and physically contiguous */ + i = 0; + while (i < list->count - 1) { + if (ex_end(&list->extents[i]) + 1 != list->extents[i + 1].e_lblk || + ex_pend(&list->extents[i]) + 1 != list->extents[i + 1].e_pblk || + (list->extents[i].e_flags & EXT2_EXTENT_FLAGS_UNINIT) != + (list->extents[i + 1].e_flags & EXT2_EXTENT_FLAGS_UNINIT)) { + i++; + continue; + } + + list->extents[i].e_len += list->extents[i + 1].e_len; + for (j = i + 1; j < list->count - 1; j++) + list->extents[j] = list->extents[j + 1]; + list->count--; + } +} + +/* must free blocks that are released */ +static int ext2fs_modify_extent_list(e2fsck_t ctx, struct extent_list *list, + struct ext2fs_extent *ex, int del) +{ + int ret; + int i, offset; + struct ext2fs_extent add_ex = *ex, add_ex2; + + /* First let's create a hole from ex->e_lblk of length ex->e_len */ + for (i = 0; i < list->count; i++) { + if (ex_end(&list->extents[i]) < add_ex.e_lblk) + continue; + + /* Case 1: No overlap */ + if (list->extents[i].e_lblk > ex_end(&add_ex)) + break; + /* + * Unmark all the blocks in bb now. All the blocks get marked + * before we exit this function. + */ + ext2fs_unmark_block_bitmap_range2(ctx->fs->block_map, + list->extents[i].e_pblk, list->extents[i].e_len); + /* Case 2: Split */ + if (list->extents[i].e_lblk < add_ex.e_lblk && + ex_end(&list->extents[i]) > ex_end(&add_ex)) { + ret = make_room(list, i + 1); + if (ret) + return ret; + list->extents[i + 1] = list->extents[i]; + offset = ex_end(&add_ex) + 1 - list->extents[i].e_lblk; + list->extents[i + 1].e_lblk += offset; + list->extents[i + 1].e_pblk += offset; + list->extents[i + 1].e_len -= offset; + list->extents[i].e_len = + add_ex.e_lblk - list->extents[i].e_lblk; + break; + } + + /* Case 3: Exact overlap */ + if (add_ex.e_lblk <= list->extents[i].e_lblk && + ex_end(&list->extents[i]) <= ex_end(&add_ex)) { + + list->extents[i].e_len = 0; + continue; + } + + /* Case 4: Partial overlap */ + if (ex_end(&list->extents[i]) > ex_end(&add_ex)) { + offset = ex_end(&add_ex) + 1 - list->extents[i].e_lblk; + list->extents[i].e_lblk += offset; + list->extents[i].e_pblk += offset; + list->extents[i].e_len -= offset; + break; + } + + if (ex_end(&add_ex) >= ex_end(&list->extents[i])) { + list->extents[i].e_len = + max(add_ex.e_lblk - list->extents[i].e_lblk, + 0); + } + } + + if (add_ex.e_len && !del) { + make_room(list, list->count); + list->extents[list->count - 1] = add_ex; + } + + ex_sort_and_merge(ctx, list); + + /* Mark all occupied blocks allocated */ + for (i = 0; i < list->count; i++) + ext2fs_mark_block_bitmap_range2(ctx->fs->block_map, + list->extents[i].e_pblk, list->extents[i].e_len); + ext2fs_mark_bb_dirty(ctx->fs); + + return 0; +} + +static int ext2fs_add_extent_to_list(e2fsck_t ctx, struct extent_list *list, + struct ext2fs_extent *ex) +{ + return ext2fs_modify_extent_list(ctx, list, ex, 0 /* add */); +} + +static int ext2fs_del_extent_to_list(e2fsck_t ctx, struct extent_list *list, + struct ext2fs_extent *ex) +{ + return ext2fs_modify_extent_list(ctx, list, ex, 1 /* delete */); +} + +static int e2fsck_fc_read_extents(e2fsck_t ctx, int ino) +{ + struct extent_list *extent_list = &ctx->fc_replay_state.fc_extent_list; + + if (extent_list->ino == ino) + return 0; + + extent_list->ino = ino; + return e2fsck_read_extents(ctx, extent_list); +} + +/* + * Flush extents in replay state on disk. @ino is the inode that is going + * to be processed next. So, we hold back flushing of the extent list + * if the next inode that's going to be processed is same as the one with + * cached extents in our replay state. That allows us to gather multiple extents + * for the inode so that we can flush all of them at once and it also saves us + * from continuously growing and shrinking the extent tree. + */ +static void e2fsck_fc_flush_extents(e2fsck_t ctx, int ino) +{ + struct extent_list *extent_list = &ctx->fc_replay_state.fc_extent_list; + + if (extent_list->ino == ino || extent_list->ino == 0) + return; + e2fsck_rewrite_extent_tree(ctx, extent_list); + ext2fs_free_mem(&extent_list->extents); + memset(extent_list, 0, sizeof(*extent_list)); +} + /* Helper struct for dentry replay routines */ struct dentry_info_args { int parent_ino, dname_len, ino, inode_len; @@ -414,6 +627,7 @@ static int ext4_fc_handle_unlink(e2fsck_t ctx, struct ext4_fc_tl *tl) ext2_filsys fs = ctx->fs; tl_to_darg(&darg, tl); + e2fsck_fc_flush_extents(ctx, darg.ino); ext2fs_unlink(ctx->fs, darg.parent_ino, darg.dname, darg.ino, 0); /* It's okay if the above call fails */ free(darg.dname); @@ -427,6 +641,8 @@ static int ext4_fc_handle_link_and_create(e2fsck_t ctx, struct ext4_fc_tl *tl) int ret, filetype, mode; tl_to_darg(&darg, tl); + + e2fsck_fc_flush_extents(ctx, 0); ret = ext2fs_read_inode(fs, darg.ino, (struct ext2_inode *)&inode_large); if (ret) @@ -467,6 +683,124 @@ out: return ret; } + +/* This function fixes the i_blocks field in the replayed indoe */ +static void ext4_fc_replay_fixup_iblocks(struct ext2_inode_large *ondisk_inode, + struct ext2_inode_large *fc_inode) +{ + if (le32_to_cpu(ondisk_inode->i_flags) & EXT4_EXTENTS_FL) { + struct ext3_extent_header *eh; + + eh = (struct ext3_extent_header *)(&ondisk_inode->i_block[0]); + if (eh->eh_magic != EXT3_EXT_MAGIC) { + memset(eh, 0, sizeof(*eh)); + eh->eh_magic = EXT3_EXT_MAGIC; + eh->eh_max = cpu_to_le16( + (sizeof(ondisk_inode->i_block) - + sizeof(struct ext3_extent_header)) / + sizeof(struct ext3_extent)); + } + } else if (le32_to_cpu(ondisk_inode->i_flags) & EXT4_INLINE_DATA_FL) { + memcpy(ondisk_inode->i_block, fc_inode->i_block, + sizeof(fc_inode->i_block)); + } +} + +static int ext4_fc_handle_inode(e2fsck_t ctx, struct ext4_fc_tl *tl) +{ + struct ext4_fc_inode *fc_inode; + struct ext2_inode_large *inode = NULL; + int ino, inode_len = EXT2_GOOD_OLD_INODE_SIZE, ret; + struct e2fsck_fc_replay_state *state = &ctx->fc_replay_state; + + fc_inode = (struct ext4_fc_inode *)ext4_fc_tag_val(tl); + ino = le32_to_cpu(fc_inode->fc_ino); + + if (EXT2_INODE_SIZE(ctx->fs->super) > EXT2_GOOD_OLD_INODE_SIZE) + inode_len += ext2fs_le16_to_cpu( + ((struct ext2_inode_large *)fc_inode->fc_raw_inode) + ->i_extra_isize); + ret = ext2fs_get_mem(inode_len, &inode); + if (ret) + return ret; + e2fsck_fc_flush_extents(ctx, ino); + + ret = ext2fs_read_inode_full(ctx->fs, ino, (struct ext2_inode *)inode, + inode_len); + if (ret) + goto out; + memcpy(inode, fc_inode->fc_raw_inode, + offsetof(struct ext2_inode_large, i_block)); + memcpy(&inode->i_generation, + &((struct ext2_inode_large *)(fc_inode->fc_raw_inode))->i_generation, + inode_len - offsetof(struct ext2_inode_large, i_generation)); + ext4_fc_replay_fixup_iblocks(inode, + (struct ext2_inode_large *)fc_inode->fc_raw_inode); + ext2fs_iblk_set(ctx->fs, EXT2_INODE(inode), + ext2fs_count_blocks(ctx->fs, ino, EXT2_INODE(inode))); + + ext2fs_inode_csum_set(ctx->fs, ino, inode); + + ret = ext2fs_write_inode_full(ctx->fs, ino, (struct ext2_inode *)inode, + inode_len); + if (inode->i_links_count) + ext2fs_mark_inode_bitmap2(ctx->fs->inode_map, ino); + else + ext2fs_unmark_inode_bitmap2(ctx->fs->inode_map, ino); + ext2fs_mark_ib_dirty(ctx->fs); + +out: + ext2fs_free_mem(&inode); + return ret; +} + +/* + * Handle add extent replay tag. + */ +static int ext4_fc_handle_add_extent(e2fsck_t ctx, struct ext4_fc_tl *tl) +{ + struct ext2fs_extent extent; + struct ext4_fc_add_range *add_range; + struct ext4_fc_del_range *del_range; + int ret = 0, ino; + + add_range = (struct ext4_fc_add_range *)ext4_fc_tag_val(tl); + ino = le32_to_cpu(add_range->fc_ino); + e2fsck_fc_flush_extents(ctx, ino); + + ret = e2fsck_fc_read_extents(ctx, ino); + if (ret) + return ret; + memset(&extent, 0, sizeof(extent)); + ext2fs_convert_extent(&extent, (struct ext3_extent *)( + add_range->fc_ex)); + return ext2fs_add_extent_to_list(ctx, + &ctx->fc_replay_state.fc_extent_list, &extent); +} + +/* + * Handle delete logical range replay tag. + */ +static int ext4_fc_handle_del_range(e2fsck_t ctx, struct ext4_fc_tl *tl) +{ + struct ext2fs_extent extent; + struct ext4_fc_del_range *del_range; + int ret, ino; + + del_range = (struct ext4_fc_del_range *)ext4_fc_tag_val(tl); + ino = le32_to_cpu(del_range->fc_ino); + e2fsck_fc_flush_extents(ctx, ino); + + memset(&extent, 0, sizeof(extent)); + extent.e_lblk = ext2fs_le32_to_cpu(del_range->fc_lblk); + extent.e_len = ext2fs_le16_to_cpu(del_range->fc_len); + ret = e2fsck_fc_read_extents(ctx, ino); + if (ret) + return ret; + return ext2fs_del_extent_to_list(ctx, + &ctx->fc_replay_state.fc_extent_list, &extent); +} + /* * Main recovery path entry point. This function returns JBD2_FC_REPLAY_CONTINUE * to indicate that it is expecting more fast commit blocks. It returns @@ -530,9 +864,16 @@ static int ext4_fc_replay(journal_t *journal, struct buffer_head *bh, ret = ext4_fc_handle_unlink(ctx, tl); break; case EXT4_FC_TAG_ADD_RANGE: + ret = ext4_fc_handle_add_extent(ctx, tl); + break; case EXT4_FC_TAG_DEL_RANGE: + ret = ext4_fc_handle_del_range(ctx, tl); + break; case EXT4_FC_TAG_INODE: + ret = ext4_fc_handle_inode(ctx, tl); + break; case EXT4_FC_TAG_TAIL: + e2fsck_fc_flush_extents(ctx, 0); case EXT4_FC_TAG_PAD: case EXT4_FC_TAG_HEAD: break; From patchwork Fri Nov 20 19:16:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404074 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=MTK2Fdxz; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rX64Flz9sTR for ; Sat, 21 Nov 2020 06:16:44 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732274AbgKTTQn (ORCPT ); Fri, 20 Nov 2020 14:16:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732200AbgKTTQm (ORCPT ); Fri, 20 Nov 2020 14:16:42 -0500 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20ECFC0613CF for ; Fri, 20 Nov 2020 11:16:41 -0800 (PST) Received: by mail-pf1-x441.google.com with SMTP id c66so8802688pfa.4 for ; Fri, 20 Nov 2020 11:16:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aq+mQbTZB8JdqawelikdF6sYXjAnMc5tHSNbbcALgwo=; b=MTK2FdxzZL6hdjHUyC6cgw5Gf382dyEZ31dbz9aPTlfRg5Kefi+yP+ZMEMh1qjnWwu j3jaUyZVshva0WTPDJS22cZOOinoFHMWwkpzAGvWMjpsO+/b0zKp70dHs5pk8canrnXg n/xo620EtpPnXBDpOUoj4x8X7cGBl/3yssaDT047IWmNQMLBf8PQSa1XHi+4Y9N08wU1 7NWTkhesvs9KAjUOnvAWryRFY622AEBQ0XVfrGfBMgE33jvI+KnojwIQhWyKRi2w5EG7 N48bqyo1TVs1jurQGCALwJC9ayIEU5N/9S1g0lnKBexk4gIypnJew2dBHATosUfNMMFm XOtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aq+mQbTZB8JdqawelikdF6sYXjAnMc5tHSNbbcALgwo=; b=FQV7W1VLNavSP+8QehJnQ9Fu3s7+eh1S/YG0FmDcGZwuIH3txlWdp+7gnf88Xllxlz mEb71zULKyXKKn0Ec8hMQQpMLh42Ccln/vdRDcA3ZWaoQnm7SCGTS3Xwi7cAZIRyrdiY HU+OPJI2fgDKrGEBDFPjulxndzlNmF08MKLs7Gf6Cbpr5iM87lUDcufeDD/XG9GKgN8v JYZPcueNyCZcHBdkQZ5Gv5XTkUE33k8+ZojowyIFpCZvpQxK8qeV+pCkHwcEDM5/Lt5k MQS1OOP6CaSM1u2cPXD2RUOlVhW4N3scxpT9vvXLLKuxk04fqTG9cffghwnRGFFJCnaI dzCg== X-Gm-Message-State: AOAM5337RBEvRqcCeK7t+RLj1Pam81hcdW44JGQqVCDl0SO1NoZjg7kN 92r9kVl3+SZbw4tv5zv2OcfblQsqYi8= X-Google-Smtp-Source: ABdhPJzVFfmZW/066pw28fatgLF42ZjZWZdO73n6/B4xm+xNpSrBrg/ecfqX3v5iWtrQ4n2eQaUUMg== X-Received: by 2002:a17:90b:19c4:: with SMTP id nm4mr10441311pjb.222.1605899800167; Fri, 20 Nov 2020 11:16:40 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:38 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 13/15] debugfs: add fast commit support to logdump Date: Fri, 20 Nov 2020 11:16:04 -0800 Message-Id: <20201120191606.2224881-14-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Add fast commit support for debugfs logdump. The debugfs output looks like this: debugfs 1.46-WIP (20-Mar-2020) debugfs: logdump Journal starts at block 1, transaction 2 Found expected sequence 2, type 1 (descriptor block) at block 1 Found expected sequence 2, type 2 (commit block) at block 10 No magic number at block 11: end of journal. *** Fast Commit Area *** tag HEAD, features 0x0, tid 3 tag INODE, inode 14 tag ADD_RANGE, inode 14, lblk 0, pblk 1091, len 1 tag DEL_RANGE, inode 14, lblk 1, len 1 tag CREAT_DENTRY, parent 12, ino 14, name "new" tag DEL_ENTRY, parent 12, ino 13, name "old" tag INODE, inode 13 tag ADD_RANGE, inode 13, lblk 0, pblk 1603, len 16 tag ADD_RANGE, inode 13, lblk 16, pblk 1092, len 240 tag CREAT_DENTRY, parent 12, ino 13, name "data" tag INODE, inode 14 tag INODE, inode 13 tag TAIL, tid 3 Signed-off-by: Harshad Shirwadkar --- debugfs/logdump.c | 122 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 117 insertions(+), 5 deletions(-) diff --git a/debugfs/logdump.c b/debugfs/logdump.c index 16889954..151be3e2 100644 --- a/debugfs/logdump.c +++ b/debugfs/logdump.c @@ -33,6 +33,7 @@ extern char *optarg; #include "debugfs.h" #include "blkid/blkid.h" #include "jfs_user.h" +#include "ext2fs/fast_commit.h" #include enum journal_location {JOURNAL_IS_INTERNAL, JOURNAL_IS_EXTERNAL}; @@ -65,6 +66,9 @@ static void dump_metadata_block(FILE *, struct journal_source *, unsigned int, unsigned int, unsigned int, int, tid_t); +static void dump_fc_block(FILE *out_file, char *buf, int blocksize, + int transaction, int *fc_done, int dump_old); + static void do_hexdump (FILE *, char *, int); #define WRAP(jsb, blocknr) \ @@ -353,6 +357,7 @@ static void dump_journal(char *cmdname, FILE *out_file, journal_header_t *header; tid_t transaction; unsigned int blocknr = 0; + int fc_done; /* First, check to see if there's an ext2 superblock header */ retval = read_journal_block(cmdname, source, 0, buf, 2048); @@ -410,7 +415,7 @@ static void dump_journal(char *cmdname, FILE *out_file, if (!blocknr) { /* Empty journal, nothing to do. */ if (!dump_old) - return; + goto fc; else blocknr = 1; } @@ -420,7 +425,7 @@ static void dump_journal(char *cmdname, FILE *out_file, ((ext2_loff_t) blocknr) * blocksize, buf, blocksize); if (retval) - return; + break; header = (journal_header_t *) buf; @@ -431,7 +436,7 @@ static void dump_journal(char *cmdname, FILE *out_file, if (magic != JBD2_MAGIC_NUMBER) { fprintf (out_file, "No magic number at block %u: " "end of journal.\n", blocknr); - return; + break; } if (sequence != transaction) { @@ -439,7 +444,7 @@ static void dump_journal(char *cmdname, FILE *out_file, "block %u: end of journal.\n", sequence, transaction, blocknr); if (!dump_old) - return; + break; } if (dump_descriptors) { @@ -473,9 +478,25 @@ static void dump_journal(char *cmdname, FILE *out_file, default: fprintf (out_file, "Unexpected block type %u at " "block %u.\n", blocktype, blocknr); - return; + break; } } + +fc: + blocknr = be32_to_cpu(jsb->s_maxlen) - jbd2_journal_get_num_fc_blks(jsb) + 1; + while (blocknr <= be32_to_cpu(jsb->s_maxlen)) { + retval = read_journal_block(cmdname, source, + ((ext2_loff_t) blocknr) * blocksize, + buf, blocksize); + if (retval) + return; + + dump_fc_block(out_file, buf, blocksize, transaction, &fc_done, + dump_old); + if (!dump_old && fc_done) + break; + blocknr++; + } } static inline size_t journal_super_tag_bytes(journal_superblock_t *jsb) @@ -496,6 +517,97 @@ static inline size_t journal_super_tag_bytes(journal_superblock_t *jsb) return sz - sizeof(__u32); } +static void dump_fc_block(FILE *out_file, char *buf, int blocksize, + int transaction, int *fc_done, int dump_old) +{ + struct ext4_fc_tl *tl; + struct ext4_fc_head *head; + struct ext4_fc_add_range *add_range; + struct ext4_fc_del_range *del_range; + struct ext4_fc_dentry_info *dentry_info; + struct ext4_fc_tail *tail; + struct ext3_extent *ex; + + *fc_done = 0; + fc_for_each_tl(buf, buf + blocksize, tl) { + switch (le16_to_cpu(tl->fc_tag)) { + case EXT4_FC_TAG_ADD_RANGE: + add_range = + (struct ext4_fc_add_range *)ext4_fc_tag_val(tl); + ex = (struct ext3_extent *)add_range->fc_ex; + fprintf(out_file, + "tag %s, inode %d, lblk %d, pblk %ld, len %d\n", + tag2str(tl->fc_tag), + le32_to_cpu(add_range->fc_ino), + le32_to_cpu(ex->ee_block), + le32_to_cpu(ex->ee_start) + + (((__u64) le16_to_cpu(ex->ee_start_hi)) << 32), + le16_to_cpu(ex->ee_len) > EXT_INIT_MAX_LEN ? + le16_to_cpu(ex->ee_len) - EXT_INIT_MAX_LEN : + le16_to_cpu(ex->ee_len)); + break; + case EXT4_FC_TAG_DEL_RANGE: + del_range = + (struct ext4_fc_del_range *)ext4_fc_tag_val(tl); + fprintf(out_file, "tag %s, inode %d, lblk %d, len %d\n", + tag2str(tl->fc_tag), + le32_to_cpu(del_range->fc_ino), + le32_to_cpu(del_range->fc_lblk), + le32_to_cpu(del_range->fc_len)); + break; + case EXT4_FC_TAG_LINK: + case EXT4_FC_TAG_UNLINK: + case EXT4_FC_TAG_CREAT: + dentry_info = + (struct ext4_fc_dentry_info *) + ext4_fc_tag_val(tl); + fprintf(out_file, + "tag %s, parent %d, ino %d, name \"%s\"\n", + tag2str(tl->fc_tag), + le32_to_cpu(dentry_info->fc_parent_ino), + le32_to_cpu(dentry_info->fc_ino), + dentry_info->fc_dname); + break; + case EXT4_FC_TAG_INODE: + fprintf(out_file, "tag %s, inode %d\n", + tag2str(tl->fc_tag), + le32_to_cpu(((struct ext4_fc_inode *) + ext4_fc_tag_val(tl))->fc_ino)); + break; + case EXT4_FC_TAG_PAD: + fprintf(out_file, "tag %s\n", tag2str(tl->fc_tag)); + break; + case EXT4_FC_TAG_TAIL: + tail = (struct ext4_fc_tail *)ext4_fc_tag_val(tl); + fprintf(out_file, "tag %s, tid %d\n", + tag2str(tl->fc_tag), + le32_to_cpu(tail->fc_tid)); + if (!dump_old && + le32_to_cpu(tail->fc_tid) < transaction) { + *fc_done = 1; + return; + } + break; + case EXT4_FC_TAG_HEAD: + fprintf(out_file, "\n*** Fast Commit Area ***\n"); + head = (struct ext4_fc_head *)ext4_fc_tag_val(tl); + fprintf(out_file, "tag %s, features 0x%x, tid %d\n", + tag2str(tl->fc_tag), + le32_to_cpu(head->fc_features), + le32_to_cpu(head->fc_tid)); + if (!dump_old && + le32_to_cpu(head->fc_tid) < transaction) { + *fc_done = 1; + return; + } + break; + default: + *fc_done = 1; + break; + } + } +} + static void dump_descriptor_block(FILE *out_file, struct journal_source *source, char *buf, From patchwork Fri Nov 20 19:16:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404073 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Tvf3ztuI; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rW506Bz9sTR for ; Sat, 21 Nov 2020 06:16:43 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732270AbgKTTQn (ORCPT ); Fri, 20 Nov 2020 14:16:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732257AbgKTTQm (ORCPT ); Fri, 20 Nov 2020 14:16:42 -0500 Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 925CDC061A04 for ; Fri, 20 Nov 2020 11:16:42 -0800 (PST) Received: by mail-pg1-x52b.google.com with SMTP id r18so8112044pgu.6 for ; Fri, 20 Nov 2020 11:16:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mGxPJmgwdj47SE+aKI9OYp28RhTZr5pYTnMDSjDyNi0=; b=Tvf3ztuIS8dsXrzNNl1CH/TaPT8q3Q0u3t4IkerGHJy/hubpf0JR8jI/MzY1Q0SPNX KKn0R43bMtFB7K3wxt29KUYuhS5/SuKwO2wtFutZPZ+qEoC3fwkavR9hHoT0mMMb932i 8iK2FZMNP0Ae7f54Twz3c4fCLo5IDREehCcun827Isk/cvR7Sl/hYq5op/tjc0NDSwhQ aChZDBt3yauLN7pG631NZbajMzmuVzyZ1rNM5LFiJzvXmQFQ81gk4NLqzhxB48zj2duN JDBismYInhisSkezGEGC6ixE3jdeY5J9ovGo3SEIG++PupN8ffaSPzBEjYhHfIl7vwor 7swg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mGxPJmgwdj47SE+aKI9OYp28RhTZr5pYTnMDSjDyNi0=; b=esejNXV9oUalDRi13NEpeus0ks8dEV71ji6QKt1ah1Qcs5R3BNcMFRvaFHiCAbjeFJ ygfoAEqyOdKoS556JnZrnV0YS5PoWmI+iO60WzAlIX8BgpEd3TvXZLvTEiEmpSP/bkvI a3VXN15W/0m/Xq04xkeZWZNCaZXM5EdIgod/FkNdZOzYy5jFIvyR8IjHnsiaBNPE8v3d /MuXmca/+F3oROiWTGirsy88T7EOj1q781jFFj7Su6fok7SJvgzwOncHa98iZbt1jgo6 EYZOtCPOiXppoSHDJcNa+lnqihB+EEqs/5MhdPr8gzC1CLgIzsrCys1N/yuQpEYodfqV XOmQ== X-Gm-Message-State: AOAM530zANmRLFy/GK/zJIVpprg9pFv8LnKXhE1zhBhw61m/RqTgFYUE TTM2hGx6y+gvle0UehGgEIkFG7R98yE= X-Google-Smtp-Source: ABdhPJz3n04hBRDWcsBKGEc8KFmuGIEHtGbZFplO0250rRhspZFDQHhq49bxqVYEnE3EaiM/f3IBEg== X-Received: by 2002:a17:90a:f0f:: with SMTP id 15mr11982476pjy.127.1605899801564; Fri, 20 Nov 2020 11:16:41 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:40 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 14/15] tests: add fast commit recovery tests Date: Fri, 20 Nov 2020 11:16:05 -0800 Message-Id: <20201120191606.2224881-15-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Add j_recover_fast_commit test that ensure that e2fsck is able to recover a disk from fast commit log. Signed-off-by: Harshad Shirwadkar --- tests/j_recover_fast_commit/commands | 4 ++++ tests/j_recover_fast_commit/expect | 23 +++++++++++++++++++++++ tests/j_recover_fast_commit/image.gz | Bin 0 -> 3595 bytes tests/j_recover_fast_commit/script | 26 ++++++++++++++++++++++++++ 4 files changed, 53 insertions(+) create mode 100644 tests/j_recover_fast_commit/commands create mode 100644 tests/j_recover_fast_commit/expect create mode 100644 tests/j_recover_fast_commit/image.gz create mode 100755 tests/j_recover_fast_commit/script diff --git a/tests/j_recover_fast_commit/commands b/tests/j_recover_fast_commit/commands new file mode 100644 index 00000000..74e20e4e --- /dev/null +++ b/tests/j_recover_fast_commit/commands @@ -0,0 +1,4 @@ +ls +ls a/ +ex a/new +ex a/data diff --git a/tests/j_recover_fast_commit/expect b/tests/j_recover_fast_commit/expect new file mode 100644 index 00000000..2fc1e53f --- /dev/null +++ b/tests/j_recover_fast_commit/expect @@ -0,0 +1,23 @@ +test_filesys: recovering journal +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +test_filesys: 14/256 files (14.3% non-contiguous), 1365/2048 blocks +Exit status is 0 +debugfs: ls + 2 (12) . 2 (12) .. 11 (20) lost+found 12 (968) a +debugfs: ls a/ + 12 (12) . 2 (12) .. 13 (12) data 14 (976) new +debugfs: ex a/new +Level Entries Logical Physical Length Flags + 0/ 0 1/ 1 0 - 0 1107 - 1107 1 +debugfs: ex a/data +Level Entries Logical Physical Length Flags + 0/ 1 1/ 1 0 - 255 1618 256 + 1/ 1 1/ 5 0 - 15 1619 - 1634 16 + 1/ 1 2/ 5 16 - 31 1601 - 1616 16 + 1/ 1 3/ 5 32 - 63 1985 - 2016 32 + 1/ 1 4/ 5 64 - 127 1537 - 1600 64 + 1/ 1 5/ 5 128 - 255 1793 - 1920 128 diff --git a/tests/j_recover_fast_commit/image.gz b/tests/j_recover_fast_commit/image.gz new file mode 100644 index 0000000000000000000000000000000000000000..b7357afc46bec8a0154d5746fa2fce86c2341efc GIT binary patch literal 3595 zcmeH}eNfT~8ppY9Z*$KxYh$%EC0B221!CQ{-F$;y_Nv9SE-STSSvB{>*GvmjkZCXT z?f_j=!B-kB-%S%0H3Y6v!7^5Yd<&I|MMB;Z6jb2%yKeTccIN)Sx$&8K{`k&!zW;ol z`OZ8rs*@8`&c;JF5K?3RngG$yQc})Dc|qP}Dsm5ehAgEclfGm;JbtkgO84-68YdB4 zsDm<_xB0<`_Mfe5Ehc<(^ZmG2x5R<(yD|IUh}iq9FWN?(5XZU@N2~<_h=?GsN5^Ot zwZ0W$-7K4dx71_oJoDV-Z!~|)54Ez0bAf6;KQnhu>)7(1gnesUs!s~_(bLOL?&h|z z2j;(5SZe|~f%H?`Jhc>LGeZuYOm4*@|JfNVe2#S*tlw;)X@adgK;_GgnB@`IN$()O zeJdVTF+(3Gx@-!k#510$y2lYkS(vN9#GrQLCa;-mV?qZU{;kwK$Rv{L6Jf{#Z20Qb z@yz}Ms3R|)&tr)^yDbq#d`(A0X*cavzLNvxIgvD1VK$rO z1ND7TxV7GM&y8Z#1w(&&^>=;7g{6-m?@l{8KJ!Tb_9_8)`n-#{^gDTjuU|duC!D&$ zw-g&1I=zp?$hSx>UPRYfcB-iNgN=ul&jpT^l}H&s&`y6Z9O@)lFlKin4*F>!XjY}z zH%zu?*}+&9T@cR1r$2t?naFsn7j8!KH0(v!d&4UQ6AAXi+Ao0GE89WI0~#d}C&YB` zQNqqK-`b;Z?K3p61O`q$?5kAp0>k|^;cYVc*)iv>>G+ztPx_vc2X@W83hHPAj!b1r zY6XV1#4Z~|Tv`Oi4Jc0kf>D_H1)flpYK!H<%0qTJsVSQBvck24`sm8=TEiXV>bV)u zl-(02z{HCOM=U<$%Q$!e#;Zh2T~=hF>WDWopLklvSs1fCy}Odu<l`9%z?r4c5`xf7S%aU z&E+*8v}xS`Qt75B{U2YB@TN`IA6TdzSLAF|!jZ$}ahX?GtJAfpeUB+= zijU8Z&02?n0K&<(kM_cNQj~1IX?b$W(*0xP z-_i?z5rOsTi%Tnl`>$~7v^=gl6@7L7z-8Ol{1CZg!>|2lrzyd9v zfMUoDdQ$?Zb&fbNiXZ`wSA*6!T&fxL@5%ub|6)+N9;(~X%%0oc*_U@JfYDyOz~PA0 zNh(zxZAqoz5$h-3JF6oyry?+E>j#lpG`UEw?qP#1vezN7g2eO#qvYh3)d0) zpTpb;=`8TsO9V3HWC%cFD7c@5SJ^Fl0VbXtImzuDtE!cV% zOQ(}EAFD2kEZ66+ZZ6Nrn7OSF`P#l9ncx;YubSQdZg^d-6IdtkKS!WE7X1D1l(1q5 HWGCd`W!a1I literal 0 HcmV?d00001 diff --git a/tests/j_recover_fast_commit/script b/tests/j_recover_fast_commit/script new file mode 100755 index 00000000..6bdb8258 --- /dev/null +++ b/tests/j_recover_fast_commit/script @@ -0,0 +1,26 @@ +#!/bin/bash + +FSCK_OPT=-fy +IMAGE=$test_dir/image.gz +CMDS=$test_dir/commands + +gunzip < $IMAGE > $TMPFILE + +# Run fsck to fix things? +EXP=$test_dir/expect +OUT=$test_name.log + +cp $TMPFILE /tmp/debugthis +$FSCK $FSCK_OPT -E journal_only -N test_filesys $TMPFILE 2>&1 | head -n 1000 | tail -n +2 > $OUT +echo "Exit status is $?" >> $OUT + +$DEBUGFS -f $CMDS $TMPFILE >> $OUT 2>/dev/null + +# Figure out what happened +if cmp -s $EXP $OUT; then + echo "$test_name: $test_description: ok" + touch $test_name.ok +else + echo "$test_name: $test_description: failed" + diff -u $EXP $OUT >> $test_name.failed +fi From patchwork Fri Nov 20 19:16:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: harshad shirwadkar X-Patchwork-Id: 1404075 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=nfLhP9Yc; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cd5rz4Dx0z9sVH for ; Sat, 21 Nov 2020 06:17:07 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732288AbgKTTQp (ORCPT ); Fri, 20 Nov 2020 14:16:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732276AbgKTTQo (ORCPT ); Fri, 20 Nov 2020 14:16:44 -0500 Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47CE2C0613CF for ; Fri, 20 Nov 2020 11:16:44 -0800 (PST) Received: by mail-pf1-x434.google.com with SMTP id v12so8767848pfm.13 for ; Fri, 20 Nov 2020 11:16:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=z0hzlz8HwLKz6qwhsD2BujHz+YW0z2tjXKA4WWmM0ew=; b=nfLhP9YcWg8HKjYImav9iw0M9S2Q22KSsxEaDF5DHCM1pdZ89XPCmtFO529MiEodTg BWdGxNMVXDxjaiFVqnc9qxZcWR8EzBI/JShFsk1f+X4tjNH+mFm424i+8ggjPcqclRS1 cg5Kds3GpStcDwLUS3RQhMHiujpAUxeAm+8h6qVW2oEoRv5l5m89Z2tbaKOET0dZP8Mf 1IHBL9obh6tFRnUM5sXcHUPOW2ZvTzAQv128tmRekFDgEiMPjlMyZmbd5clVxBxS/9iT otu4hA0/ZfHckSfnW90qupqe1HDS1Ypy3sxEgosEYOyhnDpdV0W8s7+Rg67VnBZrg/Fe Rv6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z0hzlz8HwLKz6qwhsD2BujHz+YW0z2tjXKA4WWmM0ew=; b=rAEezlfjG8ryRLiwjyXOPUe2eqhZ+YYOPEbSN0tOd/WAyAGCtTMcThBy70CxaZlVAO s4Y2zcfdeQEmoRA3/fy0bCrS2OPfZeLB2gPb/Ck4Oi8eBjrJL+eVAivFP8PnYDM8G/Na jUX7Yq89HcLy6zSY6D37V4E0nfyvTcIIDLuTP6cc3E82nUOL5eK3BZr+ywcseDRV3VU4 k32IHoZWE+xuAtpofCY47JV3xm+Aa4spQr+4FlaTQDzMM1I6iArMpN81MIOOH+nEsIdT rQk+rQud9PQm+j9C2wQ/sfOGwiAOKQBSVVt4U+vvA/R6YHFgnY55VjpfsVmWasY2aeMT X9ug== X-Gm-Message-State: AOAM530jXctOyA3XJ4arWH71AejRRmfoCgDitMNjO7yaqIHO5kYVofSY qDTjPuJ+byB98TRwtv5acYDUpELhl1k= X-Google-Smtp-Source: ABdhPJxfByswgPZkwwLZOGVeHzsyIfBPUlWdxqA1RlOJoay4qMyIIxWNhZ2U/Y4NMfcRDnydvPIQMg== X-Received: by 2002:a05:6a00:80e:b029:196:1cad:b64 with SMTP id m14-20020a056a00080eb02901961cad0b64mr15314605pfk.5.1605899803059; Fri, 20 Nov 2020 11:16:43 -0800 (PST) Received: from harshads-520.kir.corp.google.com ([2620:15c:17:10:a6ae:11ff:fe11:86a2]) by smtp.googlemail.com with ESMTPSA id o9sm4370480pjr.2.2020.11.20.11.16.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Nov 2020 11:16:41 -0800 (PST) From: Harshad Shirwadkar To: linux-ext4@vger.kernel.org Cc: tytso@mit.edu, Harshad Shirwadkar Subject: [PATCH 15/15] ext4: fix tests to account for new dumpe2fs output Date: Fri, 20 Nov 2020 11:16:06 -0800 Message-Id: <20201120191606.2224881-16-harshadshirwadkar@gmail.com> X-Mailer: git-send-email 2.29.2.454.gaff20da3a2-goog In-Reply-To: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> References: <20201120191606.2224881-1-harshadshirwadkar@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org dumpe2fs tool now is capable of reporting number of fast commit blocks. There were slight changes in the output of dumpe2fs outside of fast commits. This patch fixes the regression tests to expect the new output. Signed-off-by: Harshad Shirwadkar --- tests/d_corrupt_journal_nr_users/expect | 6 ++++-- tests/f_jnl_errno/expect.0 | 6 ++++-- tests/f_opt_extent/expect | 2 +- tests/i_bitmaps/expect | 8 +++++--- tests/j_ext_dumpe2fs/expect | 6 ++++-- tests/m_bigjournal/expect.1 | 6 ++++-- tests/m_extent_journal/expect.1 | 6 ++++-- tests/m_resize_inode_meta_bg/expect.1 | 6 ++++-- tests/m_rootdir/expect | 6 ++++-- tests/r_32to64bit/expect | 6 +++--- tests/r_32to64bit_meta/expect | 4 ++-- tests/r_32to64bit_move_itable/expect | 8 ++++---- tests/r_64to32bit/expect | 6 +++--- tests/r_64to32bit_meta/expect | 4 ++-- tests/r_move_itable_nostride/expect | 6 ++++-- tests/r_move_itable_realloc/expect | 6 ++++-- tests/t_disable_mcsum/expect | 4 ++-- tests/t_disable_mcsum_noinitbg/expect | 6 +++--- tests/t_disable_mcsum_yesinitbg/expect | 4 ++-- tests/t_enable_mcsum/expect | 6 +++--- tests/t_enable_mcsum_ext3/expect | 10 +++++----- tests/t_enable_mcsum_initbg/expect | 6 +++--- 22 files changed, 74 insertions(+), 54 deletions(-) diff --git a/tests/d_corrupt_journal_nr_users/expect b/tests/d_corrupt_journal_nr_users/expect index cdfb49a0..656d35c3 100644 --- a/tests/d_corrupt_journal_nr_users/expect +++ b/tests/d_corrupt_journal_nr_users/expect @@ -34,8 +34,10 @@ Default directory hash: half_md4 Journal backup: inode blocks Checksum type: crc32c Journal features: (none) -Journal size: 4096k -Journal length: 1024 +Total journal size: 4096k +Total journal blocks: 1024 +Max transaction length: 1024 +Fast commit length: 0 Journal sequence: 0x00000001 Journal start: 0 Journal number of users: 9999 diff --git a/tests/f_jnl_errno/expect.0 b/tests/f_jnl_errno/expect.0 index 2a9426d9..96fb01a7 100644 --- a/tests/f_jnl_errno/expect.0 +++ b/tests/f_jnl_errno/expect.0 @@ -31,8 +31,10 @@ Journal inode: 8 Default directory hash: half_md4 Journal backup: inode blocks Journal features: (none) -Journal size: 1024k -Journal length: 1024 +Total journal size: 1024k +Total journal blocks: 1024 +Max transaction length: 1024 +Fast commit length: 0 Journal sequence: 0x00000005 Journal start: 0 diff --git a/tests/f_opt_extent/expect b/tests/f_opt_extent/expect index 6d1f9d11..9d2021e8 100644 --- a/tests/f_opt_extent/expect +++ b/tests/f_opt_extent/expect @@ -35,7 +35,7 @@ Change in FS metadata: Free inodes: 65047 First block: 1 Block size: 1024 -@@ -48,8 +48,8 @@ +@@ -50,8 +50,8 @@ Block bitmap at 262 (+261) Inode bitmap at 278 (+277) Inode table at 294-549 (+293) diff --git a/tests/i_bitmaps/expect b/tests/i_bitmaps/expect index fb9d8f1f..6199bb7c 100644 --- a/tests/i_bitmaps/expect +++ b/tests/i_bitmaps/expect @@ -1,6 +1,8 @@ -46,50d45 +46,52d45 < Journal features: (none) -< Journal size: 1024k -< Journal length: 1024 +< Total journal size: 1024k +< Total journal blocks: 1024 +< Max transaction length: 1024 +< Fast commit length: 0 < Journal sequence: 0x00000001 < Journal start: 0 diff --git a/tests/j_ext_dumpe2fs/expect b/tests/j_ext_dumpe2fs/expect index db77a36d..2838bbd1 100644 --- a/tests/j_ext_dumpe2fs/expect +++ b/tests/j_ext_dumpe2fs/expect @@ -43,8 +43,10 @@ Desired extra isize: 28 Default directory hash: half_md4 Checksum type: crc32c Journal features: journal_64bit journal_checksum_v3 -Journal size: 2048k -Journal length: 2048 +Total journal size: 2048k +Total journal blocks: 2048 +Max transaction length: 2048 +Fast commit length: 0 Journal first block: 3 Journal sequence: 0x00000003 Journal start: 0 diff --git a/tests/m_bigjournal/expect.1 b/tests/m_bigjournal/expect.1 index 80f71d1f..eb0e3bc3 100644 --- a/tests/m_bigjournal/expect.1 +++ b/tests/m_bigjournal/expect.1 @@ -42,8 +42,10 @@ Journal inode: 8 Default directory hash: half_md4 Journal backup: inode blocks Journal features: (none) -Journal size: 5000M -Journal length: 1280000 +Total journal size: 5000M +Total journal blocks: 1280000 +Max transaction length: 1280000 +Fast commit length: 0 Journal sequence: 0x00000001 Journal start: 0 diff --git a/tests/m_extent_journal/expect.1 b/tests/m_extent_journal/expect.1 index cfc052a8..9a9219e9 100644 --- a/tests/m_extent_journal/expect.1 +++ b/tests/m_extent_journal/expect.1 @@ -48,8 +48,10 @@ Journal inode: 8 Default directory hash: half_md4 Journal backup: inode blocks Journal features: (none) -Journal size: 4096k -Journal length: 4096 +Total journal size: 4096k +Total journal blocks: 4096 +Max transaction length: 4096 +Fast commit length: 0 Journal sequence: 0x00000001 Journal start: 0 diff --git a/tests/m_resize_inode_meta_bg/expect.1 b/tests/m_resize_inode_meta_bg/expect.1 index 6a7f3993..7feaed9d 100644 --- a/tests/m_resize_inode_meta_bg/expect.1 +++ b/tests/m_resize_inode_meta_bg/expect.1 @@ -51,8 +51,10 @@ Journal inode: 8 Default directory hash: half_md4 Journal backup: inode blocks Journal features: (none) -Journal size: 4096k -Journal length: 1024 +Total journal size: 4096k +Total journal blocks: 1024 +Max transaction length: 1024 +Fast commit length: 0 Journal sequence: 0x00000001 Journal start: 0 diff --git a/tests/m_rootdir/expect b/tests/m_rootdir/expect index 113ffc64..dbc79772 100644 --- a/tests/m_rootdir/expect +++ b/tests/m_rootdir/expect @@ -36,8 +36,10 @@ Default directory hash: half_md4 Journal backup: inode blocks Checksum type: crc32c Journal features: (none) -Journal size: 1024k -Journal length: 1024 +Total journal size: 1024k +Total journal blocks: 1024 +Max transaction length: 1024 +Fast commit length: 0 Journal sequence: 0x00000001 Journal start: 0 diff --git a/tests/r_32to64bit/expect b/tests/r_32to64bit/expect index c6816b7f..de573b3e 100644 --- a/tests/r_32to64bit/expect +++ b/tests/r_32to64bit/expect @@ -47,7 +47,7 @@ Change in FS metadata: Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 1024 -@@ -41,16 +41,16 @@ +@@ -43,16 +43,16 @@ group:block:super:gdt:bbitmap:ibitmap:itable @@ -70,7 +70,7 @@ Change in FS metadata: 10:81921:-1:-1:270:286:2852 11:90113:-1:-1:271:287:3108 12:98305:-1:-1:272:288:3364 -@@ -66,9 +66,9 @@ +@@ -68,9 +68,9 @@ 22:180225:-1:-1:131079:131095:132641 23:188417:-1:-1:131080:131096:132897 24:196609:-1:-1:131081:131097:133153 @@ -82,7 +82,7 @@ Change in FS metadata: 28:229377:-1:-1:131085:131101:134177 29:237569:-1:-1:131086:131102:134433 30:245761:-1:-1:131087:131103:134689 -@@ -90,7 +90,7 @@ +@@ -92,7 +92,7 @@ 46:376833:-1:-1:262159:262175:265761 47:385025:-1:-1:262160:262176:266017 48:393217:-1:-1:393217:393233:393249 diff --git a/tests/r_32to64bit_meta/expect b/tests/r_32to64bit_meta/expect index c4f39266..70babbd9 100644 --- a/tests/r_32to64bit_meta/expect +++ b/tests/r_32to64bit_meta/expect @@ -46,7 +46,7 @@ Change in FS metadata: Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 1024 -@@ -55,9 +55,9 @@ +@@ -57,9 +57,9 @@ 12:98305:-1:-1:15:31:3107 13:106497:-1:-1:16:32:3363 14:114689:-1:-1:17:33:3619 @@ -59,7 +59,7 @@ Change in FS metadata: 18:147457:-1:-1:131075:131091:131617 19:155649:-1:-1:131076:131092:131873 20:163841:-1:-1:131077:131093:132129 -@@ -87,9 +87,9 @@ +@@ -89,9 +89,9 @@ 44:360449:-1:-1:262158:262174:265250 45:368641:-1:-1:262159:262175:265506 46:376833:-1:-1:262160:262176:265762 diff --git a/tests/r_32to64bit_move_itable/expect b/tests/r_32to64bit_move_itable/expect index 0a3b78e7..a1d2aebc 100644 --- a/tests/r_32to64bit_move_itable/expect +++ b/tests/r_32to64bit_move_itable/expect @@ -46,7 +46,7 @@ Change in FS metadata: Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 1024 -@@ -39,16 +39,16 @@ +@@ -41,16 +41,16 @@ group:block:super:gdt:bbitmap:ibitmap:itable @@ -69,7 +69,7 @@ Change in FS metadata: 10:81921:-1:-1:81921:81922:81923 11:90113:-1:-1:90113:90114:90115 12:98305:-1:-1:98305:98306:98307 -@@ -64,9 +64,9 @@ +@@ -66,9 +66,9 @@ 22:180225:-1:-1:180225:180226:180227 23:188417:-1:-1:188417:188418:188419 24:196609:-1:-1:196609:196610:196611 @@ -81,7 +81,7 @@ Change in FS metadata: 28:229377:-1:-1:229377:229378:229379 29:237569:-1:-1:237569:237570:237571 30:245761:-1:-1:245761:245762:245763 -@@ -88,7 +88,7 @@ +@@ -90,7 +90,7 @@ 46:376833:-1:-1:376833:376834:376835 47:385025:-1:-1:385025:385026:385027 48:393217:-1:-1:393217:393218:393219 @@ -90,7 +90,7 @@ Change in FS metadata: 50:409601:-1:-1:409601:409602:409603 51:417793:-1:-1:417793:417794:417795 52:425985:-1:-1:425985:425986:425987 -@@ -120,7 +120,7 @@ +@@ -122,7 +122,7 @@ 78:638977:-1:-1:638977:638978:638979 79:647169:-1:-1:647169:647170:647171 80:655361:-1:-1:655361:655362:655363 diff --git a/tests/r_64to32bit/expect b/tests/r_64to32bit/expect index 7dff2a05..eb609fbd 100644 --- a/tests/r_64to32bit/expect +++ b/tests/r_64to32bit/expect @@ -46,7 +46,7 @@ Change in FS metadata: Reserved GDT blocks: 256 Blocks per group: 8192 Fragments per group: 8192 -@@ -42,16 +40,16 @@ +@@ -44,16 +42,16 @@ group:block:super:gdt:bbitmap:ibitmap:itable @@ -69,7 +69,7 @@ Change in FS metadata: 10:81921:-1:-1:272:288:2854 11:90113:-1:-1:273:289:3110 12:98305:-1:-1:274:290:3366 -@@ -67,9 +65,9 @@ +@@ -69,9 +67,9 @@ 22:180225:-1:-1:131079:131095:132641 23:188417:-1:-1:131080:131096:132897 24:196609:-1:-1:131081:131097:133153 @@ -81,7 +81,7 @@ Change in FS metadata: 28:229377:-1:-1:131085:131101:134177 29:237569:-1:-1:131086:131102:134433 30:245761:-1:-1:131087:131103:134689 -@@ -91,7 +89,7 @@ +@@ -93,7 +91,7 @@ 46:376833:-1:-1:262159:262175:265761 47:385025:-1:-1:262160:262176:266017 48:393217:-1:-1:393217:393233:393249 diff --git a/tests/r_64to32bit_meta/expect b/tests/r_64to32bit_meta/expect index b17a8784..70655909 100644 --- a/tests/r_64to32bit_meta/expect +++ b/tests/r_64to32bit_meta/expect @@ -46,7 +46,7 @@ Change in FS metadata: Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 1024 -@@ -56,9 +54,9 @@ +@@ -58,9 +56,9 @@ 12:98305:-1:-1:15:31:3107 13:106497:-1:-1:16:32:3363 14:114689:-1:-1:17:33:3619 @@ -59,7 +59,7 @@ Change in FS metadata: 18:147457:-1:-1:131076:131092:131618 19:155649:-1:-1:131077:131093:131874 20:163841:-1:-1:131078:131094:132130 -@@ -88,9 +86,9 @@ +@@ -90,9 +88,9 @@ 44:360449:-1:-1:262158:262174:265250 45:368641:-1:-1:262159:262175:265506 46:376833:-1:-1:262160:262176:265762 diff --git a/tests/r_move_itable_nostride/expect b/tests/r_move_itable_nostride/expect index 098cbfc5..74c2cc0a 100644 --- a/tests/r_move_itable_nostride/expect +++ b/tests/r_move_itable_nostride/expect @@ -52,8 +52,10 @@ Journal inode: 8 Default directory hash: half_md4 Journal backup: inode blocks Journal features: (none) -Journal size: 16M -Journal length: 16384 +Total journal size: 16M +Total journal blocks: 16384 +Max transaction length: 16384 +Fast commit length: 0 Journal sequence: 0x00000001 Journal start: 0 diff --git a/tests/r_move_itable_realloc/expect b/tests/r_move_itable_realloc/expect index f9a7f515..67f2fe4a 100644 --- a/tests/r_move_itable_realloc/expect +++ b/tests/r_move_itable_realloc/expect @@ -51,8 +51,10 @@ Journal inode: 8 Default directory hash: half_md4 Journal backup: inode blocks Journal features: (none) -Journal size: 16M -Journal length: 16384 +Total journal size: 16M +Total journal blocks: 16384 +Max transaction length: 16384 +Fast commit length: 0 Journal sequence: 0x00000001 Journal start: 0 diff --git a/tests/t_disable_mcsum/expect b/tests/t_disable_mcsum/expect index 3341ad71..bfa21b89 100644 --- a/tests/t_disable_mcsum/expect +++ b/tests/t_disable_mcsum/expect @@ -34,8 +34,8 @@ Change in FS metadata: Journal backup: inode blocks -Checksum type: crc32c Journal features: (none) - Journal size: 16M - Journal length: 16384 + Total journal size: 16M + Total journal blocks: 16384 Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity diff --git a/tests/t_disable_mcsum_noinitbg/expect b/tests/t_disable_mcsum_noinitbg/expect index 62eca4e9..fe61fcbc 100644 --- a/tests/t_disable_mcsum_noinitbg/expect +++ b/tests/t_disable_mcsum_noinitbg/expect @@ -34,9 +34,9 @@ Change in FS metadata: Journal backup: inode blocks -Checksum type: crc32c Journal features: (none) - Journal size: 16M - Journal length: 16384 -@@ -48,18 +47,18 @@ + Total journal size: 16M + Total journal blocks: 16384 +@@ -50,18 +49,18 @@ Block bitmap at 262 (+261) Inode bitmap at 278 (+277) Inode table at 294-549 (+293) diff --git a/tests/t_disable_mcsum_yesinitbg/expect b/tests/t_disable_mcsum_yesinitbg/expect index 7e3485fe..b9062489 100644 --- a/tests/t_disable_mcsum_yesinitbg/expect +++ b/tests/t_disable_mcsum_yesinitbg/expect @@ -34,8 +34,8 @@ Change in FS metadata: Journal backup: inode blocks -Checksum type: crc32c Journal features: (none) - Journal size: 16M - Journal length: 16384 + Total journal size: 16M + Total journal blocks: 16384 Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity diff --git a/tests/t_enable_mcsum/expect b/tests/t_enable_mcsum/expect index cb0aef62..fcb0ed16 100644 --- a/tests/t_enable_mcsum/expect +++ b/tests/t_enable_mcsum/expect @@ -58,9 +58,9 @@ Change in FS metadata: Journal backup: inode blocks +Checksum type: crc32c Journal features: (none) - Journal size: 16M - Journal length: 16384 -@@ -47,8 +48,8 @@ + Total journal size: 16M + Total journal blocks: 16384 +@@ -49,8 +50,8 @@ Block bitmap at 262 (+261) Inode bitmap at 278 (+277) Inode table at 294-549 (+293) diff --git a/tests/t_enable_mcsum_ext3/expect b/tests/t_enable_mcsum_ext3/expect index 11c5a26d..549e60e9 100644 --- a/tests/t_enable_mcsum_ext3/expect +++ b/tests/t_enable_mcsum_ext3/expect @@ -37,9 +37,9 @@ Change in FS metadata: Journal backup: inode blocks +Checksum type: crc32c Journal features: (none) - Journal size: 16M - Journal length: 16384 -@@ -37,7 +38,7 @@ + Total journal size: 16M + Total journal blocks: 16384 +@@ -39,7 +40,7 @@ Journal start: 0 @@ -48,7 +48,7 @@ Change in FS metadata: Primary superblock at 1, Group descriptors at 2-3 Reserved GDT blocks at 4-259 Block bitmap at 260 (+259) -@@ -46,7 +47,7 @@ +@@ -48,7 +49,7 @@ 0 free blocks, 1013 free inodes, 2 directories Free blocks: Free inodes: 12-1024 @@ -57,7 +57,7 @@ Change in FS metadata: Backup superblock at 8193, Group descriptors at 8194-8195 Reserved GDT blocks at 8196-8451 Block bitmap at 8452 (+259) -@@ -55,6 +56,6 @@ +@@ -57,6 +58,6 @@ 0 free blocks, 1024 free inodes, 0 directories Free blocks: Free inodes: 1025-2048 diff --git a/tests/t_enable_mcsum_initbg/expect b/tests/t_enable_mcsum_initbg/expect index a37648bf..987141f1 100644 --- a/tests/t_enable_mcsum_initbg/expect +++ b/tests/t_enable_mcsum_initbg/expect @@ -58,9 +58,9 @@ Change in FS metadata: Journal backup: inode blocks +Checksum type: crc32c Journal features: (none) - Journal size: 16M - Journal length: 16384 -@@ -41,24 +42,24 @@ + Total journal size: 16M + Total journal blocks: 16384 +@@ -43,24 +44,24 @@ Journal start: 0