From patchwork Wed Apr 8 10:44:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267861 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=lkho6Ftn; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1C25K13z9sQx for ; Wed, 8 Apr 2020 20:45:34 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728100AbgDHKpe (ORCPT ); Wed, 8 Apr 2020 06:45:34 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:43049 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKpe (ORCPT ); Wed, 8 Apr 2020 06:45:34 -0400 Received: by mail-pf1-f194.google.com with SMTP id l1so291911pff.10 for ; Wed, 08 Apr 2020 03:45:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=lc8WOSneON5fyvmGzcBdv7Ou/aKbYA1P3kpRhP1zndU=; b=lkho6Ftn499rOzvVF6zufVTrFaeIW8103fVXIDeQCZGRcmKc8/2LvDTOzGf5uz8+uN H3pDz95z9cVhvVqKeVfLLdl+9/gAPDCZQfx97ihoZVWGnGpKH/Uqiydt1YUsmT0AnD+l LKOWGU4jX2l2b/rTqwVg5wLo3ytaAoJJhZnHyXt9Iw9EGj+OzgWrv2Nf2LS1P9InXoYB j/EzVwEvaZUYZ2hw/1mXVEVZJI1x10QkxAL0lMZBScLw2aTDgm0p/meCeB24fOgbPRtE mHsPUnKkURc20BDX7ICLVRtkn573OS/DcriKmDPsuYQHi6SnJ0RgT5cWuWWqumcJIkfw 8LwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=lc8WOSneON5fyvmGzcBdv7Ou/aKbYA1P3kpRhP1zndU=; b=RbFEcwT+wz6AHZHplMre4G6z7pJ1/7p+2XwFuYXBAuPBz/toFG4H4diebleRGhWXBJ 5Td0BuzZMMz2Er6TA9T6Jd0BtI5wRzNmF/1dVLlUtLNCTXKx9AvPAnJkuwazQQ6oFVwr zPNGTrlcUstaXiAg58zy7UXUD5/a1WjI4Bi3hVROS1hhKdYDR0FMLgJZ6/EGcZlg2f8d x8wbORsfnfyozaFtkHcf274kK3a0s0A7bEsnPbHkCjJn83MCzoJVU+nmZDbQTZeMPybv IMUoJSCBWQfAeG9janATZRaDDW+MTCzXLyJao0Av4zze7aZ0RhF4j4Zbx4DO7JPhr8kj 5pUg== X-Gm-Message-State: AGi0PuYQPVim4xFHA8yUuy+fibwlzfFywWWLWeHtbiGO8iUSUWWa7a/q GAdyyCQQqJDcb6Sul7b1zey+LHSWoSU= X-Google-Smtp-Source: APiQypJDSi1BzqFmH5i/Fb79vmc40U97hykhQl3GVM65pZU/SSSZni2Er4R8uk79a2tHm8nYjTzlGg== X-Received: by 2002:a63:2842:: with SMTP id o63mr6625114pgo.427.1586342732318; Wed, 08 Apr 2020 03:45:32 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:31 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 01/46] e2fsck: cleanup struct e2fsck_struct Date: Wed, 8 Apr 2020 19:44:29 +0900 Message-Id: <1586342714-12536-2-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Fields of "struct e2fsck_struct" are seperated into different types according to how these fields will be used when parallel fsck is enabled. remove unused @abort_code Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 279 +++++++++++++++++++++++++----------------------- 1 file changed, 143 insertions(+), 136 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 954bc982..24a9d0a8 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -227,91 +227,155 @@ typedef struct e2fsck_struct *e2fsck_t; #define MAX_EXTENT_DEPTH_COUNT 5 struct e2fsck_struct { - ext2_filsys fs; - const char *program_name; - char *filesystem_name; - char *device_name; - char *io_options; - FILE *logf; - char *log_fn; - FILE *problem_logf; - char *problem_log_fn; - int flags; /* E2fsck internal flags */ - int options; - int blocksize; /* blocksize */ - blk64_t use_superblock; /* sb requested by user */ - blk64_t superblock; /* sb used to open fs */ - blk64_t num_blocks; /* Total number of blocks */ - blk64_t free_blocks; - ino_t free_inodes; - int mount_flags; - int openfs_flags; - blkid_cache blkid; /* blkid cache */ - + /* ---- Following fields are never updated during the pass1 ---- */ + const char *program_name; + char *filesystem_name; + char *device_name; + char *io_options; + int options; /* E2F_OPT_* flags */ + int blocksize; /* blocksize */ + blk64_t use_superblock; /* sb requested by user */ + blk64_t superblock; /* sb used to open fs */ + blk64_t num_blocks; /* Total number of blocks */ + blk64_t free_blocks; + ino_t free_inodes; + int mount_flags; + int openfs_flags; + blkid_cache blkid; /* blkid cache */ #ifdef HAVE_SETJMP_H - jmp_buf abort_loc; + jmp_buf abort_loc; #endif - unsigned long abort_code; +#ifdef RESOURCE_TRACK + /* + * For timing purposes + */ + struct resource_track global_rtrack; +#endif + int bad_lost_and_found; - int (*progress)(e2fsck_t ctx, int pass, unsigned long cur, - unsigned long max); + /* + * Tuning parameters + */ + int process_inode_size; + int inode_buffer_blocks; + unsigned int htree_slack_percentage; + + /* + * ext3 journal support + */ + io_channel journal_io; + char *journal_name; + /* misc fields */ + time_t now; + /* For working around buggy init scripts */ + time_t time_fudge; + + int ext_attr_ver; + int blocks_per_page; - ext2fs_inode_bitmap inode_used_map; /* Inodes which are in use */ - ext2fs_inode_bitmap inode_bad_map; /* Inodes which are bad somehow */ - ext2fs_inode_bitmap inode_dir_map; /* Inodes which are directories */ - ext2fs_inode_bitmap inode_bb_map; /* Inodes which are in bad blocks */ - ext2fs_inode_bitmap inode_imagic_map; /* AFS inodes */ - ext2fs_inode_bitmap inode_reg_map; /* Inodes which are regular files*/ + /* Are we connected directly to a tty? */ + int interactive; + char start_meta[2], stop_meta[2]; + /* + * For the use of callers of the e2fsck functions; not used by + * e2fsck functions themselves. + */ + void *priv_data; + /* Undo file */ + char *undo_file; + /* How much are we allowed to readahead? */ + unsigned long long readahead_kb; - ext2fs_block_bitmap block_found_map; /* Blocks which are in use */ - ext2fs_block_bitmap block_dup_map; /* Blks referenced more than once */ - ext2fs_block_bitmap block_ea_map; /* Blocks which are used by EA's */ + /* ---- Following fields are shared by different threads for pass1 -*/ + /* E2fsck internal flags */ + int flags; + /* + * How we display the progress update (for unix) + */ + int progress_fd; + int progress_pos; + int progress_last_percent; + unsigned int progress_last_time; + int (*progress)(e2fsck_t ctx, int pass, unsigned long cur, + unsigned long max); + /* Metadata blocks */ + ext2fs_block_bitmap block_metadata_map; + profile_t profile; + /* Reserve blocks for root and l+f re-creation */ + blk64_t root_repair_block, lnf_repair_block; + /* + * Location of the lost and found directory + */ + ext2_ino_t lost_and_found; + + /* ---- Following fields are seperated for each thread for pass1- */ + ext2_filsys fs; + FILE *logf; + char *log_fn; + FILE *problem_logf; + char *problem_log_fn; + + /* Inodes which are in use */ + ext2fs_inode_bitmap inode_used_map; + /* Inodes which are bad somehow */ + ext2fs_inode_bitmap inode_bad_map; + /* Inodes which are directories */ + ext2fs_inode_bitmap inode_dir_map; + /* Inodes which are in bad blocks */ + ext2fs_inode_bitmap inode_bb_map; + /* AFS inodes */ + ext2fs_inode_bitmap inode_imagic_map; + /* Inodes which are regular files */ + ext2fs_inode_bitmap inode_reg_map; + /* Inodes to rebuild extent trees */ + ext2fs_inode_bitmap inodes_to_rebuild; + /* Blocks which are in use */ + ext2fs_block_bitmap block_found_map; + /* Blks referenced more than once */ + ext2fs_block_bitmap block_dup_map; + /* Blocks which are used by EA's */ + ext2fs_block_bitmap block_ea_map; /* * Inode count arrays */ - ext2_icount_t inode_count; - ext2_icount_t inode_link_info; + ext2_icount_t inode_count; + ext2_icount_t inode_link_info; - ext2_refcount_t refcount; - ext2_refcount_t refcount_extra; + ext2_refcount_t refcount; + ext2_refcount_t refcount_extra; /* * Quota blocks and inodes to be charged for each ea block. */ - ext2_refcount_t ea_block_quota_blocks; - ext2_refcount_t ea_block_quota_inodes; + ext2_refcount_t ea_block_quota_blocks; + ext2_refcount_t ea_block_quota_inodes; /* * ea_inode references from attr entries. */ - ext2_refcount_t ea_inode_refs; + ext2_refcount_t ea_inode_refs; /* * Array of flags indicating whether an inode bitmap, block * bitmap, or inode table is invalid */ - int *invalid_inode_bitmap_flag; - int *invalid_block_bitmap_flag; - int *invalid_inode_table_flag; - int invalid_bitmaps; /* There are invalid bitmaps/itable */ + int *invalid_inode_bitmap_flag; + int *invalid_block_bitmap_flag; + int *invalid_inode_table_flag; + /* There are invalid bitmaps/itable */ + int invalid_bitmaps; /* * Block buffer */ - char *block_buf; + char *block_buf; /* * For pass1_check_directory and pass1_get_blocks */ - ext2_ino_t stashed_ino; - struct ext2_inode *stashed_inode; - - /* - * Location of the lost and found directory - */ - ext2_ino_t lost_and_found; - int bad_lost_and_found; + ext2_ino_t stashed_ino; + struct ext2_inode *stashed_inode; /* * Directory information @@ -321,103 +385,46 @@ struct e2fsck_struct { /* * Indexed directory information */ - int dx_dir_info_count; - int dx_dir_info_size; - struct dx_dir_info *dx_dir_info; + int dx_dir_info_count; + int dx_dir_info_size; + struct dx_dir_info *dx_dir_info; /* * Directories to hash */ - ext2_u32_list dirs_to_hash; + ext2_u32_list dirs_to_hash; /* * Encrypted file information */ struct encrypted_file_info *encrypted_files; - /* - * Tuning parameters - */ - int process_inode_size; - int inode_buffer_blocks; - unsigned int htree_slack_percentage; - - /* - * ext3 journal support - */ - io_channel journal_io; - char *journal_name; - /* * Ext4 quota support */ - quota_ctx_t qctx; -#ifdef RESOURCE_TRACK - /* - * For timing purposes - */ - struct resource_track global_rtrack; -#endif - - /* - * How we display the progress update (for unix) - */ - int progress_fd; - int progress_pos; - int progress_last_percent; - unsigned int progress_last_time; - int interactive; /* Are we connected directly to a tty? */ - char start_meta[2], stop_meta[2]; - + quota_ctx_t qctx; /* File counts */ - __u32 fs_directory_count; - __u32 fs_regular_count; - __u32 fs_blockdev_count; - __u32 fs_chardev_count; - __u32 fs_links_count; - __u32 fs_symlinks_count; - __u32 fs_fast_symlinks_count; - __u32 fs_fifo_count; - __u32 fs_total_count; - __u32 fs_badblocks_count; - __u32 fs_sockets_count; - __u32 fs_ind_count; - __u32 fs_dind_count; - __u32 fs_tind_count; - __u32 fs_fragmented; - __u32 fs_fragmented_dir; - __u32 large_files; - __u32 fs_ext_attr_inodes; - __u32 fs_ext_attr_blocks; - __u32 extent_depth_count[MAX_EXTENT_DEPTH_COUNT]; + __u32 fs_directory_count; + __u32 fs_regular_count; + __u32 fs_blockdev_count; + __u32 fs_chardev_count; + __u32 fs_links_count; + __u32 fs_symlinks_count; + __u32 fs_fast_symlinks_count; + __u32 fs_fifo_count; + __u32 fs_total_count; + __u32 fs_badblocks_count; + __u32 fs_sockets_count; + __u32 fs_ind_count; + __u32 fs_dind_count; + __u32 fs_tind_count; + __u32 fs_fragmented; + __u32 fs_fragmented_dir; + __u32 large_files; + __u32 fs_ext_attr_inodes; + __u32 fs_ext_attr_blocks; + __u32 extent_depth_count[MAX_EXTENT_DEPTH_COUNT]; - /* misc fields */ - time_t now; - time_t time_fudge; /* For working around buggy init scripts */ - int ext_attr_ver; - profile_t profile; - int blocks_per_page; - - /* Reserve blocks for root and l+f re-creation */ - blk64_t root_repair_block, lnf_repair_block; - - /* - * For the use of callers of the e2fsck functions; not used by - * e2fsck functions themselves. - */ - void *priv_data; - ext2fs_block_bitmap block_metadata_map; /* Metadata blocks */ - - /* How much are we allowed to readahead? */ - unsigned long long readahead_kb; - - /* - * Inodes to rebuild extent trees - */ - ext2fs_inode_bitmap inodes_to_rebuild; - - /* Undo file */ - char *undo_file; }; /* Data structures to evaluate whether an extent tree needs rebuilding. */ From patchwork Wed Apr 8 10:44:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267863 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=IY2wPJxo; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CB2pwSz9sSX for ; Wed, 8 Apr 2020 20:45:42 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728207AbgDHKpl (ORCPT ); Wed, 8 Apr 2020 06:45:41 -0400 Received: from mail-pl1-f196.google.com ([209.85.214.196]:37082 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKpl (ORCPT ); Wed, 8 Apr 2020 06:45:41 -0400 Received: by mail-pl1-f196.google.com with SMTP id x1so2379287plm.4 for ; Wed, 08 Apr 2020 03:45:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0dX2/sseqROJziJSwC1uokaELfEWBTLLfM74NRFO64w=; b=IY2wPJxofvIJ3xtcr1cLoE76Th1dJn4IH4noiAsd8qa6FF7ZCekLntfMrLGw/x3bJA uvSi3iiEHPAC4cc2bhPJ4BDd6Qfo7WUbJj6996RvmBMzSzXFbbY2xkeq72Ym1rkH7va8 Snm2D8H54wlERApTcuMqL9AwfNz9haW6aI0PgA3jFMiyFjxnPQDXqmm7RTpnXH5WGr0a /cLyfoAYPoL7BZOExFA9bXCIm909L0bt3Aj9g2xiT4IxdTu5rtTjGKd2z1mPXrDpTDE/ 8I2GXiIkhanaNNy20wfpyB4X0T/fuAZCBP9sboFvCnnZmscljU2izKdfoIsNBLOab1zD t8fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0dX2/sseqROJziJSwC1uokaELfEWBTLLfM74NRFO64w=; b=AGfFPavOVX7ZGyOWeBS+lOiZYoDB1FsJt+NWdsYmKA18J7ibPssDh77eLvmMiPxpXt 5KI8Jh6AtwJkZNnZmW8aM8rgu0g/ZeuAg9dwNrLsgFDeAa3XkNz+87L1laydWtvuDIsD WkfazH6/2M8x20Vh9NENf9d+6Vtx/UbgEnZBDy50RtndOII4sxBryzF/hlTB2vt0Wagu l38/+KxS5p9vYsGVNEVeFGbW0YTxOiS+k3WTruK7ECvYQzwmmgeyxOgjxnYGkcuMcqzJ p47ff/maewg3yOoZqPK5ePsPFjIBcVFdZvDh+C31yycAWIOOxiguucFXYCf2n5GINSld WRpQ== X-Gm-Message-State: AGi0PuZZlgn2W3LS7IxG9LbVIsUFnsxeKAgoUAhfZg4LaV8kMO700rYK vFIvWYFmdXqhrXXO5O/Frcnu9kf4/n0= X-Google-Smtp-Source: APiQypIefeQ5oaDm347jLKkbgYZo9QebPy7tXmITSst/ZQJnXPhU8EXLU2Giuz0NUXH8Jg3XSOCgfw== X-Received: by 2002:a17:90b:1b04:: with SMTP id nu4mr4582170pjb.81.1586342737362; Wed, 08 Apr 2020 03:45:37 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:36 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 02/46] e2fsck: add -m option for multithread Date: Wed, 8 Apr 2020 19:44:30 +0900 Message-Id: <1586342714-12536-3-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi -m option is added but no actual functionality is added. This patch only adds the logic that when -m is specified, one of -p/-y/-n options should be specified. And when -m is specified, -C shouldn't be specified and the completion progress report won't be triggered by sending SIGUSR1/SIGUSR2 signals. This simplifies the implementation of multi-thread fsck in the future. Completion progress support with multi-thread fsck will be added back after multi-thread fsck implementation is finished. Right now, disable it to simplify the implementation of multi-thread fsck. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 1 + e2fsck/unix.c | 30 ++++++++++++++++++++----- tests/f_multithread/expect.1 | 23 +++++++++++++++++++ tests/f_multithread/expect.2 | 7 ++++++ tests/f_multithread/image.gz | 1 + tests/f_multithread/name | 1 + tests/f_multithread/script | 4 ++++ tests/f_multithread_completion/expect.1 | 2 ++ tests/f_multithread_completion/expect.2 | 23 +++++++++++++++++++ tests/f_multithread_completion/image.gz | 1 + tests/f_multithread_completion/name | 1 + tests/f_multithread_completion/script | 4 ++++ tests/f_multithread_no/expect.1 | 24 ++++++++++++++++++++ tests/f_multithread_no/expect.2 | 23 +++++++++++++++++++ tests/f_multithread_no/image.gz | 1 + tests/f_multithread_no/name | 1 + tests/f_multithread_no/script | 4 ++++ tests/f_multithread_preen/expect.1 | 11 +++++++++ tests/f_multithread_preen/expect.2 | 23 +++++++++++++++++++ tests/f_multithread_preen/image.gz | 1 + tests/f_multithread_preen/name | 1 + tests/f_multithread_preen/script | 4 ++++ tests/f_multithread_yes/expect.1 | 2 ++ tests/f_multithread_yes/expect.2 | 23 +++++++++++++++++++ tests/f_multithread_yes/image.gz | 1 + tests/f_multithread_yes/name | 1 + tests/f_multithread_yes/script | 4 ++++ 27 files changed, 216 insertions(+), 6 deletions(-) create mode 100644 tests/f_multithread/expect.1 create mode 100644 tests/f_multithread/expect.2 create mode 120000 tests/f_multithread/image.gz create mode 100644 tests/f_multithread/name create mode 100644 tests/f_multithread/script create mode 100644 tests/f_multithread_completion/expect.1 create mode 100644 tests/f_multithread_completion/expect.2 create mode 120000 tests/f_multithread_completion/image.gz create mode 100644 tests/f_multithread_completion/name create mode 100644 tests/f_multithread_completion/script create mode 100644 tests/f_multithread_no/expect.1 create mode 100644 tests/f_multithread_no/expect.2 create mode 120000 tests/f_multithread_no/image.gz create mode 100644 tests/f_multithread_no/name create mode 100644 tests/f_multithread_no/script create mode 100644 tests/f_multithread_preen/expect.1 create mode 100644 tests/f_multithread_preen/expect.2 create mode 120000 tests/f_multithread_preen/image.gz create mode 100644 tests/f_multithread_preen/name create mode 100644 tests/f_multithread_preen/script create mode 100644 tests/f_multithread_yes/expect.1 create mode 100644 tests/f_multithread_yes/expect.2 create mode 120000 tests/f_multithread_yes/image.gz create mode 100644 tests/f_multithread_yes/name create mode 100644 tests/f_multithread_yes/script diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 24a9d0a8..b5b7d88f 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -177,6 +177,7 @@ struct resource_track { #define E2F_OPT_ICOUNT_FULLMAP 0x20000 /* use an array for inode counts */ #define E2F_OPT_UNSHARE_BLOCKS 0x40000 #define E2F_OPT_CLEAR_UNINIT 0x80000 /* Hack to clear the uninit bit */ +#define E2F_OPT_MULTITHREAD 0x100000 /* Use multiple threads to speedup */ /* * E2fsck flags diff --git a/e2fsck/unix.c b/e2fsck/unix.c index b3ef0f22..c463d29e 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -75,13 +75,14 @@ int journal_enable_debug = -1; static void usage(e2fsck_t ctx) { fprintf(stderr, - _("Usage: %s [-panyrcdfktvDFV] [-b superblock] [-B blocksize]\n" + _("Usage: %s [-pamnyrcdfktvDFV] [-b superblock] [-B blocksize]\n" "\t\t[-l|-L bad_blocks_file] [-C fd] [-j external_journal]\n" "\t\t[-E extended-options] [-z undo_file] device\n"), ctx->program_name); fprintf(stderr, "%s", _("\nEmergency help:\n" " -p Automatic repair (no questions)\n" + " -m multiple threads to speedup fsck\n" " -n Make no changes to the filesystem\n" " -y Assume \"yes\" to all questions\n" " -c Check for bad blocks and add them to the badblock list\n" @@ -847,7 +848,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) phys_mem_kb = get_memory_size() / 1024; ctx->readahead_kb = ~0ULL; - while ((c = getopt(argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) + while ((c = getopt(argc, argv, "pamnyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) switch (c) { case 'C': ctx->progress = e2fsck_update_progress; @@ -888,6 +889,9 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) } ctx->options |= E2F_OPT_PREEN; break; + case 'm': + ctx->options |= E2F_OPT_MULTITHREAD; + break; case 'n': if (ctx->options & (E2F_OPT_YES|E2F_OPT_PREEN)) goto conflict_opt; @@ -1006,6 +1010,18 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) _("The -n and -l/-L options are incompatible.")); fatal_error(ctx, 0); } + if (ctx->options & E2F_OPT_MULTITHREAD) { + if ((ctx->options & (E2F_OPT_YES|E2F_OPT_NO|E2F_OPT_PREEN)) == 0) { + com_err(ctx->program_name, 0, "%s", + _("The -m option should be used together with one of -p/-y/-n options.")); + fatal_error(ctx, 0); + } + if (ctx->progress) { + com_err(ctx->program_name, 0, "%s", + _("Only one of the options -C or -m may be specified.")); + fatal_error(ctx, 0); + } + } if (ctx->options & E2F_OPT_NO) ctx->options |= E2F_OPT_READONLY; @@ -1112,10 +1128,12 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) #ifdef SA_RESTART sa.sa_flags = SA_RESTART; #endif - sa.sa_handler = signal_progress_on; - sigaction(SIGUSR1, &sa, 0); - sa.sa_handler = signal_progress_off; - sigaction(SIGUSR2, &sa, 0); + if ((ctx->options & E2F_OPT_MULTITHREAD) == 0) { + sa.sa_handler = signal_progress_on; + sigaction(SIGUSR1, &sa, 0); + sa.sa_handler = signal_progress_off; + sigaction(SIGUSR2, &sa, 0); + } #endif /* Update our PATH to include /sbin if we need to run badblocks */ diff --git a/tests/f_multithread/expect.1 b/tests/f_multithread/expect.1 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread/expect.1 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread/expect.2 b/tests/f_multithread/expect.2 new file mode 100644 index 00000000..a833aefc --- /dev/null +++ b/tests/f_multithread/expect.2 @@ -0,0 +1,7 @@ +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 0 diff --git a/tests/f_multithread/image.gz b/tests/f_multithread/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread/name b/tests/f_multithread/name new file mode 100644 index 00000000..df838ea6 --- /dev/null +++ b/tests/f_multithread/name @@ -0,0 +1 @@ +test "e2fsck -m" option \ No newline at end of file diff --git a/tests/f_multithread/script b/tests/f_multithread/script new file mode 100644 index 00000000..0fe96cd0 --- /dev/null +++ b/tests/f_multithread/script @@ -0,0 +1,4 @@ +FSCK_OPT="-fy -m" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_completion/expect.1 b/tests/f_multithread_completion/expect.1 new file mode 100644 index 00000000..61cac9bb --- /dev/null +++ b/tests/f_multithread_completion/expect.1 @@ -0,0 +1,2 @@ +../e2fsck/e2fsck: Only one of the options -C or -m may be specified. +Exit status is 8 diff --git a/tests/f_multithread_completion/expect.2 b/tests/f_multithread_completion/expect.2 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_completion/expect.2 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_completion/image.gz b/tests/f_multithread_completion/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_completion/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_completion/name b/tests/f_multithread_completion/name new file mode 100644 index 00000000..a959045d --- /dev/null +++ b/tests/f_multithread_completion/name @@ -0,0 +1 @@ +test "e2fsck -m" option conflicts with "-C" \ No newline at end of file diff --git a/tests/f_multithread_completion/script b/tests/f_multithread_completion/script new file mode 100644 index 00000000..bf23cd61 --- /dev/null +++ b/tests/f_multithread_completion/script @@ -0,0 +1,4 @@ +FSCK_OPT="-fy -m -C 1" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_no/expect.1 b/tests/f_multithread_no/expect.1 new file mode 100644 index 00000000..d14c4083 --- /dev/null +++ b/tests/f_multithread_no/expect.1 @@ -0,0 +1,24 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? no + +Free blocks count wrong (11602, counted=11597). +Fix? no + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? no + +Free inodes count wrong (2997, counted=2992). +Fix? no + + +test_filesys: ********** WARNING: Filesystem still has errors ********** + +test_filesys: 11/3008 files (0.0% non-contiguous), 398/12000 blocks +Exit status is 4 diff --git a/tests/f_multithread_no/expect.2 b/tests/f_multithread_no/expect.2 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_no/expect.2 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_no/image.gz b/tests/f_multithread_no/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_no/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_no/name b/tests/f_multithread_no/name new file mode 100644 index 00000000..fa49692e --- /dev/null +++ b/tests/f_multithread_no/name @@ -0,0 +1 @@ +test "e2fsck -m" option works with "-n" \ No newline at end of file diff --git a/tests/f_multithread_no/script b/tests/f_multithread_no/script new file mode 100644 index 00000000..b93deb3a --- /dev/null +++ b/tests/f_multithread_no/script @@ -0,0 +1,4 @@ +FSCK_OPT="-fn -m" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_preen/expect.1 b/tests/f_multithread_preen/expect.1 new file mode 100644 index 00000000..b4b0cd9a --- /dev/null +++ b/tests/f_multithread_preen/expect.1 @@ -0,0 +1,11 @@ +../e2fsck/e2fsck: Bad magic number in super-block while trying to open test.img +test_filesys: +The superblock could not be read or does not describe a valid ext2/ext3/ext4 +filesystem. If the device is valid and it really contains an ext2/ext3/ext4 +filesystem (and not swap or ufs or something else), then the superblock +is corrupt, and you might try running e2fsck with an alternate superblock: + e2fsck -b 8193 + or + e2fsck -b 32768 + +Exit status is 8 diff --git a/tests/f_multithread_preen/expect.2 b/tests/f_multithread_preen/expect.2 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_preen/expect.2 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_preen/image.gz b/tests/f_multithread_preen/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_preen/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_preen/name b/tests/f_multithread_preen/name new file mode 100644 index 00000000..90d199df --- /dev/null +++ b/tests/f_multithread_preen/name @@ -0,0 +1 @@ +test "e2fsck -m" option works with "-p" \ No newline at end of file diff --git a/tests/f_multithread_preen/script b/tests/f_multithread_preen/script new file mode 100644 index 00000000..ecb79cd6 --- /dev/null +++ b/tests/f_multithread_preen/script @@ -0,0 +1,4 @@ +FSCK_OPT="-fp -m" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_yes/expect.1 b/tests/f_multithread_yes/expect.1 new file mode 100644 index 00000000..8b780ecf --- /dev/null +++ b/tests/f_multithread_yes/expect.1 @@ -0,0 +1,2 @@ +../e2fsck/e2fsck: The -m option should be used together with one of -p/-y/-n options. +Exit status is 8 diff --git a/tests/f_multithread_yes/expect.2 b/tests/f_multithread_yes/expect.2 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_yes/expect.2 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_yes/image.gz b/tests/f_multithread_yes/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_yes/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_yes/name b/tests/f_multithread_yes/name new file mode 100644 index 00000000..3a703195 --- /dev/null +++ b/tests/f_multithread_yes/name @@ -0,0 +1 @@ +test "e2fsck -m" option works with "-y" \ No newline at end of file diff --git a/tests/f_multithread_yes/script b/tests/f_multithread_yes/script new file mode 100644 index 00000000..38891f6a --- /dev/null +++ b/tests/f_multithread_yes/script @@ -0,0 +1,4 @@ +FSCK_OPT="-f -m" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck From patchwork Wed Apr 8 10:44:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267862 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=f4fEqESC; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1C95Vxdz9sQx for ; Wed, 8 Apr 2020 20:45:41 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728206AbgDHKpl (ORCPT ); Wed, 8 Apr 2020 06:45:41 -0400 Received: from mail-pj1-f68.google.com ([209.85.216.68]:53124 "EHLO mail-pj1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728173AbgDHKpl (ORCPT ); Wed, 8 Apr 2020 06:45:41 -0400 Received: by mail-pj1-f68.google.com with SMTP id ng8so1002669pjb.2 for ; Wed, 08 Apr 2020 03:45:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=31GUYr+hjKDs7YNLdUYhGFexT5oZx4Og0VokEOkoOJA=; b=f4fEqESC3OdE+P+AZsObq8SoWBmsgLAoHzTJ3Wb26Rv9Y7w2zfT85P4MygiWxMjqy4 G3JEBnTo3m2BEkppn8dIvGkbUfT+nq2etI6ZiJDFPjtO9YRvqjbmRpaZ9NbRaqP9yUTd JkhYAHR72J79C/c+EM8vKdDCdfQ6+/1HXCFzQ/939RqUHnPSFeyyD6Cun1RsA2w5w4Nt T9ySRC/g9SFTt7u731lAcCA8JI3LufYfKHmaQoy8HAdEpia9bDXPRxWbwXfq9kJj7/+0 bfAeD7RgN7Q4p0Yq5DhDw2waK7FMEbRtJg/xB1aT3wM7hruTqys2dsqNdEYnNw+ny73S v3ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=31GUYr+hjKDs7YNLdUYhGFexT5oZx4Og0VokEOkoOJA=; b=Jq7kDKpmd7lBSxpv/PdNYA1oSYiyn6vfNvOhFHL0UFG6GBbVD4rSBRodBCCl8gZPxK ElTUTofQs+E5GCykTjM//0HWoO1jOxkIPbvG1Y9LibA3Ccb6CEunGBNaNnkzZN3fj4PD OKb6mXdZQH7BMaAH/8fpjRpn1JRXNZYKxfrZV3sn0GkpCmUCTIChC3ugKVN786Gopwsi ej02dwE43RcCiGpNeg5ljrkFsIkYmL4Y9ACQqQMOoGrN+///FMvFDTUdc2V+KIHV8rV9 sBcyogP4u6l8h0CxZUMltdQ9bYPUiJ9zwUdv0Y3uP91Ni8T+ZlE70nGEroOBGSGXycnr cnKw== X-Gm-Message-State: AGi0PuYFVon8l0TfSaEAOMKsBG5ZWn8A6z1/ebM0qSUxMrwpxrzqLl7f dWNk5qrluUJg/SH6sF6vd8F2zVWB4ao= X-Google-Smtp-Source: APiQypKQKhk9wP83HFTbSxjeG30NXU8/o3en8HmPOCaPRhVgxfJKWLhZa1Y4dPhgBn/gRezHEDezYw== X-Received: by 2002:a17:90a:368d:: with SMTP id t13mr1956425pjb.175.1586342739468; Wed, 08 Apr 2020 03:45:39 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:38 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 03/46] e2fsck: copy context when using multi-thread fsck Date: Wed, 8 Apr 2020 19:44:31 +0900 Message-Id: <1586342714-12536-4-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch only copy the context to a new one when -m is enabled. It doesn't actually start any thread. When pass1 test finishes, the new context is copied back to the original context. Since the signal handler only changes the original context, so add global_ctx in "struct e2fsck_struct" and use that to check whether there is any signal of canceling. This patch handles the long jump properly so that all the existing tests can be passed even the context has been copied. Otherwise, test f_expisize_ea_del would fail when aborting. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 2 + e2fsck/pass1.c | 116 ++++++++++++++++++++++++++++++++++++++++++++---- e2fsck/unix.c | 1 + 3 files changed, 110 insertions(+), 9 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index b5b7d88f..c63b1852 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -229,6 +229,8 @@ typedef struct e2fsck_struct *e2fsck_t; struct e2fsck_struct { /* ---- Following fields are never updated during the pass1 ---- */ + /* Global context to get the cancel flag */ + e2fsck_t global_ctx; const char *program_name; char *filesystem_name; char *device_name; diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index a57c1c06..b093a734 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1154,7 +1154,22 @@ static int quota_inum_is_reserved(ext2_filsys fs, ext2_ino_t ino) return 0; } -void e2fsck_pass1(e2fsck_t ctx) +static int e2fsck_should_abort(e2fsck_t ctx) +{ + e2fsck_t global_ctx; + + if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + return 1; + + if (ctx->global_ctx) { + global_ctx = ctx->global_ctx; + if (global_ctx->flags & E2F_FLAG_SIGNAL_MASK) + return 1; + } + return 0; +} + +void e2fsck_pass1_thread(e2fsck_t ctx) { int i; __u64 max_sizes; @@ -1370,7 +1385,7 @@ void e2fsck_pass1(e2fsck_t ctx) if (ino > ino_threshold) pass1_readahead(ctx, &ra_group, &ino_threshold); ehandler_operation(old_op); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto endit; if (pctx.errcode == EXT2_ET_BAD_BLOCK_IN_INODE_TABLE) { /* @@ -1967,13 +1982,13 @@ void e2fsck_pass1(e2fsck_t ctx) FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto endit; if (process_inode_count >= ctx->process_inode_size) { process_inodes(ctx, block_buf); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto endit; } } @@ -2086,6 +2101,89 @@ endit: else ctx->invalid_bitmaps++; } + +static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) +{ + errcode_t retval; + e2fsck_t thread_context; + + retval = ext2fs_get_mem(sizeof(struct e2fsck_struct), &thread_context); + if (retval) { + com_err(global_ctx->program_name, retval, "while allocating memory"); + return retval; + } + memcpy(thread_context, global_ctx, sizeof(struct e2fsck_struct)); + thread_context->fs->priv_data = thread_context; + thread_context->global_ctx = global_ctx; + + *thread_ctx = thread_context; + return 0; +} + +static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + int flags = global_ctx->flags; +#ifdef HAVE_SETJMP_H + jmp_buf old_jmp; + + memcpy(old_jmp, global_ctx->abort_loc, sizeof(jmp_buf)); +#endif + memcpy(global_ctx, thread_ctx, sizeof(struct e2fsck_struct)); +#ifdef HAVE_SETJMP_H + memcpy(global_ctx->abort_loc, old_jmp, sizeof(jmp_buf)); +#endif + /* Keep the global singal flags*/ + global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | + (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); + + global_ctx->fs->priv_data = global_ctx; + ext2fs_free_mem(&thread_ctx); + return 0; +} + +void e2fsck_pass1_multithread(e2fsck_t ctx) +{ + errcode_t retval; + e2fsck_t thread_ctx; + + retval = e2fsck_pass1_thread_prepare(ctx, &thread_ctx); + if (retval) { + com_err(ctx->program_name, 0, + _("while preparing pass1 thread\n")); + ctx->flags |= E2F_FLAG_ABORT; + return; + } + +#ifdef HAVE_SETJMP_H + /* + * When fatal_error() happens, jump to here. The thread + * context's flags will be saved, but its abort_loc will + * be overwritten by original jump buffer for the later + * tests. + */ + if (setjmp(thread_ctx->abort_loc)) { + thread_ctx->flags &= ~E2F_FLAG_SETJMP_OK; + e2fsck_pass1_thread_join(ctx, thread_ctx); + return; + } + thread_ctx->flags |= E2F_FLAG_SETJMP_OK; +#endif + + e2fsck_pass1_thread(thread_ctx); + retval = e2fsck_pass1_thread_join(ctx, thread_ctx); + if (retval) { + com_err(ctx->program_name, 0, + _("while joining pass1 thread\n")); + ctx->flags |= E2F_FLAG_ABORT; + return; + } +} + +void e2fsck_pass1(e2fsck_t ctx) +{ + e2fsck_pass1_multithread(ctx); +} + #undef FINISH_INODE_LOOP /* @@ -2148,7 +2246,7 @@ static void process_inodes(e2fsck_t ctx, char *block_buf) ehandler_operation(buf); check_blocks(ctx, &pctx, block_buf, &inodes_to_process[i].ea_ibody_quota); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) break; } ctx->stashed_inode = old_stashed_inode; @@ -3318,7 +3416,7 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, inlinedata_fs = ext2fs_has_feature_inline_data(ctx->fs->super); if (check_ext_attr(ctx, pctx, block_buf, &ea_block_quota)) { - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto out; pb.num_blocks += EXT2FS_B2C(ctx->fs, ea_block_quota.blocks); } @@ -3373,7 +3471,7 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, } end_problem_latch(ctx, PR_LATCH_BLOCK); end_problem_latch(ctx, PR_LATCH_TOOBIG); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto out; if (pctx->errcode) fix_problem(ctx, PR_1_BLOCK_ITERATE, pctx); @@ -3854,7 +3952,7 @@ static int process_bad_block(ext2_filsys fs, *block_nr = 0; return BLOCK_CHANGED; } - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) return BLOCK_ABORT; } else mark_block_used(ctx, blk); @@ -3951,7 +4049,7 @@ static int process_bad_block(ext2_filsys fs, *block_nr = 0; return BLOCK_CHANGED; } - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) return BLOCK_ABORT; return 0; } diff --git a/e2fsck/unix.c b/e2fsck/unix.c index c463d29e..8226aff7 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -1444,6 +1444,7 @@ int main (int argc, char *argv[]) } reserve_stdio_fds(); + ctx->global_ctx = NULL; set_up_logging(ctx); if (ctx->logf) { int i; From patchwork Wed Apr 8 10:44:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267864 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=jPoO3xZE; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CD49kMz9sSX for ; Wed, 8 Apr 2020 20:45:44 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728209AbgDHKpo (ORCPT ); Wed, 8 Apr 2020 06:45:44 -0400 Received: from mail-pj1-f66.google.com ([209.85.216.66]:37134 "EHLO mail-pj1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKpo (ORCPT ); Wed, 8 Apr 2020 06:45:44 -0400 Received: by mail-pj1-f66.google.com with SMTP id k3so1001550pjj.2 for ; Wed, 08 Apr 2020 03:45:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EtWMOQzTpeZ5k9wo+uBZU1UwKXRnzhThMrRrbpVkECI=; b=jPoO3xZELJh6NLXnMw3t+islapL4sHoZ19+s/l5SsA1jym2jiIixtvRRWy0gJ5+h9c lgIleM9po6FTLNoyKkYbmT0Zla8RH0jISfeR/5knkV1TRMNVfgzbR5gRJdDaLeM/BEbZ 4MsJIl3sHyDCkMvnQ4n2fo8jjx3v7qoxnJvNNbNIeb/1g6fAISieshVJwsJcCvj3ytN9 9RgupAR3NJTGL28vMqmnwbrMmj2hwSqH9iCU+9ht/zbQQRzKNcevNPjYPV6dtV/xMNe8 fexd90AcZXVTHQWODLyg/rlTdEUvBMUdZ3EYvMXyQeKjQUq/XRIFfE8hSahbUddIbX5z lTag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EtWMOQzTpeZ5k9wo+uBZU1UwKXRnzhThMrRrbpVkECI=; b=eZ0k8t6pFuA4MCJX9FrPwQXdX5k0WTdyt5XDCEUwpWGBZAY3hke2NwhXo30gzhmASk +MM8yS7BGrlXqxji8HBXKyl46ygsKrhhK7yQltFsTRhqX4Ra7CAeW8cOu6UA1pFCwNtO uwYcaZibEPuv5a43Gl1V9MH3usG5D0qRVHvyaqdCCzXsk6mkM4FTBJMD2EkGDW+8baWb ajEwBALjHqwbwVMnwDGIrkJCGmVcrtIXGMb3Wl+7QxWoFqUPUrR+XzvRTkIq013mfP1U NsTciFsSski2hL2/y639XFTIzhE61WRHcxlONvapJI99m/qykhjNPR0w+SWIheuDJFjg +I8A== X-Gm-Message-State: AGi0PubrLJpg5CQoH4koPJpy2QiUv56IRdVZ0SW8qF4I0+E0Nuqu2jih YxXdHWm3BHbMVCOCAe3sTBxzmj9n4rM= X-Google-Smtp-Source: APiQypJF9u+EXpXuVgPsjZfcNmSXneQ/50HP5WZ3Quy+46gN2+P7t1vZwXOy43Oel8R/77YbgF9Csw== X-Received: by 2002:a17:90a:da03:: with SMTP id e3mr4785714pjv.61.1586342741595; Wed, 08 Apr 2020 03:45:41 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:41 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 04/46] e2fsck: copy fs when using multi-thread fsck Date: Wed, 8 Apr 2020 19:44:32 +0900 Message-Id: <1586342714-12536-5-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch only copy the fs to a new one when -m is enabled. It doesn't actually start any thread. When pass1 test finishes, the new fs is copied back to the original context. This patch handles the fs fields in dblist, inode_map and block_map properly. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 41 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index b093a734..2fcc466e 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -49,6 +49,8 @@ #include "e2fsck.h" #include +/* todo remove this finally */ +#include #include #include "problem.h" @@ -2102,10 +2104,23 @@ endit: ctx->invalid_bitmaps++; } +static void e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) +{ + memcpy(dest, src, sizeof(struct struct_ext2_filsys)); + if (dest->dblist) + dest->dblist->fs = dest; + if (dest->inode_map) + dest->inode_map->fs = dest; + if (dest->block_map) + dest->block_map->fs = dest; +} + static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) { errcode_t retval; e2fsck_t thread_context; + ext2_filsys thread_fs; + ext2_filsys global_fs = global_ctx->fs; retval = ext2fs_get_mem(sizeof(struct e2fsck_struct), &thread_context); if (retval) { @@ -2113,18 +2128,32 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre return retval; } memcpy(thread_context, global_ctx, sizeof(struct e2fsck_struct)); - thread_context->fs->priv_data = thread_context; thread_context->global_ctx = global_ctx; + retval = ext2fs_get_mem(sizeof(struct struct_ext2_filsys), &thread_fs); + if (retval) { + com_err(global_ctx->program_name, retval, "while allocating memory"); + goto out_context; + } + + e2fsck_pass1_copy_fs(thread_fs, global_fs); + thread_fs->priv_data = thread_context; + + thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; +out_context: + ext2fs_free_mem(&thread_context); + return retval; } static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) { - int flags = global_ctx->flags; + int flags = global_ctx->flags; + ext2_filsys thread_fs = thread_ctx->fs; + ext2_filsys global_fs = global_ctx->fs; #ifdef HAVE_SETJMP_H - jmp_buf old_jmp; + jmp_buf old_jmp; memcpy(old_jmp, global_ctx->abort_loc, sizeof(jmp_buf)); #endif @@ -2136,7 +2165,11 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); - global_ctx->fs->priv_data = global_ctx; + e2fsck_pass1_copy_fs(global_fs, thread_fs); + global_fs->priv_data = global_ctx; + global_ctx->fs = global_fs; + + ext2fs_free_mem(&thread_ctx->fs); ext2fs_free_mem(&thread_ctx); return 0; } From patchwork Wed Apr 8 10:44:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267865 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=rBbeZo3P; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CG4fb1z9sSX for ; Wed, 8 Apr 2020 20:45:46 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728212AbgDHKpq (ORCPT ); Wed, 8 Apr 2020 06:45:46 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:32961 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKpq (ORCPT ); Wed, 8 Apr 2020 06:45:46 -0400 Received: by mail-pl1-f194.google.com with SMTP id ay1so2391653plb.0 for ; Wed, 08 Apr 2020 03:45:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IaN541sTTl4RtV1K3SmEOOqaEpmyQ4q5okG6pQ5zcgs=; b=rBbeZo3PBvCNRxsJKq6je2xM774nCJoPnIi5z9o9LY0gmZA6ny2IksG9bHtECMX0sL eUsdSowETR67ZuaGOV7RTF8E5HzzdVqtrkJb/vgb0dnZE1C/LcAPhdki2A8gb9gA8oof WHBzNa+V4QcKWwrQYDGhvXhFJY8bHpYCj4k88zlNNlysdjoWv3ljYGaPwkXUfHss++1t HG4voZuZZxmXtACw4gfNho6vPT39dBEtIqdbavYmjkQG/9QqYxIi8uRxnkyJ+9aJkul/ rk/FZ92LiHZgHUg5sZlNfT9NGzts2MZcun7Psz+V+SUKfSj2jyg94JjUphKpWJgWtTY3 pGKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IaN541sTTl4RtV1K3SmEOOqaEpmyQ4q5okG6pQ5zcgs=; b=DwbroFg4rF50u4JHgtbautFMYENBjMG0mVMNcphe4Ndl2Xj06CDuG0ZFvhJEFafwr9 jtjgDRnbOB79wtD+qF3LQGKfWk7icF1EmLmSoX3Ap3MXoIvUNbvpOOpedanJqYzktDI9 0q49GTGouQZd3HKgr0n5TLgYG8o/NCuRIw8COK2m/vAq3OIRFVpPE4Tm28ecRY7eBUOX 3l45aaAX29zPygt0D9YAry0L+z5B/02hg2LEmaLNiQazj31OJs4Lkr8gGgEc6qO+dp+U KaLJCOehC+LwiNwijUaHXdsB45NxBRi0CxATlEb2+uQYypLg+rBeeJDqHVRANag/A6u3 oWqQ== X-Gm-Message-State: AGi0PuaLqrW0ulDw+qLrwPs96XRGYCy5v8NZqauV6/643JGNNp69hayj 8NEvzLTW2C8j1UipJToUXcTPy31Mny8= X-Google-Smtp-Source: APiQypJUFAlYasJDYwcDlgmOJD60eg2eo94O5MXF1PwOAPKl6vy5ajg0K+E6UEMY2JR+BFfDK7bFbA== X-Received: by 2002:a17:902:a605:: with SMTP id u5mr6526439plq.175.1586342743719; Wed, 08 Apr 2020 03:45:43 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:43 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 05/46] e2fsck: copy dblist when using multi-thread fsck Date: Wed, 8 Apr 2020 19:44:33 +0900 Message-Id: <1586342714-12536-6-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch only copy the dblist of the fs to a new one when -m is specified. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 2fcc466e..baf720ce 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2104,8 +2104,29 @@ endit: ctx->invalid_bitmaps++; } -static void e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) +static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) { + errcode_t retval; + + memcpy(dest, src, sizeof(struct struct_ext2_filsys)); + if (dest->dblist) { + retval = ext2fs_copy_dblist(src->dblist, &dest->dblist); + if (retval) + return retval; + /* The ext2fs_copy_dblist() uses the src->fs as the fs */ + dest->dblist->fs = dest; + } + if (dest->inode_map) + dest->inode_map->fs = dest; + if (dest->block_map) + dest->block_map->fs = dest; + return 0; +} + +static void e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) +{ + if (dest->dblist) + ext2fs_free_dblist(dest->dblist); memcpy(dest, src, sizeof(struct struct_ext2_filsys)); if (dest->dblist) dest->dblist->fs = dest; @@ -2136,12 +2157,18 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre goto out_context; } - e2fsck_pass1_copy_fs(thread_fs, global_fs); + retval = e2fsck_pass1_copy_fs(thread_fs, global_fs); + if (retval) { + com_err(global_ctx->program_name, retval, "while copying fs"); + goto out_fs; + } thread_fs->priv_data = thread_context; thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; +out_fs: + ext2fs_free_mem(&thread_fs); out_context: ext2fs_free_mem(&thread_context); return retval; @@ -2165,7 +2192,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); - e2fsck_pass1_copy_fs(global_fs, thread_fs); + e2fsck_pass1_merge_fs(global_fs, thread_fs); global_fs->priv_data = global_ctx; global_ctx->fs = global_fs; From patchwork Wed Apr 8 10:44:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267866 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ODGvkFuw; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CH5sGYz9sSX for ; Wed, 8 Apr 2020 20:45:47 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728214AbgDHKpr (ORCPT ); Wed, 8 Apr 2020 06:45:47 -0400 Received: from mail-pj1-f67.google.com ([209.85.216.67]:40202 "EHLO mail-pj1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKpr (ORCPT ); Wed, 8 Apr 2020 06:45:47 -0400 Received: by mail-pj1-f67.google.com with SMTP id kx8so996833pjb.5 for ; Wed, 08 Apr 2020 03:45:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=POJHMabIZ8KwFcNcEOZj0UEMAUO9xdB4xE8m54J3PJI=; b=ODGvkFuw6cIEZexXpiNE6uZEiemBwLLHgyQGiYQgJh4g46KGo5OVMxMvhxFw+BcXku kEC2TI8TKirPooP98RBMzvQm3xHkguffeYTiTiEIcoR8cVUBWJcd5iGWmGdi4CGQ9Iq5 kdnxDe71b25uTtdmK956Yc8Vh+EMjUAXsasCsM16BgAiC1qhddj27qSI/CwvAv8gzufH 6i5JmCsevJ+HhXnRUSsgEetdJvRy81mlZGZwyagEJzS1ayP6K3FSXTPaFZtcXj3J2XC+ FqDMxK7WPfF1dXobnqJFZRDMQ/iUpkJDa3wlsm7hyjM2R6iVsppnRwg07vZ9ApqUb0D5 S6Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=POJHMabIZ8KwFcNcEOZj0UEMAUO9xdB4xE8m54J3PJI=; b=n6xNn8my6pJ8RSrFw8ixZ9Z35025Zbz5lT+q7QUR4F5L/MUqogCVYsenNHb4Ks4EuC WNNjD3W9FhV13tdImXdkcfUKfqK8rolKBBbrcU4vVMoVsoYga+L4kS50bhNB3su8yl6e 4yYPPLuZXSqWg3iO5ZaomJ9/nQng+iysBS23SaMQR1cQhBl41xXORmc4dK6wHCsHpTlt rWi6ocGG8oqif+w0i/faRWWlilqC4LoL3JHvv4LMNhbYPf7y/UsPGtYi7eaPugYYJcYD cma93tb5SjfEn61GN+cvXCYXexwRvDygVNZDWkao6G8fP3bIYMP4YenNeJy65Heur8Eb g5PA== X-Gm-Message-State: AGi0PuZq0q0liXJ71xk23+laaguOnCDBhsxHpfNGr8KZG56iYb6FKJaU X/F+scrsc3e7eBYwHXvBb36eb1iFrbM= X-Google-Smtp-Source: APiQypKL9eQHc9iHG9UFkQaZALDlLcjbKVgncxylsICEs142NMxfPAodQtUBtQNQ9YCfC1FSRQRbFg== X-Received: by 2002:a17:902:b941:: with SMTP id h1mr6472785pls.329.1586342745820; Wed, 08 Apr 2020 03:45:45 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:45 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 06/46] e2fsck: clear icache when using multi-thread fsck Date: Wed, 8 Apr 2020 19:44:34 +0900 Message-Id: <1586342714-12536-7-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi icache of fs will be rebuilt when needed, so after copying fs, icache can be inited to NULL. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index baf720ce..599c69aa 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2120,6 +2120,13 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) dest->inode_map->fs = dest; if (dest->block_map) dest->block_map->fs = dest; + + /* icache will be rebuilt if needed, so do not copy from @src */ + if (src->icache) { + ext2fs_free_inode_cache(src->icache); + src->icache = NULL; + } + dest->icache = NULL; return 0; } @@ -2134,6 +2141,13 @@ static void e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) dest->inode_map->fs = dest; if (dest->block_map) dest->block_map->fs = dest; + + /* icache will be rebuilt if needed, so do not copy from @src */ + if (src->icache) { + ext2fs_free_inode_cache(src->icache); + src->icache = NULL; + } + dest->icache = NULL; } static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) From patchwork Wed Apr 8 10:44:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267867 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=mEKVhFw9; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CK48Vfz9sSG for ; Wed, 8 Apr 2020 20:45:49 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728219AbgDHKpt (ORCPT ); Wed, 8 Apr 2020 06:45:49 -0400 Received: from mail-pj1-f68.google.com ([209.85.216.68]:53128 "EHLO mail-pj1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKpt (ORCPT ); Wed, 8 Apr 2020 06:45:49 -0400 Received: by mail-pj1-f68.google.com with SMTP id ng8so1002749pjb.2 for ; Wed, 08 Apr 2020 03:45:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xn/9gW4cVNsKY+C/0Cg/50ED51EAz2uYRVZSESOP2rA=; b=mEKVhFw9l0TWDeIOwnyKe3Kvx59CkvLwgSp+QZkkoiLICNpi6XxU7BOwEPH8OE3UxO 1BBSZgQQgYo2SBL3StGT6lsjR/O9CXs7OsKe9QyLs4ZJeydb9Jq1Tr2uycE0oOFEIeHb Jnkr82nCM4zyGxCtmZ1wZXclgX0u7dY374gTTCY0aWqRUc3KvPs291msMpHkj2pJ9zu9 kUafp0LIwrO18Bu9gQLROeXRM86jyiDiXbST7mzP3rA15m+3bT87Z5JrIrlGxHaa0hbj eVz0FYmTeUwfB0sJof+ncRtTVe3hO8POISwwB9AvT6mbBtkaBJLLMlS0BZ5Yy2QG6+D2 jEqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xn/9gW4cVNsKY+C/0Cg/50ED51EAz2uYRVZSESOP2rA=; b=fM9XFaNhXuD0UDzRtDhkeOEC2k0MBrElDM2VQxF/MSKy+wTid1+y4JbYuiWPmUTqDr 3hgSVdAd/ELTIkZMBGbrWQoGonb46nWoB5YjR5VpY/lRsOrkEyePO6dsLQnvxCOITS+H EynIDNvAKZUnG3AFxrVhAA8wiwYEDEN5P1exAZ2GaJeFxwFsYqP9/6aZ/jSa9PcgrEUT oh0aCCJfH086sP86ETFQ+ECNcKb12DsL2cYpEfYTFTPVp2R+7Cqlm8kEYVEVXSoNl/YC 6cFPmBSjsv5EcwpTl3m+hYEJD/6G7bEhYj97X2UxdCbvL+UrwC3lcqixqsQ/pIQfYuyn VQ/w== X-Gm-Message-State: AGi0PuY/fEIA14LIKXBupBnMkmQYJG9Ij0QMStT0Sju9RzvBKvQOtj8w N+xJOi7wkbBxfFiYtXqtrzJ5OQeH9CI= X-Google-Smtp-Source: APiQypK/zJMFeBuICaoojoSnsEq1SR5vfAo3iEEasC3TA5/odkwEh2Kn5+YOesNLwOc6SwQJjUGYDA== X-Received: by 2002:a17:902:be12:: with SMTP id r18mr6836746pls.303.1586342747917; Wed, 08 Apr 2020 03:45:47 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:47 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 07/46] e2fsck: add assert when copying context Date: Wed, 8 Apr 2020 19:44:35 +0900 Message-Id: <1586342714-12536-8-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Adding the assert would simplify the copying of context. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 599c69aa..2fafeb38 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -46,6 +46,7 @@ #ifdef HAVE_ERRNO_H #include #endif +#include #include "e2fsck.h" #include @@ -2157,6 +2158,18 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre ext2_filsys thread_fs; ext2_filsys global_fs = global_ctx->fs; + assert(global_ctx->inode_used_map == NULL); + assert(global_ctx->inode_dir_map == NULL); + assert(global_ctx->inode_bb_map == NULL); + assert(global_ctx->inode_imagic_map == NULL); + assert(global_ctx->inode_reg_map == NULL); + assert(global_ctx->inodes_to_rebuild == NULL); + + assert(global_ctx->block_found_map == NULL); + assert(global_ctx->block_dup_map == NULL); + assert(global_ctx->block_ea_map == NULL); + assert(global_ctx->block_metadata_map == NULL); + retval = ext2fs_get_mem(sizeof(struct e2fsck_struct), &thread_context); if (retval) { com_err(global_ctx->program_name, retval, "while allocating memory"); From patchwork Wed Apr 8 10:44:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267868 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=FvYbiFBc; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CM5WN7z9sSX for ; Wed, 8 Apr 2020 20:45:51 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728221AbgDHKpv (ORCPT ); Wed, 8 Apr 2020 06:45:51 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:39983 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKpv (ORCPT ); Wed, 8 Apr 2020 06:45:51 -0400 Received: by mail-pl1-f194.google.com with SMTP id h11so2367542plk.7 for ; Wed, 08 Apr 2020 03:45:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=irjlyi2cDB70rZmTo+0RxipBN5EBcB4b7jmEtTgHO6Q=; b=FvYbiFBceF83OOiKc4YOcaSdAJwjC6feKtshD4Pvyy19RFKpoaMdesfSIjIhrDEOJ5 KZD6FW7AgSiKSqTmapBeC7j4LgvfbkWUcf982FZ4csnIfshYQjsB8bhmM0QU2pI2zvzJ /5NCMmo2NVI6XiRfki9KTn6kP0yw4jdDSkO/h6N24UTUn8TG32jRek1UexHcpjk4a0AK KKgY3OfjkCJ//pD276VxvqrTKFBVTu40NW4g9V0L1X+Xs/2WNP2BGqn39aDd9ASF36Ul qjWJzVcqIzPfkV4FmI4UjkDRXm0KcukESd3mdpQNs/3XAiWqYs0Um9VIfPrDeiLat73u yQYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=irjlyi2cDB70rZmTo+0RxipBN5EBcB4b7jmEtTgHO6Q=; b=MVhkytLGXqJRzfoAyAOE5vaw1Fx/DSAHZp8lKosCpHib4iseBKD6JMp7hoRs89lX5F Uh/4hMl5DfwEXwCRrYB1Q51RTd/G4q7jWAeaPB4n81Vc7VR60vjb+uG24Fi67WsNifBJ 2JvVC1hNCELKQJnsy3vCLuOKbHW5eiVlAMebb0wjhXxK2EjkFqo7Ka/137eCIWy/1VaB UEiKfe2NuVbpaoOaNfI9xEqf8ZJ/RgJN99QGThl0l85Ny8iQkDsUYwnEteAIu1tMxzqh lxahImW18O4sHU/ufMzjueHBa4gELPCmvcKKiyuNhVQ9IKYk5c+Aj+iMRy6BWbRxRAPV nA8Q== X-Gm-Message-State: AGi0PuYyMp1ipKjvQ8e+mHpO9JoeglLFE1Az8xaBqmfnF6x1xg+RWyhf U5jjBWL4XMYcgWzVCIouClUS87lwBPM= X-Google-Smtp-Source: APiQypJ6D+Qlmnf7zdu5knptewvmblkBNroFxxZHa6ebwbvJ/AehYpo1JLID4eaIvhroYEWMZVVPXw== X-Received: by 2002:a17:902:aa48:: with SMTP id c8mr6411617plr.95.1586342750015; Wed, 08 Apr 2020 03:45:50 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:49 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 08/46] e2fsck: copy bitmaps when copying context Date: Wed, 8 Apr 2020 19:44:36 +0900 Message-Id: <1586342714-12536-9-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch copies bitmap when the copying context. In the multi-thread fsck, each thread use different bitmap that copied from the glboal bitmap. And Bitmaps from multiple threads will be merged into a global one after the pass1 finishes. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 121 +++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 98 insertions(+), 23 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 2fafeb38..f3337bde 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2105,22 +2105,45 @@ endit: ctx->invalid_bitmaps++; } +#define PASS1_COPY_FS_BITMAP(_dest, _src, _map_filed) \ +do { \ + errcode_t _ret; \ + if (_src->_map_filed) { \ + _ret = ext2fs_copy_bitmap(_src->_map_filed, &_dest->_map_filed);\ + if (_ret) \ + return _ret; \ + _dest->_map_filed->fs = _dest; \ + \ + ext2fs_free_generic_bmap(_src->_map_filed); \ + _src->_map_filed = NULL; \ + } \ +} while (0) + +#define PASS1_COPY_CTX_BITMAP(_dest, _src, _map_filed) \ +do { \ + errcode_t _ret; \ + if (_src->_map_filed) { \ + _ret = ext2fs_copy_bitmap(_src->_map_filed, &_dest->_map_filed);\ + if (_ret) \ + return _ret; \ + _dest->_map_filed->fs = _dest->fs; \ + \ + ext2fs_free_generic_bmap(_src->_map_filed); \ + _src->_map_filed = NULL; \ + } \ +} while (0) + static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) { errcode_t retval; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); - if (dest->dblist) { - retval = ext2fs_copy_dblist(src->dblist, &dest->dblist); - if (retval) - return retval; - /* The ext2fs_copy_dblist() uses the src->fs as the fs */ - dest->dblist->fs = dest; - } - if (dest->inode_map) - dest->inode_map->fs = dest; - if (dest->block_map) - dest->block_map->fs = dest; + /* + * PASS1_COPY_FS_BITMAP might return directly from this function, + * so please do NOT leave any garbage behind after returning. + */ + PASS1_COPY_FS_BITMAP(dest, src, inode_map); + PASS1_COPY_FS_BITMAP(dest, src, block_map); /* icache will be rebuilt if needed, so do not copy from @src */ if (src->icache) { @@ -2128,20 +2151,32 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) src->icache = NULL; } dest->icache = NULL; + + if (src->dblist) { + retval = ext2fs_copy_dblist(src->dblist, &dest->dblist); + if (retval) + return retval; + /* The ext2fs_copy_dblist() uses the src->fs as the fs */ + dest->dblist->fs = dest; + + ext2fs_free_dblist(src->dblist); + src->dblist = NULL; + } + return 0; } -static void e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) +static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) { - if (dest->dblist) - ext2fs_free_dblist(dest->dblist); + errcode_t retval = 0; + memcpy(dest, src, sizeof(struct struct_ext2_filsys)); - if (dest->dblist) - dest->dblist->fs = dest; - if (dest->inode_map) - dest->inode_map->fs = dest; - if (dest->block_map) - dest->block_map->fs = dest; + /* + * PASS1_COPY_FS_BITMAP might return directly from this function, + * so please do NOT leave any garbage behind after returning. + */ + PASS1_COPY_FS_BITMAP(dest, src, inode_map); + PASS1_COPY_FS_BITMAP(dest, src, block_map); /* icache will be rebuilt if needed, so do not copy from @src */ if (src->icache) { @@ -2149,6 +2184,18 @@ static void e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) src->icache = NULL; } dest->icache = NULL; + + if (dest->dblist) { + retval = ext2fs_copy_dblist(src->dblist, &dest->dblist); + if (retval == 0) { + /* The ext2fs_copy_dblist() uses the src->fs as the fs */ + dest->dblist->fs = dest; + } + + ext2fs_free_dblist(src->dblist); + src->dblist = NULL; + } + return retval; } static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) @@ -2201,8 +2248,9 @@ out_context: return retval; } -static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) +static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { + errcode_t retval; int flags = global_ctx->flags; ext2_filsys thread_fs = thread_ctx->fs; ext2_filsys global_fs = global_ctx->fs; @@ -2219,13 +2267,40 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); - e2fsck_pass1_merge_fs(global_fs, thread_fs); + retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); + if (retval) { + com_err(global_ctx->program_name, 0, _("while merging fs\n")); + return retval; + } global_fs->priv_data = global_ctx; global_ctx->fs = global_fs; + /* + * PASS1_COPY_CTX_BITMAP might return directly from this function, + * so please do NOT leave any garbage behind after returning. + */ + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_used_map); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_dir_map); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_bb_map); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_imagic_map); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_reg_map); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inodes_to_rebuild); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_found_map); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_dup_map); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_ea_map); + PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_metadata_map); + return 0; +} + +static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + errcode_t retval; + + retval = e2fsck_pass1_thread_join_one(global_ctx, thread_ctx); ext2fs_free_mem(&thread_ctx->fs); ext2fs_free_mem(&thread_ctx); - return 0; + + return retval; } void e2fsck_pass1_multithread(e2fsck_t ctx) From patchwork Wed Apr 8 10:44:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267869 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=fnBPfdqC; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CP6XYRz9sQx for ; Wed, 8 Apr 2020 20:45:53 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728228AbgDHKpx (ORCPT ); Wed, 8 Apr 2020 06:45:53 -0400 Received: from mail-pj1-f66.google.com ([209.85.216.66]:32929 "EHLO mail-pj1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKpx (ORCPT ); Wed, 8 Apr 2020 06:45:53 -0400 Received: by mail-pj1-f66.google.com with SMTP id cp9so2069778pjb.0 for ; Wed, 08 Apr 2020 03:45:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6zXhq3xOXGn+EHl8xMpXGWkGwAKQtIqVmt83aW0MwwE=; b=fnBPfdqCFIlLdPj7iN+KpT8sxEK8G5ge2uUnTVNGYOQgBWZALrCRe1hc42Evw15BkM ludoYA5G0oIWP5GbbVFDouYnJP/rdxclAemaZwLzaLCeqowsf0UHBjUoj/zEGKa1X0BO cRJvvKyoRu+n/nZCeZOaV1Gha6uSDSUozPlOojnexd5IJ9lXX2iCcjSTk3pXqyDdfgqe 4JzyEp9V5hwn7v2d9N6wOpAGJRv4WYAAW5AhbRg/K+flMThwFrFEVxLQ0Ms2Mc8ii03Z 8pGoqU/mjWsximj5ZAXy1rRLA9UuxuCOVjhQoj/H3CRWkwJi0iP0fry8y9A31qZhah5w cGBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6zXhq3xOXGn+EHl8xMpXGWkGwAKQtIqVmt83aW0MwwE=; b=POKmfM6KqdqTc0M0IXipoYVJtXmy9SzowI+ecc9CsWGMUeRLDKGJ/kE8xUg6g5/36k gxLJI2TihGFwn0YmJqysnfdvvYoRouMp+VTAhYXhWKN0dcMLbHMNAadIKpGs7eX1fOxH MlODNjn/D5LRT8k2Ag/pIRVQNs99HbSWISL9O/UYzMPYqfma277WQoDN9CqhCeUh2Gel sh6FJ1seWt4VhYsb6YIRL/PlYGlcofkr3J+80kgCeam6f+JigKrU044cTU8rLyiyisQ8 FJJ3j20alTDTqTDsKs/6gujWvyisp9C4h2KkzoLdUMufMW+h7bVKGVBXByAMEb0ITwdP L8+A== X-Gm-Message-State: AGi0PuYhzETpnLcWo/KrFRbcizNfrcd3wwM+RspIgXEREEidStbElMNM 195CIe0iRIHm3ysnnnIQA5AFNAM3pV8= X-Google-Smtp-Source: APiQypLX6ffl6rcAGswYR8SrG+6KDMUpxGHPS3m21lWm2KFC8rDgQNVztKw+6JmydjGu+MUA7HDkYQ== X-Received: by 2002:a17:902:444:: with SMTP id 62mr6599429ple.109.1586342752127; Wed, 08 Apr 2020 03:45:52 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:51 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 09/46] e2fsck: copy badblocks when copying fs Date: Wed, 8 Apr 2020 19:44:37 +0900 Message-Id: <1586342714-12536-10-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch copies badblocks when the copying fs. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 55 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 47 insertions(+), 8 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index f3337bde..de56562c 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2163,10 +2163,23 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) src->dblist = NULL; } + if (src->badblocks) { + retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); + if (retval) + goto out_dblist; + + ext2fs_badblocks_list_free(src->badblocks); + src->badblocks = NULL; + } return 0; + +out_dblist: + ext2fs_free_dblist(dest->dblist); + dest->dblist = NULL; + return retval; } -static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) +static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) { errcode_t retval = 0; @@ -2178,6 +2191,32 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) PASS1_COPY_FS_BITMAP(dest, src, inode_map); PASS1_COPY_FS_BITMAP(dest, src, block_map); + if (src->dblist) { + retval = ext2fs_copy_dblist(src->dblist, &dest->dblist); + if (retval) + return retval; + /* The ext2fs_copy_dblist() uses the src->fs as the fs */ + dest->dblist->fs = dest; + } + + if (src->badblocks) { + retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); + if (retval) + goto out_dblist; + } + return 0; +out_dblist: + ext2fs_free_dblist(dest->dblist); + dest->dblist = NULL; + return retval; +} + +static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) +{ + errcode_t retval; + + retval = _e2fsck_pass1_merge_fs(dest, src); + /* icache will be rebuilt if needed, so do not copy from @src */ if (src->icache) { ext2fs_free_inode_cache(src->icache); @@ -2185,16 +2224,16 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) } dest->icache = NULL; - if (dest->dblist) { - retval = ext2fs_copy_dblist(src->dblist, &dest->dblist); - if (retval == 0) { - /* The ext2fs_copy_dblist() uses the src->fs as the fs */ - dest->dblist->fs = dest; - } - + if (src->dblist) { ext2fs_free_dblist(src->dblist); src->dblist = NULL; } + + if (src->badblocks) { + ext2fs_badblocks_list_free(src->badblocks); + src->badblocks = NULL; + } + return retval; } From patchwork Wed Apr 8 10:44:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267870 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Vf6NjiOS; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CT6w4bz9sQx for ; Wed, 8 Apr 2020 20:45:57 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728231AbgDHKp5 (ORCPT ); Wed, 8 Apr 2020 06:45:57 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:34551 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726734AbgDHKp5 (ORCPT ); Wed, 8 Apr 2020 06:45:57 -0400 Received: by mail-pg1-f194.google.com with SMTP id l14so3165829pgb.1 for ; Wed, 08 Apr 2020 03:45:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xP864B6n00DWI8Mf+UC9hdnca8hS7dAYH/apD6+/5jQ=; b=Vf6NjiOSYxwoElUMRBvVpJ6fM2YOOjTfHE6KuD82aiCYcbaEb4t3eR8A9ytJwFKSdx JQThT100+o4IkTZBx0Htb6DurIlSnP3+z+rFWKXL1jcVbpa05ROa08bmyfFzFAUs2e1q SbBjaS+adeTIK96wcMqCLrhyPLdgOfkneXWNnUKboTCJJkzH5puzZTg2Fq3kBloc49me ucp4KjriBLhUyVFQQC+EIDD+Yr2yxMabMeHW+qsG3A4JLco1Puqjiv2ojWx4g2jeWku5 Hf/Uk3LbYzsyZoNl4Q+V+E+2YGMFEZ4WDOvhBS1/8tp4ZlEfE7Glh1UwZRhYF7Y3BiiQ 9Czg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xP864B6n00DWI8Mf+UC9hdnca8hS7dAYH/apD6+/5jQ=; b=qN9aI8dfZ8to0Q03JvOtUyvfZd6CbMN4eUM4x9ishhvs9lzIS06A5H6Dewxvrp0JNF AWCFsbFeUyzc4Ig9XulbujHOl2iBdTRQ/VC2UWwgnUPVhHo1UCg8sxByo1+iL9gzrlcA +WUw8S4iQJoQiXpiG0f8VPUSy0oHsr57xXEuerGuP501a52RsaqZ8SpMo+POU10IZXmp 9KlWQNo6GSwK124dJ28uBfb9S6lfkeki1uuITk51KouE/CkbkT4IYYUIrRFo+6uEwIqy 6mhXuueyxh2hC7LjgdpQKhaSnj35v8Mz8XP6JyKevr7PtMDt0SdDQVltMOu1b/c71Bih nj6A== X-Gm-Message-State: AGi0PuYRM3bq/eZ8pu07sGdkUezMkzHnbO9FmadJW3MfqmI8MxfcSSXn dMGGWHAyXBU/HqQhML28SQbz8vd8Gkg= X-Google-Smtp-Source: APiQypLZN1ea24lQKPgLoVCCEpP0Er3xlSdObLeAdiwgnR+Ut5M6konG9nMhoPOZDKKuwgeSYRXIzg== X-Received: by 2002:a63:cf49:: with SMTP id b9mr6468776pgj.221.1586342754530; Wed, 08 Apr 2020 03:45:54 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:53 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 10/46] e2fsck: open io-channel when copying fs Date: Wed, 8 Apr 2020 19:44:38 +0900 Message-Id: <1586342714-12536-11-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch also add writethrough flag to the thread io-channel. When multiple threads write the same disk, we don't want the data being saved in memory cache. This will be useful in the future, but even without that flag, the tests can be passed too. This patch also cleanup the io channel cache of the global context. Otherwise, after pass1 step, the next steps would old data saved in the cache. And the cached data might have already been overwritten in the pass1 step. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 1 + e2fsck/pass1.c | 71 ++++++++++++++++++++++++++++++++++++++++++-- e2fsck/unix.c | 1 + lib/ext2fs/ext2_io.h | 2 ++ lib/ext2fs/undo_io.c | 19 ++++++++++++ lib/ext2fs/unix_io.c | 24 +++++++++++++-- 6 files changed, 112 insertions(+), 6 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index c63b1852..58fb49c5 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -244,6 +244,7 @@ struct e2fsck_struct { ino_t free_inodes; int mount_flags; int openfs_flags; + io_manager io_manager; blkid_cache blkid; /* blkid cache */ #ifdef HAVE_SETJMP_H jmp_buf abort_loc; diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index de56562c..900d6cad 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2133,7 +2133,37 @@ do { \ } \ } while (0) -static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) +static errcode_t pass1_open_io_channel(ext2_filsys fs, + const char *io_options, + io_manager manager, int flags) +{ + errcode_t retval; + unsigned int io_flags = 0; + + if (flags & EXT2_FLAG_RW) + io_flags |= IO_FLAG_RW; + if (flags & EXT2_FLAG_EXCLUSIVE) + io_flags |= IO_FLAG_EXCLUSIVE; + if (flags & EXT2_FLAG_DIRECT_IO) + io_flags |= IO_FLAG_DIRECT_IO; + retval = manager->open(fs->device_name, io_flags, &fs->io); + if (retval) + return retval; + + if (io_options && + (retval = io_channel_set_options(fs->io, io_options))) + goto out_close; + fs->image_io = fs->io; + fs->io->app_data = fs; + + return 0; +out_close: + io_channel_close(fs->io); + return retval; +} + +static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, e2fsck_t src_context, + ext2_filsys src) { errcode_t retval; @@ -2171,6 +2201,33 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) ext2fs_badblocks_list_free(src->badblocks); src->badblocks = NULL; } + + /* disable it for now */ + src_context->openfs_flags &= ~EXT2_FLAG_EXCLUSIVE; + retval = pass1_open_io_channel(dest, src_context->io_options, + src_context->io_manager, + src_context->openfs_flags); + if (retval) + goto out_dblist; + + /* Block size might not be default */ + io_channel_set_blksize(dest->io, src->io->block_size); + ehandler_init(dest->io); + + assert(dest->io->magic == src->io->magic); + assert(dest->io->manager == src->io->manager); + assert(strcmp(dest->io->name, src->io->name) == 0); + assert(dest->io->block_size == src->io->block_size); + assert(dest->io->read_error == src->io->read_error); + assert(dest->io->write_error == src->io->write_error); + assert(dest->io->refcount == src->io->refcount); + assert(dest->io->flags == src->io->flags); + assert(dest->io->app_data == dest); + assert(src->io->app_data == src); + assert(dest->io->align == src->io->align); + + /* The data should be written to disk immediately */ + dest->io->flags |= CHANNEL_FLAGS_WRITETHROUGH; return 0; out_dblist: @@ -2181,9 +2238,15 @@ out_dblist: static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) { - errcode_t retval = 0; + errcode_t retval = 0; + io_channel dest_io; + io_channel dest_image_io; + dest_io = dest->io; + dest_image_io = dest->image_io; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); + dest->io = dest_io; + dest->image_io = dest_image_io; /* * PASS1_COPY_FS_BITMAP might return directly from this function, * so please do NOT leave any garbage behind after returning. @@ -2204,6 +2267,7 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) if (retval) goto out_dblist; } + io_channel_close(src->io); return 0; out_dblist: ext2fs_free_dblist(dest->dblist); @@ -2270,7 +2334,8 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre goto out_context; } - retval = e2fsck_pass1_copy_fs(thread_fs, global_fs); + io_channel_flush_cleanup(global_fs->io); + retval = e2fsck_pass1_copy_fs(thread_fs, global_ctx, global_fs); if (retval) { com_err(global_ctx->program_name, retval, "while copying fs"); goto out_fs; diff --git a/e2fsck/unix.c b/e2fsck/unix.c index 8226aff7..fff7376c 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -1524,6 +1524,7 @@ restart: } ctx->openfs_flags = flags; + ctx->io_manager = io_ptr; retval = try_open_fs(ctx, flags, io_ptr, &fs); if (!ctx->superblock && !(ctx->options & E2F_OPT_PREEN) && diff --git a/lib/ext2fs/ext2_io.h b/lib/ext2fs/ext2_io.h index 5540900a..4ad2fec8 100644 --- a/lib/ext2fs/ext2_io.h +++ b/lib/ext2fs/ext2_io.h @@ -81,6 +81,7 @@ struct struct_io_manager { errcode_t (*write_blk)(io_channel channel, unsigned long block, int count, const void *data); errcode_t (*flush)(io_channel channel); + errcode_t (*flush_cleanup)(io_channel channel); errcode_t (*write_byte)(io_channel channel, unsigned long offset, int count, const void *data); errcode_t (*set_option)(io_channel channel, const char *option, @@ -113,6 +114,7 @@ struct struct_io_manager { #define io_channel_read_blk(c,b,n,d) ((c)->manager->read_blk((c),b,n,d)) #define io_channel_write_blk(c,b,n,d) ((c)->manager->write_blk((c),b,n,d)) #define io_channel_flush(c) ((c)->manager->flush((c))) +#define io_channel_flush_cleanup(c) ((c)->manager->flush_cleanup((c))) #define io_channel_bumpcount(c) ((c)->refcount++) /* io_manager.c */ diff --git a/lib/ext2fs/undo_io.c b/lib/ext2fs/undo_io.c index 19862414..1391a534 100644 --- a/lib/ext2fs/undo_io.c +++ b/lib/ext2fs/undo_io.c @@ -1020,6 +1020,24 @@ static errcode_t undo_flush(io_channel channel) return retval; } +/* + * Flush data buffers to disk and cleanup the cache. + */ +static errcode_t undo_flush_cleanup(io_channel channel) +{ + errcode_t retval = 0; + struct undo_private_data *data; + + EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL); + data = (struct undo_private_data *) channel->private_data; + EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL); + + if (data->real) + retval = io_channel_flush_cleanup(data->real); + + return retval; +} + static errcode_t undo_set_option(io_channel channel, const char *option, const char *arg) { @@ -1091,6 +1109,7 @@ static struct struct_io_manager struct_undo_manager = { .read_blk = undo_read_blk, .write_blk = undo_write_blk, .flush = undo_flush, + .flush_cleanup = undo_flush_cleanup, .write_byte = undo_write_byte, .set_option = undo_set_option, .get_stats = undo_get_stats, diff --git a/lib/ext2fs/unix_io.c b/lib/ext2fs/unix_io.c index 628e60c3..c9defd4b 100644 --- a/lib/ext2fs/unix_io.c +++ b/lib/ext2fs/unix_io.c @@ -1030,9 +1030,9 @@ static errcode_t unix_write_byte(io_channel channel, unsigned long offset, } /* - * Flush data buffers to disk. + * Flush data buffers to disk and invalidate cache if needed */ -static errcode_t unix_flush(io_channel channel) +static errcode_t _unix_flush(io_channel channel, int invalidate) { struct unix_private_data *data; errcode_t retval = 0; @@ -1042,7 +1042,7 @@ static errcode_t unix_flush(io_channel channel) EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL); #ifndef NO_IO_CACHE - retval = flush_cached_blocks(channel, data, 0); + retval = flush_cached_blocks(channel, data, invalidate); #endif #ifdef HAVE_FSYNC if (!retval && fsync(data->dev) != 0) @@ -1051,6 +1051,22 @@ static errcode_t unix_flush(io_channel channel) return retval; } +/* + * Flush data buffers to disk. + */ +static errcode_t unix_flush(io_channel channel) +{ + return _unix_flush(channel, 0); +} + +/* + * Flush data buffers to disk and invalidate cache. + */ +static errcode_t unix_flush_cleanup(io_channel channel) +{ + return _unix_flush(channel, 1); +} + static errcode_t unix_set_option(io_channel channel, const char *option, const char *arg) { @@ -1225,6 +1241,7 @@ static struct struct_io_manager struct_unix_manager = { .discard = unix_discard, .cache_readahead = unix_cache_readahead, .zeroout = unix_zeroout, + .flush_cleanup = unix_flush_cleanup, }; io_manager unix_io_manager = &struct_unix_manager; @@ -1246,6 +1263,7 @@ static struct struct_io_manager struct_unixfd_manager = { .discard = unix_discard, .cache_readahead = unix_cache_readahead, .zeroout = unix_zeroout, + .flush_cleanup = unix_flush_cleanup, }; io_manager unixfd_io_manager = &struct_unixfd_manager; From patchwork Wed Apr 8 10:44:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267871 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=lQxjVmQH; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CV4q0gz9sSX for ; Wed, 8 Apr 2020 20:45:58 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728232AbgDHKp6 (ORCPT ); Wed, 8 Apr 2020 06:45:58 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:38901 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728229AbgDHKp6 (ORCPT ); Wed, 8 Apr 2020 06:45:58 -0400 Received: by mail-pf1-f193.google.com with SMTP id c21so2231853pfo.5 for ; Wed, 08 Apr 2020 03:45:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=THMxYIUynLkJkJS+9c3F2Unnr62wETxmWRutxcsukhw=; b=lQxjVmQH1o+2njwP1WYM/nYUQhUQS46zn4hBvyRFqhf5YBHAYTelOjFHwoCzcYuK2a JHZrHP1zeSHJpfzZ8uui+d7wclKkPCSkT7KKhQ5gIR/9txtWRQnCHphTcNY2aQaglXYk h+76fhdyuBcJkABteWZaJGSCY5aruWRCAcJ7/gKBjnwfx21lu+4a7O4n2rv10uXd3nbX aecTf8LqsmINQPjizf6UnR4lE/B7EDcboe1hgzWOFhPDQ3v2tgwbtKdePGeHjNcm3zvC n5xt3xPEEEgkKySX8EerjM9KrPCO+crBKvFQpwbOII0PxG5YfCvLbZpAVRRZfh5Kw30C WPMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=THMxYIUynLkJkJS+9c3F2Unnr62wETxmWRutxcsukhw=; b=eKwl9XPN/rJDcJewIASjmSm1qVpn3Hq2KrgpboC4qoG/fYhbkS2Sz/O/N5+I3X9CaI qSyToTyClRUZqI9zoFjuqKG8nwPf/nBa/d4L4Z684MQkZOsLLY7odkQ4h1A5C+Yau9D1 DRea7kdH7gqrlnbI9W3MOdao+Hl+DDWanWZwcqvA1vaMsOuMA30GW022V1vMVEqRTc2n ntBzk/rsqctVP8Ob1HK6wAjC2bbEKHlK3jfZlJiT0ns1ypg4z9nCotpWswDipAzQxUu1 ygz3m0JKSba9sG+mtmiWCOiPljE4Um6Eak3qJ+bJ8NqEZY4IPWabBgfCUbgyWav7rS9D t7kA== X-Gm-Message-State: AGi0PubHKVF/r8lGRaNXwq5yteHxb7l3KEsNYRrInXGoSi1EKDuYT22w BAY2irFxcBq3+x4cfrspZF5tmeu/50M= X-Google-Smtp-Source: APiQypLHK24TtsseDUpblk//h5QjUZRWriGDU2ZpkKmcXt+ypoxqz1IaOAD0kQ6wcDuitSUAsxEREQ== X-Received: by 2002:a63:fd11:: with SMTP id d17mr6721147pgh.213.1586342756638; Wed, 08 Apr 2020 03:45:56 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:56 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 11/46] e2fsck: create logs for mult-threads Date: Wed, 8 Apr 2020 19:44:39 +0900 Message-Id: <1586342714-12536-12-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi When multi-threads are used, different logs should be created for different threads. Each thread has log files with subfix of ".$THREAD_INDEX". And this patch adds f_multithread_logfile test case. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 2 ++ e2fsck/logfile.c | 11 +++++++++- e2fsck/pass1.c | 24 ++++++++++++++++----- tests/f_multithread_logfile/expect.1 | 23 ++++++++++++++++++++ tests/f_multithread_logfile/image.gz | 1 + tests/f_multithread_logfile/name | 1 + tests/f_multithread_logfile/script | 32 ++++++++++++++++++++++++++++ 7 files changed, 88 insertions(+), 6 deletions(-) create mode 100644 tests/f_multithread_logfile/expect.1 create mode 120000 tests/f_multithread_logfile/image.gz create mode 100644 tests/f_multithread_logfile/name create mode 100644 tests/f_multithread_logfile/script diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 58fb49c5..25aaea20 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -380,6 +380,8 @@ struct e2fsck_struct { */ ext2_ino_t stashed_ino; struct ext2_inode *stashed_inode; + /* Thread index, if global_ctx is null, this field is useless */ + int thread_index; /* * Directory information diff --git a/e2fsck/logfile.c b/e2fsck/logfile.c index 3eeefd19..4e8de342 100644 --- a/e2fsck/logfile.c +++ b/e2fsck/logfile.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "e2fsck.h" #include @@ -291,6 +292,8 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) struct string s, s1, s2; char *s0 = 0, *log_dir = 0, *log_fn = 0; int log_dir_wait = 0; + int string_size; + char string_index[4]; s.s = s1.s = s2.s = 0; @@ -307,6 +310,13 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) goto out; expand_logfn(ctx, log_fn, &s); + if (ctx->global_ctx) { + assert(ctx->thread_index < 1000); + sprintf(string_index, "%03d", ctx->thread_index); + append_string(&s, ".", 1); + append_string(&s, string_index, 0); + } + if ((log_fn[0] == '/') || !log_dir || !log_dir[0]) s0 = s.s; @@ -325,7 +335,6 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) append_string(&s2, log_dir, 0); append_string(&s2, "/", 1); append_string(&s2, s.s, 0); - printf("%s\n", s2.s); } if (s0) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 900d6cad..ed49b59b 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2342,6 +2342,9 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre } thread_fs->priv_data = thread_context; + thread_context->thread_index = 0; + set_up_logging(thread_context); + thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; @@ -2354,12 +2357,14 @@ out_context: static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { - errcode_t retval; - int flags = global_ctx->flags; - ext2_filsys thread_fs = thread_ctx->fs; - ext2_filsys global_fs = global_ctx->fs; + errcode_t retval; + int flags = global_ctx->flags; + ext2_filsys thread_fs = thread_ctx->fs; + ext2_filsys global_fs = global_ctx->fs; + FILE *global_logf = global_ctx->logf; + FILE *global_problem_logf = global_ctx->problem_logf; #ifdef HAVE_SETJMP_H - jmp_buf old_jmp; + jmp_buf old_jmp; memcpy(old_jmp, global_ctx->abort_loc, sizeof(jmp_buf)); #endif @@ -2378,6 +2383,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx } global_fs->priv_data = global_ctx; global_ctx->fs = global_fs; + global_ctx->logf = global_logf; + global_ctx->problem_logf = global_problem_logf; /* * PASS1_COPY_CTX_BITMAP might return directly from this function, @@ -2393,6 +2400,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_dup_map); PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_ea_map); PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_metadata_map); + return 0; } @@ -2402,6 +2410,12 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) retval = e2fsck_pass1_thread_join_one(global_ctx, thread_ctx); ext2fs_free_mem(&thread_ctx->fs); + if (thread_ctx->logf) + fclose(thread_ctx->logf); + if (thread_ctx->problem_logf) { + fputs("\n", thread_ctx->problem_logf); + fclose(thread_ctx->problem_logf); + } ext2fs_free_mem(&thread_ctx); return retval; diff --git a/tests/f_multithread_logfile/expect.1 b/tests/f_multithread_logfile/expect.1 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_logfile/expect.1 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_logfile/image.gz b/tests/f_multithread_logfile/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_logfile/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_logfile/name b/tests/f_multithread_logfile/name new file mode 100644 index 00000000..b8fcb997 --- /dev/null +++ b/tests/f_multithread_logfile/name @@ -0,0 +1 @@ +test "e2fsck -m" option works with "--E log_filename=" diff --git a/tests/f_multithread_logfile/script b/tests/f_multithread_logfile/script new file mode 100644 index 00000000..d7042a03 --- /dev/null +++ b/tests/f_multithread_logfile/script @@ -0,0 +1,32 @@ +LOG_FNAME="f_multithread_logfile_xxx" +FSCK_OPT="-fy -m -y -E log_filename=$LOG_FNAME" +SKIP_VERIFY="true" +ONE_PASS_ONLY="true" +SKIP_CLEANUP="true" + +rm -f $LOG_FNAME.* $LOG_FNAME + +. $cmd_dir/run_e2fsck + +rm -f $test_name.ok $test_name.failed +cmp -s $OUT1 $EXP1 +status1=$? + +if [ "$status1" -eq 0 ]; then + if [ ! -f $LOG_FNAME -o ! -f $LOG_FNAME.000 ]; then + echo "$LOG_FNAME or $LOG_FNAME.000 is not created" > $test_name.failed + echo "$test_name: $test_description: failed" + else + echo "$test_name: $test_description: ok" + touch $test_name.ok + fi +else + diff $DIFF_OPTS $test_dir/expect.1 \ + $test_name.1.log >> $test_name.failed + echo "$test_name: $test_description: failed" +fi + +unset IMAGE FSCK_OPT SECOND_FSCK_OPT OUT1 OUT2 EXP1 EXP2 +unset SKIP_VERIFY SKIP_CLEANUP SKIP_GUNZIP ONE_PASS_ONLY PREP_CMD +unset DESCRIPTION SKIP_UNLINK AFTER_CMD PASS_ZERO +unset LOG_FINAME From patchwork Wed Apr 8 10:44:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267872 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=F4viFzu8; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1CZ4ZnQz9sQx for ; Wed, 8 Apr 2020 20:46:02 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728237AbgDHKqC (ORCPT ); Wed, 8 Apr 2020 06:46:02 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:34557 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728229AbgDHKqC (ORCPT ); Wed, 8 Apr 2020 06:46:02 -0400 Received: by mail-pg1-f195.google.com with SMTP id l14so3165897pgb.1 for ; Wed, 08 Apr 2020 03:45:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/Kqpghk4PVi+Y9uAXdJ0hkRRkzWpjcrImjeHsqLCbJk=; b=F4viFzu8sYdWaFmfECD22vwYGydCpNMAKHb1/2IEyW+vasDiRdt5yuHnXJ7tR9sCTi 7KMoPzbe0irWz8j5ZknwPiKpVCNUJ19hea4Gw4olCZkipA0iEN91qusnfwV8b80pMPfo u85UeUPJLPxiJI6wUpZu9jnRzqXwdfx3XsjHdt8fddb1J24drLVFwHKRPHhpTwIBgkFc yUO3K2Gl3HIJBNBJ0bDHbN12kSguT0yCH+BlIkimWZGITvKnz5Ugln0YO9xHxp1LH5tW UJ6DfEp88rOZluSRNh71oqMyEV7h/4dXmZ3U6t8WmQpRl1O5e/1k3YAlTtDUDDcC0yb+ fgRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/Kqpghk4PVi+Y9uAXdJ0hkRRkzWpjcrImjeHsqLCbJk=; b=pMih7X/yNFplb4PS894jgFrUCIU05OR85trEKT8y1dCHi0tEhPvxa91kFfDEgjCac+ O9U3+fARknTJrqP7ehfbHPLlh1OP2aTYbJtqls28jyq9ytH63D6Cc8iqIofAeayIdl0V yd7zwPQqUTMJG8YJ5NlS6QciUWH7KECPhLbekD1a19MHsXiekO7Tv3zNnOB3DaDparU3 RS1P7Pl65GhBiuP33mkHEZoBUnnNdcNuogZrpcx2i86Ivsw4zG/BJUN+wXqLm6pUpplk u6i6kszZIvdmiwHz241RanKeTzeE/aW8gYzEk5qrGrCJ185eY1yfqpHgQiOMbnjV5Sw7 T2nQ== X-Gm-Message-State: AGi0PuZAwQ/o3YwJPlvyGXxi6sIm0gaehot3f0cpzJ52Utm+sUkzzIa3 yHT5ZXGQVb4xO2dvMyeqwerdumWMNhg= X-Google-Smtp-Source: APiQypIf8k0HGhb6L6InVDDhTUS/iiL/HHF5tqYzsjrVFSjRnVCjZXxB8++5EaPy5EuIpwjxJEqLWA== X-Received: by 2002:aa7:9a5d:: with SMTP id x29mr6695992pfj.284.1586342758741; Wed, 08 Apr 2020 03:45:58 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:45:58 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 12/46] e2fsck: create one thread to fsck Date: Wed, 8 Apr 2020 19:44:40 +0900 Message-Id: <1586342714-12536-13-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch creates only one thread to do pass1 check. The same codes can be used to create multiple threads, but other functions need to be modified to get ready for that. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- configure.ac | 6 ++ e2fsck/e2fsck.h | 11 ++++ e2fsck/pass1.c | 144 ++++++++++++++++++++++++++++++++++++++++++------ 3 files changed, 143 insertions(+), 18 deletions(-) diff --git a/configure.ac b/configure.ac index 18e434bc..a8d3784c 100644 --- a/configure.ac +++ b/configure.ac @@ -107,6 +107,12 @@ if test "$GCC" = yes; then fi AC_PROG_CPP dnl +dnl Add pthread to the CFLAGS/LDFLAGS +dnl +CFLAGS="$CFLAGS -pthread" +LDFLAGS="$CFLAGS -pthread" +LDFLAGS_STATIC="$LDFLAGS_STATIC -pthread" +dnl dnl Alpha computers use fast and imprecise floating point code that may dnl miss exceptions by default. Force sane options if we're using GCC. AC_MSG_CHECKING(for additional special compiler flags) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 25aaea20..7e00648e 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -433,6 +433,17 @@ struct e2fsck_struct { }; +struct e2fsck_thread_info { + /* ID returned by pthread_create() */ + pthread_t eti_thread_id; + /* Application-defined thread index */ + int eti_thread_index; + /* Thread has been started */ + int eti_started; + /* Context used for this thread */ + e2fsck_t eti_thread_ctx; +}; + /* Data structures to evaluate whether an extent tree needs rebuilding. */ struct extent_tree_level { unsigned int num_extents; diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index ed49b59b..890819f7 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -47,6 +47,7 @@ #include #endif #include +#include #include "e2fsck.h" #include @@ -1172,7 +1173,7 @@ static int e2fsck_should_abort(e2fsck_t ctx) return 0; } -void e2fsck_pass1_thread(e2fsck_t ctx) +void _e2fsck_pass1(e2fsck_t ctx) { int i; __u64 max_sizes; @@ -2421,18 +2422,38 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) return retval; } -void e2fsck_pass1_multithread(e2fsck_t ctx) +static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, + int num_threads, e2fsck_t global_ctx) { - errcode_t retval; - e2fsck_t thread_ctx; + errcode_t rc; + errcode_t ret = 0; + int i; + struct e2fsck_thread_info *pinfo; - retval = e2fsck_pass1_thread_prepare(ctx, &thread_ctx); - if (retval) { - com_err(ctx->program_name, 0, - _("while preparing pass1 thread\n")); - ctx->flags |= E2F_FLAG_ABORT; - return; + for (i = 0; i < num_threads; i++) { + pinfo = &infos[i]; + + if (!pinfo->eti_started) + continue; + + rc = pthread_join(pinfo->eti_thread_id, NULL); + if (rc) { + com_err(global_ctx->program_name, rc, + _("while joining thread\n")); + if (ret == 0) + ret = rc; + } + e2fsck_pass1_thread_join(global_ctx, infos[i].eti_thread_ctx); } + free(infos); + + return ret; +} + +static void *e2fsck_pass1_thread(void *arg) +{ + struct e2fsck_thread_info *info = arg; + e2fsck_t thread_ctx = info->eti_thread_ctx; #ifdef HAVE_SETJMP_H /* @@ -2443,20 +2464,107 @@ void e2fsck_pass1_multithread(e2fsck_t ctx) */ if (setjmp(thread_ctx->abort_loc)) { thread_ctx->flags &= ~E2F_FLAG_SETJMP_OK; - e2fsck_pass1_thread_join(ctx, thread_ctx); - return; + goto out; } thread_ctx->flags |= E2F_FLAG_SETJMP_OK; #endif - e2fsck_pass1_thread(thread_ctx); - retval = e2fsck_pass1_thread_join(ctx, thread_ctx); + _e2fsck_pass1(thread_ctx); + +out: + return NULL; +} + +static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, + int num_threads, e2fsck_t global_ctx) +{ + struct e2fsck_thread_info *infos; + pthread_attr_t attr; + errcode_t retval; + errcode_t ret; + struct e2fsck_thread_info *tmp_pinfo; + int i; + e2fsck_t thread_ctx; + + retval = pthread_attr_init(&attr); if (retval) { - com_err(ctx->program_name, 0, - _("while joining pass1 thread\n")); - ctx->flags |= E2F_FLAG_ABORT; - return; + com_err(global_ctx->program_name, retval, + _("while setting pthread attribute\n")); + return retval; + } + + infos = calloc(num_threads, sizeof(struct e2fsck_thread_info)); + if (infos == NULL) { + retval = -ENOMEM; + com_err(global_ctx->program_name, retval, + _("while allocating memory for threads\n")); + pthread_attr_destroy(&attr); + return retval; } + + for (i = 0; i < num_threads; i++) { + tmp_pinfo = &infos[i]; + tmp_pinfo->eti_thread_index = i; + retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx); + if (retval) { + com_err(global_ctx->program_name, retval, + _("while preparing pass1 thread\n")); + break; + } + tmp_pinfo->eti_thread_ctx = thread_ctx; + + retval = pthread_create(&tmp_pinfo->eti_thread_id, &attr, + &e2fsck_pass1_thread, tmp_pinfo); + if (retval) { + com_err(global_ctx->program_name, retval, + _("while creating thread\n")); + e2fsck_pass1_thread_join(global_ctx, thread_ctx); + break; + } + + tmp_pinfo->eti_started = 1; + } + + /* destroy the thread attribute object, since it is no longer needed */ + ret = pthread_attr_destroy(&attr); + if (ret) { + com_err(global_ctx->program_name, ret, + _("while destroying thread attribute\n")); + if (retval == 0) + retval = ret; + } + + if (retval) { + e2fsck_pass1_threads_join(infos, num_threads, global_ctx); + return retval; + } + *pinfo = infos; + return 0; +} + +static void e2fsck_pass1_multithread(e2fsck_t global_ctx) +{ + struct e2fsck_thread_info *infos = NULL; + int num_threads = 1; + errcode_t retval; + + retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); + if (retval) { + com_err(global_ctx->program_name, retval, + _("while starting pass1 threads\n")); + goto out_abort; + } + + retval = e2fsck_pass1_threads_join(infos, num_threads, global_ctx); + if (retval) { + com_err(global_ctx->program_name, retval, + _("while joining pass1 threads\n")); + goto out_abort; + } + return; +out_abort: + global_ctx->flags |= E2F_FLAG_ABORT; + return; } void e2fsck_pass1(e2fsck_t ctx) From patchwork Wed Apr 8 10:44:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267873 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ZCE137VI; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cc4xG0z9sSG for ; Wed, 8 Apr 2020 20:46:04 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728241AbgDHKqE (ORCPT ); Wed, 8 Apr 2020 06:46:04 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:42217 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728239AbgDHKqE (ORCPT ); Wed, 8 Apr 2020 06:46:04 -0400 Received: by mail-pg1-f195.google.com with SMTP id g6so3140694pgs.9 for ; Wed, 08 Apr 2020 03:46:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=q7hmk7BP1ayFiCU/lJeIe+xT9L5cltDsdCDQDhW96gY=; b=ZCE137VILTJkgYYOE/9T6SqOyGR5TguITiqeOxICLIaYhoQOsGiW8y2g5OB4mXLYdU Bu1JP7UqC6nY23o0fiOfbayS/lgfgudCLYxz6gAQxGcXJp86kcdtB+zhNdDGELWO5sYF tdub+l13WL8nLsaESyXSx9fSYuc38i6OGZNysI3zheEW8U4OXVkF5c3NpVNN1axo8XhE 8xfLVVkKMsxd6r6fNQSJWeGpjDn38XwPgjKaByQG/FAcfVId+NCy/r6xsLzZQI8k2szK 5AHUt3ni5Wx/5wlRP5TwFIQhmBBQMqbSid4ipFxHt20Cz415NMJRyRlL4cHlHDGYsjmi YQag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=q7hmk7BP1ayFiCU/lJeIe+xT9L5cltDsdCDQDhW96gY=; b=fi68AVnjo/MfHncq3vBgjSgx9F8vvn36Hpu2n9iUaDtYu8mE/x5SnsmtlaEsq/NgRp lqHCNwIdXBgNWdNTUiqOvEU8cJhRysNJOSBMIfhn4/V8MG3Xlf7YR3rSt7PyfEiaKkE9 aPFQqyxDj0xj6OSoD+40iZkMrKYhUJvEE+3HoZaTxnqu/4DYRI0pRPRcnb28oRY079G2 pfSEHpOBZuHOKDYQPZF6zsr5vEzei0c8TbFDu8+hUXjsPKpGSqLWgxoAvYJwph/QrVdj 1CHqAvNvrFhoXF1tr/91AEKrgAIIYOabMRv4PTQFtZEcBoQJO2yqlUPxDbKARMXE7XSm DXBQ== X-Gm-Message-State: AGi0PuZ6TUB3SPWT9DmXBzzCMQub1Bmg0PtOnHgJCoOa1K6DZDMO7pc5 PJqPyGV3zu1shCSYgq6i4i07wLGwvtQ= X-Google-Smtp-Source: APiQypLEPlIe7inAqJtAOkH2XpH0Raj5Yv/PJTNgkwsrpLzavvRUnQrurOxDFbS0DUS+JbaJdD399A== X-Received: by 2002:aa7:86d3:: with SMTP id h19mr7408085pfo.297.1586342760841; Wed, 08 Apr 2020 03:46:00 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.45.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:00 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 13/46] e2fsck: add start/end group for thread Date: Wed, 8 Apr 2020 19:44:41 +0900 Message-Id: <1586342714-12536-14-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi When multi-threads are used for check, each thread needs to jump to different group in pass1 check. This patch adds the group jumping support. But still, only one thread is used to check. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 18 +++++++++++++-- e2fsck/logfile.c | 5 +++-- e2fsck/pass1.c | 46 +++++++++++++++++++++++++++++++++------ e2fsck/problem.c | 5 +++++ e2fsck/problem.h | 3 +++ lib/ext2fs/ext2_err.et.in | 3 +++ 6 files changed, 69 insertions(+), 11 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 7e00648e..f42da58e 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -227,6 +227,20 @@ typedef struct e2fsck_struct *e2fsck_t; #define MAX_EXTENT_DEPTH_COUNT 5 +/* + * Fields that used for multi-thread + */ +struct e2fsck_thread { + /* The start group number for this thread */ + dgrp_t et_group_start; + /* The end (not included) group number for this thread*/ + dgrp_t et_group_end; + /* The next group number to check */ + dgrp_t et_group_next; + /* Thread index */ + int et_thread_index; +}; + struct e2fsck_struct { /* ---- Following fields are never updated during the pass1 ---- */ /* Global context to get the cancel flag */ @@ -380,8 +394,8 @@ struct e2fsck_struct { */ ext2_ino_t stashed_ino; struct ext2_inode *stashed_inode; - /* Thread index, if global_ctx is null, this field is useless */ - int thread_index; + /* if @global_ctx is null, this field is useless */ + struct e2fsck_thread thread_info; /* * Directory information diff --git a/e2fsck/logfile.c b/e2fsck/logfile.c index 4e8de342..de0885e6 100644 --- a/e2fsck/logfile.c +++ b/e2fsck/logfile.c @@ -311,8 +311,9 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) expand_logfn(ctx, log_fn, &s); if (ctx->global_ctx) { - assert(ctx->thread_index < 1000); - sprintf(string_index, "%03d", ctx->thread_index); + assert(ctx->thread_info.et_thread_index < 1000); + sprintf(string_index, "%03d", + ctx->thread_info.et_thread_index); append_string(&s, ".", 1); append_string(&s, string_index, 0); } diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 890819f7..811d3991 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1378,6 +1378,19 @@ void _e2fsck_pass1(e2fsck_t ctx) /* Set up ctx->lost_and_found if possible */ (void) e2fsck_get_lost_and_found(ctx, 0); + if (ctx->global_ctx) { +#if 0 + printf("jumping to %d\n", ctx->thread_info.et_group_start); +#endif + pctx.errcode = ext2fs_inode_scan_goto_blockgroup(scan, + ctx->thread_info.et_group_start); + if (pctx.errcode) { + fix_problem(ctx, PR_1_PASS_HEADER, &pctx); + ctx->flags |= E2F_FLAG_ABORT; + goto endit; + } + } + while (1) { if (ino % (fs->super->s_inodes_per_group * 4) == 1) { if (e2fsck_mmp_update(fs)) @@ -1429,6 +1442,8 @@ void _e2fsck_pass1(e2fsck_t ctx) ext2fs_mark_inode_bitmap2(ctx->inode_used_map, ino); continue; } + if (pctx.errcode == EXT2_ET_SCAN_FINISHED) + break; if (pctx.errcode && pctx.errcode != EXT2_ET_INODE_CSUM_INVALID && pctx.errcode != EXT2_ET_INODE_IS_GARBAGE) { @@ -2302,12 +2317,15 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) return retval; } -static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) +static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, + e2fsck_t *thread_ctx, + int thread_index) { - errcode_t retval; - e2fsck_t thread_context; - ext2_filsys thread_fs; - ext2_filsys global_fs = global_ctx->fs; + errcode_t retval; + e2fsck_t thread_context; + ext2_filsys thread_fs; + ext2_filsys global_fs = global_ctx->fs; + struct e2fsck_thread *tinfo; assert(global_ctx->inode_used_map == NULL); assert(global_ctx->inode_dir_map == NULL); @@ -2343,9 +2361,15 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre } thread_fs->priv_data = thread_context; - thread_context->thread_index = 0; + thread_context->thread_info.et_thread_index = thread_index; set_up_logging(thread_context); + assert(thread_index == 0); + tinfo = &thread_context->thread_info; + tinfo->et_group_start = 0; + tinfo->et_group_next = 0; + tinfo->et_group_end = thread_fs->group_desc_count; + thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; @@ -2505,7 +2529,7 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, for (i = 0; i < num_threads; i++) { tmp_pinfo = &infos[i]; tmp_pinfo->eti_thread_index = i; - retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx); + retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, i); if (retval) { com_err(global_ctx->program_name, retval, _("while preparing pass1 thread\n")); @@ -2584,6 +2608,7 @@ static errcode_t scan_callback(ext2_filsys fs, { struct scan_callback_struct *scan_struct; e2fsck_t ctx; + struct e2fsck_thread *tinfo; scan_struct = (struct scan_callback_struct *) priv_data; ctx = scan_struct->ctx; @@ -2595,6 +2620,13 @@ static errcode_t scan_callback(ext2_filsys fs, ctx->fs->group_desc_count)) return EXT2_ET_CANCEL_REQUESTED; + if (ctx->global_ctx) { + tinfo = &ctx->thread_info; + tinfo->et_group_next++; + if (tinfo->et_group_next >= tinfo->et_group_end) + return EXT2_ET_SCAN_FINISHED; + } + return 0; } diff --git a/e2fsck/problem.c b/e2fsck/problem.c index c7c0ba98..23183db9 100644 --- a/e2fsck/problem.c +++ b/e2fsck/problem.c @@ -1269,6 +1269,11 @@ static struct e2fsck_problem problem_table[] = { N_("Encrypted @i %i has corrupt encryption @a.\n"), PROMPT_CLEAR_INODE, 0, 0, 0, 0 }, + /* Failed to goto block group */ + { PR_1_SCAN_GOTO, + N_("failed to goto block group"), + PROMPT_NONE, PR_FATAL, 0, 0, 0 }, + /* Pass 1b errors */ /* Pass 1B: Rescan for duplicate/bad blocks */ diff --git a/e2fsck/problem.h b/e2fsck/problem.h index c7f65f6d..37cad7f8 100644 --- a/e2fsck/problem.h +++ b/e2fsck/problem.h @@ -707,6 +707,9 @@ struct problem_context { /* Encrypted inode has corrupt encryption extended attribute */ #define PR_1_CORRUPT_ENCRYPTION_XATTR 0x01008B +/* Failed to goto block group */ +#define PR_1_SCAN_GOTO 0x01008C + /* * Pass 1b errors */ diff --git a/lib/ext2fs/ext2_err.et.in b/lib/ext2fs/ext2_err.et.in index 0c76fee6..cdb37423 100644 --- a/lib/ext2fs/ext2_err.et.in +++ b/lib/ext2fs/ext2_err.et.in @@ -548,4 +548,7 @@ ec EXT2_ET_EA_INODE_CORRUPTED, ec EXT2_ET_NO_GDESC, "Group descriptors not loaded" +ec EXT2_ET_SCAN_FINISHED, + "Scanning finished" + end From patchwork Wed Apr 8 10:44:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267874 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=QIHVm7Kc; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cd1VnYz9sSX for ; Wed, 8 Apr 2020 20:46:05 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728243AbgDHKqE (ORCPT ); Wed, 8 Apr 2020 06:46:04 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:39551 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728229AbgDHKqE (ORCPT ); Wed, 8 Apr 2020 06:46:04 -0400 Received: by mail-pf1-f194.google.com with SMTP id k15so2229980pfh.6 for ; Wed, 08 Apr 2020 03:46:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=AqvRCs3ZlO4YydxJaPbgxf7EDC02H6wPxawH/CU/hrw=; b=QIHVm7KclOvjAmFAXVLTBxUEgbVKzfWUQE42/5dOMP6e7oJ88Y+sUcsZtODv6B+UVj x4duBRqZMa/j7J3bq8hPUIrAVatg7SMXTzJrz6VmrBkh/Y4V7m9j6mmkX/jg61OagKZc /M5kRPq7xVXxsXBl0UMhq6YE7kOSG5wxbdE64fpZeEEGxCp8SYAJHt6L0DVVNYtCYvE7 bduwiFrs9ISYZG9nmbgr8boBNdiSIiZfIB6pDbt/n42oC+HMMNQmyftktB1D577bz6rQ 6AbK89RANqKCtWBPNY1qi/HhRkaV7A+Nw701irabhQqaOVoyVhlQ7Kk6DbWG4aL+s46f ZaIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=AqvRCs3ZlO4YydxJaPbgxf7EDC02H6wPxawH/CU/hrw=; b=iqEb4uB5NMdsiI6a257TMRAhnn8ulePzlXWjGWKOFdwlwR+LFIQEWv+i3izD8ak1U6 eRxp2ktQ2gk7fdNKIWcJ//DIRD6TxX6sEGYwA8Ncu0FbMalH6MXaTWs4xewf3dMtAYP5 YXicV3+ZoqPCqfd8/4PqxPbU2TBBH8NdTd4oG/F/FIlq9AiSeNRmxzEmKKTEv9lvzK7R emZzdvpqZiHWwhjMBx9MaKqIye64nSd77EFr5wbBHRgtNI44AQIu1+cNO8uwtBjoZbQP j/0w2sXyJ1091nrfpwFgiPcpzPs7jQ6VMoTVtbkXXXCTAYNHtOLWyt+kpU6fMrpYqje5 GQ+g== X-Gm-Message-State: AGi0PuYvDtyBpEfciUiXE4bqeZ7Zz45uzvjN4rcriQrUdKZDVp0PCXnL kvvZHkFKMiMiwNrUAdnb/AV2UH09d9Y= X-Google-Smtp-Source: APiQypJDf1FkojlMlxAeDXHecbbXhe6V/kmd/RieOegAG8Qrcb+j60U9FYNDFOu8sUhN70fnZ595EQ== X-Received: by 2002:a05:6a00:2b4:: with SMTP id q20mr7263007pfs.36.1586342763011; Wed, 08 Apr 2020 03:46:03 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:02 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 14/46] e2fsck: split groups to different threads Date: Wed, 8 Apr 2020 19:44:42 +0900 Message-Id: <1586342714-12536-15-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi The start/end groups of a thread is calculated according to the thread number. But still, only one thread is used to check. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 811d3991..8147e944 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2319,13 +2319,15 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx, - int thread_index) + int thread_index, + int num_threads) { errcode_t retval; e2fsck_t thread_context; ext2_filsys thread_fs; ext2_filsys global_fs = global_ctx->fs; struct e2fsck_thread *tinfo; + dgrp_t average_group; assert(global_ctx->inode_used_map == NULL); assert(global_ctx->inode_dir_map == NULL); @@ -2364,11 +2366,20 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, thread_context->thread_info.et_thread_index = thread_index; set_up_logging(thread_context); - assert(thread_index == 0); + /* + * Distribute work to multiple threads: + * Each thread work on fs->group_desc_count / nthread groups. + */ tinfo = &thread_context->thread_info; - tinfo->et_group_start = 0; - tinfo->et_group_next = 0; - tinfo->et_group_end = thread_fs->group_desc_count; + average_group = thread_fs->group_desc_count / num_threads; + if (average_group == 0) + average_group = 1; + tinfo->et_group_start = average_group * thread_index; + if (thread_index == num_threads - 1) + tinfo->et_group_end = thread_fs->group_desc_count; + else + tinfo->et_group_end = average_group * (thread_index + 1); + tinfo->et_group_next = tinfo->et_group_start; thread_context->fs = thread_fs; *thread_ctx = thread_context; @@ -2529,7 +2540,8 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, for (i = 0; i < num_threads; i++) { tmp_pinfo = &infos[i]; tmp_pinfo->eti_thread_index = i; - retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, i); + retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, + i, num_threads); if (retval) { com_err(global_ctx->program_name, retval, _("while preparing pass1 thread\n")); From patchwork Wed Apr 8 10:44:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267875 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=OxoWyfhs; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cg04G1z9sQx for ; Wed, 8 Apr 2020 20:46:07 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728247AbgDHKqG (ORCPT ); Wed, 8 Apr 2020 06:46:06 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:46953 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728229AbgDHKqG (ORCPT ); Wed, 8 Apr 2020 06:46:06 -0400 Received: by mail-pl1-f195.google.com with SMTP id s23so2354575plq.13 for ; Wed, 08 Apr 2020 03:46:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HxXssIsjNIiHDGoUNg67XbcvjpGUrFHeXSou9kKoFWE=; b=OxoWyfhscUvMCpEbKVwkbBJTCYdu1ZyZc6YXtsd/hOss0HWDh2B1DoVSPR/xzJlQYq zc7AsOJCIv0ycUDJMweY7BTO7X5KTSYgz/l7SU8egyZghKoBofiaOhkTEU8sBRNz1gag xBYYCu+s/J6kgDB+OHv4Y08CPNmy2TUzjKx1qBlARoskIfE0InJKlGDyIvc+peqZPVLa MBzrXFekMHk7BVPHNCsYhk7FRbI9kRomLo0SKYvNk+8N6SUnJSgi9WxeVdtEq8IbqOMS H4Gb6hWqKOUG9O6Dww/73xGCtAPS3p5avsBdneehRyFJWGUx+i9/yTsWpH4GkknM58YN /2bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HxXssIsjNIiHDGoUNg67XbcvjpGUrFHeXSou9kKoFWE=; b=b7LKVnQEvdvBWEn1yxQkg3MXsHmshfh9VHR/hKkT5hb97HHyZznEQU7ot9hhPa03rS yZdzg9pFgCR9p3j8+t/qdoBtpJrCXdabBfsYH9r/c8c4K4gqs5blG53emB7uqO6+Haz0 DiacWZT+Ktqtn1i5V6wGQhN8IsJaoi21oQRTJX2JOHseDpyadAHsekCtWt1/qDGFpmQF 9jo72/Blqctsn78PqawyrSDdWxdhdN+6vf7uoUIM6jXCyXSRq1n5NyNK3nzr2Fqnhl4b ZMRKo6Vy+Rto8JyQrckmHUxbM99ooGZUUj0I4kPx8I4ypRu76xMn2NCs585NS3zmjzxz gKDQ== X-Gm-Message-State: AGi0PuYS73y1zmdQ7ZSR/v6WBCsK0ZXGiGawSw1N3nl3msHaWeWbTMkK W1WhKw/YA1V6UU076Vwyk+OcsegVplI= X-Google-Smtp-Source: APiQypIMTU4Y+eSMAKK11TP7kKGiJeioRkMm6XKTj2EMieBP6NO+hspmleeUPmDpxHiQNBvJeiG+Ow== X-Received: by 2002:a17:90a:8546:: with SMTP id a6mr4732673pjw.8.1586342765146; Wed, 08 Apr 2020 03:46:05 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:04 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 15/46] e2fsck: print thread log properly Date: Wed, 8 Apr 2020 19:44:43 +0900 Message-Id: <1586342714-12536-16-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi When multi-thread fsck is enabled, logs printed from multiple threads could overlap with each other. The overlap sometimes makes the logs unreadable because log_out() is used multiple times for a single line. This patch adds leading [Thread XXX] to each logs if multi-thread is enabed by -m option. This patch also adds message to show the group ranges and inode numbers for each thread, which is useful for debuging multi-thread check. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 4 ++ e2fsck/pass1.c | 17 ++++++++- e2fsck/problem.c | 4 ++ e2fsck/util.c | 56 ++++++++++++++++++++++++++-- tests/f_multithread/expect.1 | 4 +- tests/f_multithread_logfile/expect.1 | 4 +- tests/f_multithread_no/expect.1 | 4 +- 7 files changed, 86 insertions(+), 7 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index f42da58e..48afc8f3 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -239,6 +239,10 @@ struct e2fsck_thread { dgrp_t et_group_next; /* Thread index */ int et_thread_index; + /* Scanned inode number */ + ext2_ino_t et_inode_number; + char et_log_buf[2048]; + char et_log_length; }; struct e2fsck_struct { diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 8147e944..7a66bdf9 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1453,6 +1453,8 @@ void _e2fsck_pass1(e2fsck_t ctx) } if (!ino) break; + if (ctx->global_ctx) + ctx->thread_info.et_inode_number++; pctx.ino = ino; pctx.inode = inode; ctx->stashed_ino = ino; @@ -2380,7 +2382,12 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, else tinfo->et_group_end = average_group * (thread_index + 1); tinfo->et_group_next = tinfo->et_group_start; - + tinfo->et_inode_number = 0; + tinfo->et_log_buf[0] = '\0'; + tinfo->et_log_length = 0; + if (thread_context->options & E2F_OPT_MULTITHREAD) + log_out(thread_context, _("Scan group range [%d, %d)\n"), + tinfo->et_group_start, tinfo->et_group_end); thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; @@ -2421,6 +2428,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->fs = global_fs; global_ctx->logf = global_logf; global_ctx->problem_logf = global_problem_logf; + global_ctx->global_ctx = NULL; /* * PASS1_COPY_CTX_BITMAP might return directly from this function, @@ -2507,6 +2515,12 @@ static void *e2fsck_pass1_thread(void *arg) _e2fsck_pass1(thread_ctx); out: + if (thread_ctx->options & E2F_OPT_MULTITHREAD) + log_out(thread_ctx, + _("Scanned group range [%lu, %lu), inodes %lu\n"), + thread_ctx->thread_info.et_group_start, + thread_ctx->thread_info.et_group_end, + thread_ctx->thread_info.et_inode_number); return NULL; } @@ -2634,6 +2648,7 @@ static errcode_t scan_callback(ext2_filsys fs, if (ctx->global_ctx) { tinfo = &ctx->thread_info; + //printf("iii group %d finished\n", tinfo->et_group_next); tinfo->et_group_next++; if (tinfo->et_group_next >= tinfo->et_group_end) return EXT2_ET_SCAN_FINISHED; diff --git a/e2fsck/problem.c b/e2fsck/problem.c index 23183db9..78ea195e 100644 --- a/e2fsck/problem.c +++ b/e2fsck/problem.c @@ -2476,6 +2476,10 @@ int fix_problem(e2fsck_t ctx, problem_t code, struct problem_context *pctx) if (*message) message = _(message); if (!suppress) { + if ((ctx->options & E2F_OPT_MULTITHREAD) && ctx->global_ctx) + printf("[Thread %d] ", + ctx->thread_info.et_thread_index); + if ((ctx->options & E2F_OPT_PREEN) && !(ptr->flags & PR_PREEN_NOHDR)) { printf("%s: ", ctx->device_name ? diff --git a/e2fsck/util.c b/e2fsck/util.c index db6a1cc1..3f5abfeb 100644 --- a/e2fsck/util.c +++ b/e2fsck/util.c @@ -11,6 +11,7 @@ #include "config.h" #include +#include #include #include #include @@ -88,13 +89,62 @@ out: exit(exit_value); } +static void thread_log_out(struct e2fsck_thread *tinfo) +{ + printf("[Thread %d] %s", tinfo->et_thread_index, + tinfo->et_log_buf); + tinfo->et_log_length = 0; + tinfo->et_log_buf[0] = '\0'; +} + void log_out(e2fsck_t ctx, const char *fmt, ...) { va_list pvar; + struct e2fsck_thread *tinfo; + int buf_size; + int msg_size; + int left_size; + int fmt_length = strlen(fmt); + + if ((ctx->options & E2F_OPT_MULTITHREAD) && ctx->global_ctx) { + tinfo = &ctx->thread_info; + buf_size = sizeof(tinfo->et_log_buf); + left_size = buf_size - tinfo->et_log_length; + + va_start(pvar, fmt); + msg_size = vsnprintf(tinfo->et_log_buf + tinfo->et_log_length, + left_size, fmt, pvar); + va_end(pvar); + + if (msg_size >= left_size) { + tinfo->et_log_buf[tinfo->et_log_length] = '\0'; + + assert(msg_size < buf_size); + if (msg_size < buf_size) { + thread_log_out(tinfo); + + va_start(pvar, fmt); + msg_size = vsnprintf(tinfo->et_log_buf, buf_size, + fmt, pvar); + va_end(pvar); + + tinfo->et_log_length += msg_size; + tinfo->et_log_buf[tinfo->et_log_length] = '\0'; + } + } else { + tinfo->et_log_length += msg_size; + tinfo->et_log_buf[tinfo->et_log_length] = '\0'; + } + + if (tinfo->et_log_length > 0 && + tinfo->et_log_buf[tinfo->et_log_length - 1] == '\n') + thread_log_out(tinfo); + } else { + va_start(pvar, fmt); + vprintf(fmt, pvar); + va_end(pvar); + } - va_start(pvar, fmt); - vprintf(fmt, pvar); - va_end(pvar); if (ctx->logf) { va_start(pvar, fmt); vfprintf(ctx->logf, fmt, pvar); diff --git a/tests/f_multithread/expect.1 b/tests/f_multithread/expect.1 index e2b954d0..8d2acd2b 100644 --- a/tests/f_multithread/expect.1 +++ b/tests/f_multithread/expect.1 @@ -1,6 +1,8 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... -Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scan group range [0, 2) +[Thread 0] Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts diff --git a/tests/f_multithread_logfile/expect.1 b/tests/f_multithread_logfile/expect.1 index e2b954d0..8d2acd2b 100644 --- a/tests/f_multithread_logfile/expect.1 +++ b/tests/f_multithread_logfile/expect.1 @@ -1,6 +1,8 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... -Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scan group range [0, 2) +[Thread 0] Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts diff --git a/tests/f_multithread_no/expect.1 b/tests/f_multithread_no/expect.1 index d14c4083..f85a3382 100644 --- a/tests/f_multithread_no/expect.1 +++ b/tests/f_multithread_no/expect.1 @@ -1,6 +1,8 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... -Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scan group range [0, 2) +[Thread 0] Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts From patchwork Wed Apr 8 10:44:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267876 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=SN86pZS/; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cj6157z9sQx for ; Wed, 8 Apr 2020 20:46:09 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728249AbgDHKqJ (ORCPT ); Wed, 8 Apr 2020 06:46:09 -0400 Received: from mail-pg1-f196.google.com ([209.85.215.196]:34570 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728229AbgDHKqJ (ORCPT ); Wed, 8 Apr 2020 06:46:09 -0400 Received: by mail-pg1-f196.google.com with SMTP id l14so3166050pgb.1 for ; Wed, 08 Apr 2020 03:46:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/M5iITtDPr9Fx1MobQuL0MQj7lmTDUjEonV4FCeaj9I=; b=SN86pZS/hOJg2tOtNlFWRDYkmV6iMmiIRKu7bjZugsD5qSAwsuZe1r7iE5ntYjcsz0 Be8SEBeTet52k5kpYRXTnf1bHs6mQuTDtXWIrxWLrdLt2M1xaR2wc94WuF/MeRId99hn z7wglfgItIhjyNIllzjxSvhG3vNQEVh9yKd6eA8ku5l82mFU4lMxKmgjbbjaGzD5cMNc xJyR7g6nBy4QIPxg9vrTxl3eJ4QGhC4yB7QIoes1O73G3BTrYMV9fbvnqiFkwX2bOAbF w462kDg6waaymDfpH8EJyuPC9/+/anuX7sGpo42a78hGs4TxobdfNzMQsuAR9S4f8VXj rXMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/M5iITtDPr9Fx1MobQuL0MQj7lmTDUjEonV4FCeaj9I=; b=JoKTN/tWTXgiy/Up+XA/Xrzu8O4rNZUHpRc1X7bQkGF36mm2piP5hn6mmq/zcFF+jZ wsmoAPZMZeEMmj28FDwOM92wUqAZCqEdb+Tg6D2kdwzw0KE6YjaW7cE9u/iYrnHFjIC4 TRBWHda1s9eRINLTkVGtzCUpp+aZAX8aLIfcQL0nt65hpYjbRsdY1PGZni/ex1gPMy+N TqRjs4Jwa2Y9HVnV21baj9mhTxVTtg8ZHiu5TpxDSyKagCF4AkHT19MF6+EUUY/eds4R tpIvdaSF1Hs2cldWFvDbTp+d33VfTcLsX7yptn0uiuqYfdODgF6RHUC+jasRqE+osPDx CD+w== X-Gm-Message-State: AGi0Pub7EY56DVgiWxC66JZfT8tIa487xNw7Ez0dGGAZUoZhBfu/r8Br BOOArciwAkxXCHxaUW3aBEWYA1oLgGM= X-Google-Smtp-Source: APiQypIqZHA7yfyh+ToVVPze1gHRHTaaTHlx1Aid9yG9QnLSxXPI6ohHAnyRBlgjXIwg6DZVW/u+1w== X-Received: by 2002:a63:a502:: with SMTP id n2mr6110749pgf.449.1586342767363; Wed, 08 Apr 2020 03:46:07 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:06 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 16/46] e2fsck: merge bitmaps after thread completes Date: Wed, 8 Apr 2020 19:44:44 +0900 Message-Id: <1586342714-12536-17-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi A new method merge_bmap has been added to bitmap operations. But only red-black bitmap has that operation now. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 149 +++++++++++++++++++++++++++++--------- lib/ext2fs/bitmaps.c | 7 ++ lib/ext2fs/blkmap64_rb.c | 25 +++++++ lib/ext2fs/bmap64.h | 2 + lib/ext2fs/ext2fs.h | 4 + lib/ext2fs/gen_bitmap64.c | 21 ++++++ 6 files changed, 173 insertions(+), 35 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 7a66bdf9..2c2973c7 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2123,31 +2123,57 @@ endit: ctx->invalid_bitmaps++; } -#define PASS1_COPY_FS_BITMAP(_dest, _src, _map_filed) \ +#define PASS1_COPY_FS_BITMAP(_dest, _src, _map_filed) \ +do { \ + errcode_t _ret; \ + if (_src->_map_filed) { \ + _ret = ext2fs_copy_bitmap(_src->_map_filed, \ + &_dest->_map_filed); \ + if (_ret) \ + return _ret; \ + _dest->_map_filed->fs = _dest; \ + } \ +} while (0) + +#define PASS1_MERGE_FS_BITMAP(_dest, _src, _map_field) \ do { \ - errcode_t _ret; \ - if (_src->_map_filed) { \ - _ret = ext2fs_copy_bitmap(_src->_map_filed, &_dest->_map_filed);\ - if (_ret) \ - return _ret; \ - _dest->_map_filed->fs = _dest; \ - \ - ext2fs_free_generic_bmap(_src->_map_filed); \ - _src->_map_filed = NULL; \ - } \ + errcode_t _ret = 0; \ + if (_src->_map_field) { \ + if (_dest->_map_field == NULL) { \ + _dest->_map_field = _src->_map_field; \ + _src->_map_field = NULL; \ + } else { \ + _ret = ext2fs_merge_bitmap(_src->_map_field, \ + _dest->_map_field); \ + if (_ret) \ + return _ret; \ + } \ + _dest->_map_field->fs = _dest; \ + } \ } while (0) -#define PASS1_COPY_CTX_BITMAP(_dest, _src, _map_filed) \ +#define PASS1_MERGE_CTX_BITMAP(_dest, _src, _map_field) \ do { \ - errcode_t _ret; \ - if (_src->_map_filed) { \ - _ret = ext2fs_copy_bitmap(_src->_map_filed, &_dest->_map_filed);\ - if (_ret) \ - return _ret; \ - _dest->_map_filed->fs = _dest->fs; \ - \ - ext2fs_free_generic_bmap(_src->_map_filed); \ - _src->_map_filed = NULL; \ + errcode_t _ret = 0; \ + if (_src->_map_field) { \ + if (_dest->_map_field == NULL) { \ + _dest->_map_field = _src->_map_field; \ + _src->_map_field = NULL; \ + } else { \ + _ret = ext2fs_merge_bitmap(_src->_map_field, \ + _dest->_map_field); \ + if (_ret) \ + return _ret; \ + } \ + _dest->_map_field->fs = _dest->fs; \ + } \ +} while (0) + +#define PASS1_FREE_CTX_BITMAP(_src, _map_field) \ +do { \ + if (_src->_map_field) { \ + ext2fs_free_generic_bmap(_src->_map_field); \ + _src->_map_field = NULL; \ } \ } while (0) @@ -2186,6 +2212,9 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, e2fsck_t src_context, errcode_t retval; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); + dest->inode_map = NULL; + dest->block_map = NULL; + /* * PASS1_COPY_FS_BITMAP might return directly from this function, * so please do NOT leave any garbage behind after returning. @@ -2259,18 +2288,24 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) errcode_t retval = 0; io_channel dest_io; io_channel dest_image_io; + ext2fs_inode_bitmap inode_map; + ext2fs_block_bitmap block_map; dest_io = dest->io; dest_image_io = dest->image_io; + inode_map = dest->inode_map; + block_map = dest->block_map; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->io = dest_io; dest->image_io = dest_image_io; + dest->inode_map = inode_map; + dest->block_map = block_map; /* - * PASS1_COPY_FS_BITMAP might return directly from this function, + * PASS1_MERGE_FS_BITMAP might return directly from this function, * so please do NOT leave any garbage behind after returning. */ - PASS1_COPY_FS_BITMAP(dest, src, inode_map); - PASS1_COPY_FS_BITMAP(dest, src, block_map); + PASS1_MERGE_FS_BITMAP(dest, src, inode_map); + PASS1_MERGE_FS_BITMAP(dest, src, block_map); if (src->dblist) { retval = ext2fs_copy_dblist(src->dblist, &dest->dblist); @@ -2299,6 +2334,11 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) retval = _e2fsck_pass1_merge_fs(dest, src); + if (src->inode_map) + ext2fs_free_generic_bmap(src->inode_map); + if (src->block_map) + ext2fs_free_generic_bmap(src->block_map); + /* icache will be rebuilt if needed, so do not copy from @src */ if (src->icache) { ext2fs_free_inode_cache(src->icache); @@ -2406,6 +2446,17 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_filsys global_fs = global_ctx->fs; FILE *global_logf = global_ctx->logf; FILE *global_problem_logf = global_ctx->problem_logf; + ext2fs_inode_bitmap inode_used_map = global_ctx->inode_used_map; + ext2fs_inode_bitmap inode_dir_map = global_ctx->inode_dir_map; + ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; + ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; + ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; + ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; + ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; + ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; + ext2fs_block_bitmap block_metadata_map = global_ctx->block_metadata_map; + ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; + #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2415,6 +2466,18 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx #ifdef HAVE_SETJMP_H memcpy(global_ctx->abort_loc, old_jmp, sizeof(jmp_buf)); #endif + + global_ctx->inode_used_map = inode_used_map; + global_ctx->inode_dir_map = inode_dir_map; + global_ctx->inode_bb_map = inode_bb_map; + global_ctx->inode_imagic_map = inode_imagic_map; + global_ctx->inodes_to_rebuild = inodes_to_rebuild; + global_ctx->inode_reg_map = inode_reg_map; + global_ctx->block_found_map = block_found_map; + global_ctx->block_dup_map = block_dup_map; + global_ctx->block_ea_map = block_ea_map; + global_ctx->block_metadata_map = block_metadata_map; + /* Keep the global singal flags*/ global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); @@ -2434,16 +2497,16 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx * PASS1_COPY_CTX_BITMAP might return directly from this function, * so please do NOT leave any garbage behind after returning. */ - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_used_map); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_dir_map); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_bb_map); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_imagic_map); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inode_reg_map); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, inodes_to_rebuild); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_found_map); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_dup_map); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_ea_map); - PASS1_COPY_CTX_BITMAP(global_ctx, thread_ctx, block_metadata_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_used_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_dir_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_bb_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_imagic_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_reg_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inodes_to_rebuild); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_found_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_dup_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_ea_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_metadata_map); return 0; } @@ -2460,6 +2523,16 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) fputs("\n", thread_ctx->problem_logf); fclose(thread_ctx->problem_logf); } + PASS1_FREE_CTX_BITMAP(thread_ctx, inode_used_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, inode_dir_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, inode_bb_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, inode_imagic_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, inode_reg_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, inodes_to_rebuild); + PASS1_FREE_CTX_BITMAP(thread_ctx, block_found_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, block_dup_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, block_ea_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, block_metadata_map); ext2fs_free_mem(&thread_ctx); return retval; @@ -2486,7 +2559,13 @@ static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, if (ret == 0) ret = rc; } - e2fsck_pass1_thread_join(global_ctx, infos[i].eti_thread_ctx); + rc = e2fsck_pass1_thread_join(global_ctx, infos[i].eti_thread_ctx); + if (rc) { + com_err(global_ctx->program_name, rc, + _("while joining pass1 thread\n")); + if (ret == 0) + ret = rc; + } } free(infos); diff --git a/lib/ext2fs/bitmaps.c b/lib/ext2fs/bitmaps.c index e25db2c6..baa7c627 100644 --- a/lib/ext2fs/bitmaps.c +++ b/lib/ext2fs/bitmaps.c @@ -45,6 +45,13 @@ errcode_t ext2fs_copy_bitmap(ext2fs_generic_bitmap src, { return (ext2fs_copy_generic_bmap(src, dest)); } + +errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src, + ext2fs_generic_bitmap dest) +{ + return ext2fs_merge_generic_bmap(src, dest); +} + void ext2fs_set_bitmap_padding(ext2fs_generic_bitmap map) { ext2fs_set_generic_bmap_padding(map); diff --git a/lib/ext2fs/blkmap64_rb.c b/lib/ext2fs/blkmap64_rb.c index 1fd55274..42a10536 100644 --- a/lib/ext2fs/blkmap64_rb.c +++ b/lib/ext2fs/blkmap64_rb.c @@ -968,11 +968,36 @@ static void rb_print_stats(ext2fs_generic_bitmap_64 bitmap EXT2FS_ATTR((unused)) } #endif +static errcode_t rb_merge_bmap(ext2fs_generic_bitmap_64 src, + ext2fs_generic_bitmap_64 dest) +{ + struct ext2fs_rb_private *src_bp, *dest_bp; + struct bmap_rb_extent *src_ext; + struct rb_node *src_node; + errcode_t retval = 0; + + src_bp = (struct ext2fs_rb_private *) src->private; + dest_bp = (struct ext2fs_rb_private *) dest->private; + src_bp->rcursor = NULL; + dest_bp->rcursor = NULL; + + src_node = ext2fs_rb_first(&src_bp->root); + while (src_node) { + src_ext = node_to_extent(src_node); + rb_insert_extent(src_ext->start, src_ext->count, dest_bp); + + src_node = ext2fs_rb_next(src_node); + } + + return retval; +} + struct ext2_bitmap_ops ext2fs_blkmap64_rbtree = { .type = EXT2FS_BMAP64_RBTREE, .new_bmap = rb_new_bmap, .free_bmap = rb_free_bmap, .copy_bmap = rb_copy_bmap, + .merge_bmap = rb_merge_bmap, .resize_bmap = rb_resize_bmap, .mark_bmap = rb_mark_bmap, .unmark_bmap = rb_unmark_bmap, diff --git a/lib/ext2fs/bmap64.h b/lib/ext2fs/bmap64.h index de334548..09a5886b 100644 --- a/lib/ext2fs/bmap64.h +++ b/lib/ext2fs/bmap64.h @@ -72,6 +72,8 @@ struct ext2_bitmap_ops { void (*free_bmap)(ext2fs_generic_bitmap_64 bitmap); errcode_t (*copy_bmap)(ext2fs_generic_bitmap_64 src, ext2fs_generic_bitmap_64 dest); + errcode_t (*merge_bmap)(ext2fs_generic_bitmap_64 src, + ext2fs_generic_bitmap_64 dest); errcode_t (*resize_bmap)(ext2fs_generic_bitmap_64 bitmap, __u64 new_end, __u64 new_real_end); diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 93ecf29c..2cc6d76e 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -837,6 +837,8 @@ extern void ext2fs_free_block_bitmap(ext2fs_block_bitmap bitmap); extern void ext2fs_free_inode_bitmap(ext2fs_inode_bitmap bitmap); extern errcode_t ext2fs_copy_bitmap(ext2fs_generic_bitmap src, ext2fs_generic_bitmap *dest); +errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src, + ext2fs_generic_bitmap dest); extern errcode_t ext2fs_write_inode_bitmap(ext2_filsys fs); extern errcode_t ext2fs_write_block_bitmap (ext2_filsys fs); extern errcode_t ext2fs_read_inode_bitmap (ext2_filsys fs); @@ -1433,6 +1435,8 @@ void ext2fs_set_generic_bmap_padding(ext2fs_generic_bitmap bmap); errcode_t ext2fs_resize_generic_bmap(ext2fs_generic_bitmap bmap, __u64 new_end, __u64 new_real_end); +errcode_t ext2fs_merge_generic_bmap(ext2fs_generic_bitmap gen_src, + ext2fs_generic_bitmap gen_dest); errcode_t ext2fs_compare_generic_bmap(errcode_t neq, ext2fs_generic_bitmap bm1, ext2fs_generic_bitmap bm2); diff --git a/lib/ext2fs/gen_bitmap64.c b/lib/ext2fs/gen_bitmap64.c index 6e4d8b71..ccba4427 100644 --- a/lib/ext2fs/gen_bitmap64.c +++ b/lib/ext2fs/gen_bitmap64.c @@ -344,6 +344,27 @@ errcode_t ext2fs_copy_generic_bmap(ext2fs_generic_bitmap gen_src, return 0; } +errcode_t ext2fs_merge_generic_bmap(ext2fs_generic_bitmap gen_src, + ext2fs_generic_bitmap gen_dest) +{ + ext2fs_generic_bitmap_64 src = (ext2fs_generic_bitmap_64) gen_src; + ext2fs_generic_bitmap_64 dest = (ext2fs_generic_bitmap_64) gen_dest; + + if (!src || !dest) + return EINVAL; + + if (!EXT2FS_IS_64_BITMAP(src) || !EXT2FS_IS_64_BITMAP(dest)) + return EINVAL; + + if (src->bitmap_ops != dest->bitmap_ops) + return EINVAL; + + if (src->bitmap_ops->merge_bmap == NULL) + return EOPNOTSUPP; + + return src->bitmap_ops->merge_bmap(src, dest); +} + errcode_t ext2fs_resize_generic_bmap(ext2fs_generic_bitmap gen_bmap, __u64 new_end, __u64 new_real_end) From patchwork Wed Apr 8 10:44:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267877 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=BBI6mKQo; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cn2GCBz9sQx for ; Wed, 8 Apr 2020 20:46:13 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728253AbgDHKqN (ORCPT ); Wed, 8 Apr 2020 06:46:13 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:37896 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728229AbgDHKqM (ORCPT ); Wed, 8 Apr 2020 06:46:12 -0400 Received: by mail-pf1-f196.google.com with SMTP id c21so2232072pfo.5 for ; Wed, 08 Apr 2020 03:46:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xSQFkvZhHJAaBHZvRdjI9Ehr5XIo2pfMcBKvptY0qUc=; b=BBI6mKQoLve/9aDHy7IG8sYIzGDehzaa67yB3YCRYcwdWBj9dEDFvE/Nzu8tNNgczj 5EVEK+Tjtb7rTKd6+sgSyX/ydju2tqCgbXDJCo1iqxZ0F4nVChUyd4ANSAEH4LuI7Czw qtvbJ6v7wXqc/H68ttvNwiUh/Mzji5449qWQs48OsHFy8H4CQtZVXpu6RSMI+AI0NT06 sLsQN1bpPGpjgw6XI7tPQgNlEzruvmLG7BcRWXjCVI00+IRQ5RWohYm17GZQozVTr+Go jPwOdsR1ARlDejtV4vRd6dYailm3BKp4Ya7hbfZcGdLmyjAdlWDmsSQFk8qQVDpqbRys 9Xww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xSQFkvZhHJAaBHZvRdjI9Ehr5XIo2pfMcBKvptY0qUc=; b=oEaNv+mIZ7A+Jd2GdC19Q9Gx+br/E87uUEeUrlTk5KjpmsK15Kadxm+2lMeFWmKxlP zqkaOMXWkEQALg7GOc1/4mDxEfb10xl4H9Unoul/4N+CbPBjs1CzVHwKAVXQm5lQMipZ h5byUge+Hcws7qFmHBBIQhn9gVg2pXFdvNQVsozce5ZvqpdKviQgMPV/ujDVnBJ7gqvl alI0H+Om89th49673KJSps6FsSAGvGg+LWAcgBBL6RLM07f0ouukxL+vQEmDS07IHQIv m+NiFNh8dRji7zZlzEqLiV+4pbTFPadYixWqeRjDmYvOWjlpGKchUgox0g39zacXs1fX 8O6A== X-Gm-Message-State: AGi0Pua6vJ8uIoAXATJhp0Gc+HsPY8W0DK1FZ7v61eKqR3GLCU/tv1jc ye6DK6Yr7mO/kBMY/SuANfOGdJcTt4k= X-Google-Smtp-Source: APiQypJLoaw3aeuxhMflqLI5bkPrIAj1uIIe7YAxLMhaInaufrU+tjGWIkOpnZvLRLTgUhy+lUBaAg== X-Received: by 2002:aa7:9f47:: with SMTP id h7mr7151656pfr.292.1586342769476; Wed, 08 Apr 2020 03:46:09 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:08 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 17/46] e2fsck: do not change global variables Date: Wed, 8 Apr 2020 19:44:45 +0900 Message-Id: <1586342714-12536-18-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Global variables used in pass1 check are changed to local variables in this patch. This will avoid conflict between threads. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 81 +++++++++++++++++++++++++++++++------------------- 1 file changed, 50 insertions(+), 31 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 2c2973c7..85d18c55 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -84,7 +84,6 @@ static void alloc_bb_map(e2fsck_t ctx); static void alloc_imagic_map(e2fsck_t ctx); static void mark_inode_bad(e2fsck_t ctx, ino_t ino); static void handle_fs_bad_blocks(e2fsck_t ctx); -static void process_inodes(e2fsck_t ctx, char *block_buf); static EXT2_QSORT_TYPE process_inode_cmp(const void *a, const void *b); static errcode_t scan_callback(ext2_filsys fs, ext2_inode_scan scan, dgrp_t group, void * priv_data); @@ -119,15 +118,15 @@ struct process_inode_block { }; struct scan_callback_struct { - e2fsck_t ctx; - char *block_buf; + e2fsck_t ctx; + char *block_buf; + struct process_inode_block *inodes_to_process; + int *process_inode_count; }; -/* - * For the inodes to process list. - */ -static struct process_inode_block *inodes_to_process; -static int process_inode_count; +static void process_inodes(e2fsck_t ctx, char *block_buf, + struct process_inode_block *inodes_to_process, + int *process_inode_count); static __u64 ext2_max_sizes[EXT2_MAX_BLOCK_LOG_SIZE - EXT2_MIN_BLOCK_LOG_SIZE + 1]; @@ -136,10 +135,10 @@ static __u64 ext2_max_sizes[EXT2_MAX_BLOCK_LOG_SIZE - * Free all memory allocated by pass1 in preparation for restarting * things. */ -static void unwind_pass1(ext2_filsys fs EXT2FS_ATTR((unused))) +static void unwind_pass1(struct process_inode_block *inodes_to_process) { ext2fs_free_mem(&inodes_to_process); - inodes_to_process = 0; + inodes_to_process = NULL; } /* @@ -1176,7 +1175,6 @@ static int e2fsck_should_abort(e2fsck_t ctx) void _e2fsck_pass1(e2fsck_t ctx) { int i; - __u64 max_sizes; ext2_filsys fs = ctx->fs; ext2_ino_t ino = 0; struct ext2_inode *inode = NULL; @@ -1199,6 +1197,8 @@ void _e2fsck_pass1(e2fsck_t ctx) ext2_ino_t ino_threshold = 0; dgrp_t ra_group = 0; struct ea_quota ea_ibody_quota; + struct process_inode_block *inodes_to_process; + int process_inode_count; init_resource_track(&rtrack, ctx->fs->io); clear_problem_context(&pctx); @@ -1223,17 +1223,6 @@ void _e2fsck_pass1(e2fsck_t ctx) mtrace_print("Pass 1"); #endif -#define EXT2_BPP(bits) (1ULL << ((bits) - 2)) - - for (i = EXT2_MIN_BLOCK_LOG_SIZE; i <= EXT2_MAX_BLOCK_LOG_SIZE; i++) { - max_sizes = EXT2_NDIR_BLOCKS + EXT2_BPP(i); - max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i); - max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i) * EXT2_BPP(i); - max_sizes = (max_sizes * (1UL << i)); - ext2_max_sizes[i - EXT2_MIN_BLOCK_LOG_SIZE] = max_sizes; - } -#undef EXT2_BPP - imagic_fs = ext2fs_has_feature_imagic_inodes(sb); extent_fs = ext2fs_has_feature_extents(sb); inlinedata_fs = ext2fs_has_feature_inline_data(sb); @@ -1357,6 +1346,8 @@ void _e2fsck_pass1(e2fsck_t ctx) ctx->stashed_inode = inode; scan_struct.ctx = ctx; scan_struct.block_buf = block_buf; + scan_struct.inodes_to_process = inodes_to_process; + scan_struct.process_inode_count = &process_inode_count; ext2fs_set_inode_callback(scan, scan_callback, &scan_struct); if (ctx->progress && ((ctx->progress)(ctx, 1, 0, ctx->fs->group_desc_count))) @@ -2007,13 +1998,15 @@ void _e2fsck_pass1(e2fsck_t ctx) goto endit; if (process_inode_count >= ctx->process_inode_size) { - process_inodes(ctx, block_buf); + process_inodes(ctx, block_buf, inodes_to_process, + &process_inode_count); if (e2fsck_should_abort(ctx)) goto endit; } } - process_inodes(ctx, block_buf); + process_inodes(ctx, block_buf, inodes_to_process, + &process_inode_count); ext2fs_close_inode_scan(scan); scan = NULL; @@ -2088,7 +2081,7 @@ void _e2fsck_pass1(e2fsck_t ctx) * master superblock. */ ctx->use_superblock = 0; - unwind_pass1(fs); + unwind_pass1(inodes_to_process); goto endit; } @@ -2671,12 +2664,34 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, return 0; } +static void init_ext2_max_sizes() +{ + int i; + __u64 max_sizes; + + /* + * Init ext2_max_sizes which will be immutable and shared between + * threads + */ +#define EXT2_BPP(bits) (1ULL << ((bits) - 2)) + + for (i = EXT2_MIN_BLOCK_LOG_SIZE; i <= EXT2_MAX_BLOCK_LOG_SIZE; i++) { + max_sizes = EXT2_NDIR_BLOCKS + EXT2_BPP(i); + max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i); + max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i) * EXT2_BPP(i); + max_sizes = (max_sizes * (1UL << i)); + ext2_max_sizes[i - EXT2_MIN_BLOCK_LOG_SIZE] = max_sizes; + } +#undef EXT2_BPP +} + static void e2fsck_pass1_multithread(e2fsck_t global_ctx) { struct e2fsck_thread_info *infos = NULL; int num_threads = 1; errcode_t retval; + init_ext2_max_sizes(); retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); if (retval) { com_err(global_ctx->program_name, retval, @@ -2718,7 +2733,9 @@ static errcode_t scan_callback(ext2_filsys fs, scan_struct = (struct scan_callback_struct *) priv_data; ctx = scan_struct->ctx; - process_inodes((e2fsck_t) fs->priv_data, scan_struct->block_buf); + process_inodes((e2fsck_t) fs->priv_data, scan_struct->block_buf, + scan_struct->inodes_to_process, + scan_struct->process_inode_count); if (ctx->progress) if ((ctx->progress)(ctx, 1, group+1, @@ -2739,7 +2756,9 @@ static errcode_t scan_callback(ext2_filsys fs, /* * Process the inodes in the "inodes to process" list. */ -static void process_inodes(e2fsck_t ctx, char *block_buf) +static void process_inodes(e2fsck_t ctx, char *block_buf, + struct process_inode_block *inodes_to_process, + int *process_inode_count) { int i; struct ext2_inode *old_stashed_inode; @@ -2751,15 +2770,15 @@ static void process_inodes(e2fsck_t ctx, char *block_buf) #if 0 printf("begin process_inodes: "); #endif - if (process_inode_count == 0) + if (*process_inode_count == 0) return; old_operation = ehandler_operation(0); old_stashed_inode = ctx->stashed_inode; old_stashed_ino = ctx->stashed_ino; - qsort(inodes_to_process, process_inode_count, + qsort(inodes_to_process, *process_inode_count, sizeof(struct process_inode_block), process_inode_cmp); clear_problem_context(&pctx); - for (i=0; i < process_inode_count; i++) { + for (i=0; i < *process_inode_count; i++) { pctx.inode = ctx->stashed_inode = (struct ext2_inode *) &inodes_to_process[i].inode; pctx.ino = ctx->stashed_ino = inodes_to_process[i].ino; @@ -2777,7 +2796,7 @@ static void process_inodes(e2fsck_t ctx, char *block_buf) } ctx->stashed_inode = old_stashed_inode; ctx->stashed_ino = old_stashed_ino; - process_inode_count = 0; + *process_inode_count = 0; #if 0 printf("end process inodes\n"); #endif From patchwork Wed Apr 8 10:44:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267878 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ar0+YFrz; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cp1sKcz9sSg for ; Wed, 8 Apr 2020 20:46:14 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728254AbgDHKqN (ORCPT ); Wed, 8 Apr 2020 06:46:13 -0400 Received: from mail-pf1-f195.google.com ([209.85.210.195]:34350 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqN (ORCPT ); Wed, 8 Apr 2020 06:46:13 -0400 Received: by mail-pf1-f195.google.com with SMTP id v23so2880508pfm.1 for ; Wed, 08 Apr 2020 03:46:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ws6JihLZVQXUPNY9KIQBSbq4h/d2hpaWaq4K4VnEKlk=; b=ar0+YFrzoFrfQF7DPnQaqon/YhNJgQmNLrt5SOHhIWQTqH2UgjZsIQPMPAUTrYncLK chg9hiJy4MLDnBoZMceXhZtbZYTpGQqOk1OgjLXF85bT8yN4kVVW9gud/r2CRhdhSyOJ X5EEZ+fIZISKVWwEQ0IioXyFLQLGxrr0XubBZclPF42xW9XkrbQ+Tj0CYUItsygT9nDR 5E58Opj2cImTV+zYUiEvElUyK5dYAy+7UxoEi6zFAt2LfG+q1w+EEWSRrJCJ1E6CUE8G MlYD4Y7ChIlSGyR000vD+skV+a2jbJkxrUigulnbDqVF4IuewsBMM6TQfCmIG7Ybq+BS o/mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ws6JihLZVQXUPNY9KIQBSbq4h/d2hpaWaq4K4VnEKlk=; b=iujREvDiU6qmctAU98CTwaGWzbxlvGhzV2RlTJHZVofUP6wWpgbtA3HeY9pO8OwzP+ 1S7MUzeSz6vkLvvWfruJI/BG75furhkJuGytfIl+H6jf2LmpXtc4usiNiIpMCWRC7Chm Hexy48C6FHStJZwztMd7fHrPOJWk6ryujvPQ970ok5D78dlFZY9gkivhcDuAVeYLaEPp uL8tM5enQFry11Rhbt5kAnXNDv3d1FIQxLChTHrDHUnVZzJSS9shtWXrBGqhoVGdyb0m x+WA8pR7wbbyZSmwjMAkennMoibw1frXRgSDTm3SUn35VPHWyQeBIIKYcruirxCobm7F QwjA== X-Gm-Message-State: AGi0PubBhjOroYsMT/oDeGrcw2HPz1mnhX/j7D9uVYg7FqKFUEY4Yiuo UXYAUp0xPE3P+Pf7DHQfVYpbcXgQ6Wk= X-Google-Smtp-Source: APiQypIJBQT095vw4OQj7vyOXsWX7uFNYBdLh7FddyTqJLTDmOdgUldvkPg6EFTZlHG2fIC2xfHqzg== X-Received: by 2002:aa7:984e:: with SMTP id n14mr7216167pfq.291.1586342771595; Wed, 08 Apr 2020 03:46:11 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:11 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 18/46] e2fsck: optimize the inserting of dir_info_db Date: Wed, 8 Apr 2020 19:44:46 +0900 Message-Id: <1586342714-12536-19-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Binary search is now used when inserting an dir info to the array. Memmove is now used when moving array. Both of them improves the performance of inserting. This patch is also a prepartion for the merging of two dir db arrays. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/dirinfo.c | 154 +++++++++++++++++++++++++++++------------------ 1 file changed, 96 insertions(+), 58 deletions(-) diff --git a/e2fsck/dirinfo.c b/e2fsck/dirinfo.c index cceadac3..c5183261 100644 --- a/e2fsck/dirinfo.c +++ b/e2fsck/dirinfo.c @@ -7,6 +7,7 @@ #undef DIRINFO_DEBUG +#include #include "config.h" #include "e2fsck.h" #include @@ -122,6 +123,89 @@ static void setup_db(e2fsck_t ctx) "directory map"); } +/* + * Return the min index that has ino larger or equal to @ino + * If not found, return -1 + */ +static int e2fsck_dir_info_min_larger_equal(struct dir_info_db *dir_info, + ext2_ino_t ino) +{ + int low = 0; + int high = dir_info->count - 1; + int mid; + int tmp_ino; + int index = -1; + + while (low <= high) { + mid = (low + high) / 2; + tmp_ino = dir_info->array[mid].ino; + if (ino == tmp_ino) { + return mid; + } else if (ino < tmp_ino) { + /* + * The mid ino is larger than @ino, remember the index + * here so we won't miss this ino + */ + index = mid; + high = mid - 1; + } else { + low = mid + 1; + } + } + return index; +} + +/* + * + * Insert an inode into the sorted array. The array should have at least one + * free slot. + * + * Normally, add_dir_info is called with each inode in + * sequential order; but once in a while (like when pass 3 + * needs to recreate the root directory or lost+found + * directory) it is called out of order. In those cases, we + * need to move the dir_info entries down to make room, since + * the dir_info array needs to be sorted by inode number for + * get_dir_info()'s sake. + */ +static void e2fsck_insert_dir_info(struct dir_info_db *dir_info, ext2_ino_t ino, ext2_ino_t parent) +{ + int index; + struct dir_info *dir; + int dir_size = sizeof(*dir); + struct dir_info *array = dir_info->array; + int array_count = dir_info->count; + + /* + * Removing this check won't break anything. But since seqential ino + * inserting happens a lot, this check avoids binary search. + */ + if (array_count == 0 || array[array_count - 1].ino < ino) { + dir = &array[array_count]; + dir_info->count++; + goto out; + } + + index = e2fsck_dir_info_min_larger_equal(dir_info, ino); + if (index >= 0 && array[index].ino == ino) { + dir = &array[index]; + goto out; + } + if (index < 0) { + dir = &array[array_count]; + dir_info->count++; + goto out; + } + + dir = &array[index]; + memmove((char *)dir + dir_size, dir, dir_size * (array_count - index)); + dir_info->count++; +out: + dir->ino = ino; + dir->dotdot = parent; + dir->parent = parent; +} + /* * This subroutine is called during pass1 to create a directory info * entry. During pass1, the passed-in parent is 0; it will get filled @@ -171,30 +255,7 @@ void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent) } #endif - /* - * Normally, add_dir_info is called with each inode in - * sequential order; but once in a while (like when pass 3 - * needs to recreate the root directory or lost+found - * directory) it is called out of order. In those cases, we - * need to move the dir_info entries down to make room, since - * the dir_info array needs to be sorted by inode number for - * get_dir_info()'s sake. - */ - if (ctx->dir_info->count && - ctx->dir_info->array[ctx->dir_info->count-1].ino >= ino) { - for (i = ctx->dir_info->count-1; i > 0; i--) - if (ctx->dir_info->array[i-1].ino < ino) - break; - dir = &ctx->dir_info->array[i]; - if (dir->ino != ino) - for (j = ctx->dir_info->count++; j > i; j--) - ctx->dir_info->array[j] = ctx->dir_info->array[j-1]; - } else - dir = &ctx->dir_info->array[ctx->dir_info->count++]; - - dir->ino = ino; - dir->dotdot = parent; - dir->parent = parent; + e2fsck_insert_dir_info(ctx->dir_info, ino, parent); } /* @@ -204,7 +265,7 @@ void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent) static struct dir_info *e2fsck_get_dir_info(e2fsck_t ctx, ext2_ino_t ino) { struct dir_info_db *db = ctx->dir_info; - int low, high, mid; + int index; if (!db) return 0; @@ -245,43 +306,20 @@ static struct dir_info *e2fsck_get_dir_info(e2fsck_t ctx, ext2_ino_t ino) if (db->last_lookup && db->last_lookup->ino == ino) return db->last_lookup; - low = 0; - high = ctx->dir_info->count-1; - if (ino == ctx->dir_info->array[low].ino) { -#ifdef DIRINFO_DEBUG - printf("(%d,%d,%d)\n", ino, - ctx->dir_info->array[low].dotdot, - ctx->dir_info->array[low].parent); -#endif - return &ctx->dir_info->array[low]; - } - if (ino == ctx->dir_info->array[high].ino) { -#ifdef DIRINFO_DEBUG - printf("(%d,%d,%d)\n", ino, - ctx->dir_info->array[high].dotdot, - ctx->dir_info->array[high].parent); -#endif - return &ctx->dir_info->array[high]; - } + index = e2fsck_dir_info_min_larger_equal(ctx->dir_info, ino); + if (index < 0) + return NULL; - while (low < high) { - mid = (low+high)/2; - if (mid == low || mid == high) - break; - if (ino == ctx->dir_info->array[mid].ino) { + assert(ino <= ctx->dir_info->array[index].ino); + if (ino == ctx->dir_info->array[index].ino) { #ifdef DIRINFO_DEBUG - printf("(%d,%d,%d)\n", ino, - ctx->dir_info->array[mid].dotdot, - ctx->dir_info->array[mid].parent); + printf("(%d,%d,%d)\n", ino, + ctx->dir_info->array[index].dotdot, + ctx->dir_info->array[index].parent); #endif - return &ctx->dir_info->array[mid]; - } - if (ino < ctx->dir_info->array[mid].ino) - high = mid; - else - low = mid; + return &ctx->dir_info->array[index]; } - return 0; + return NULL; } static void e2fsck_put_dir_info(e2fsck_t ctx EXT2FS_NO_TDB_UNUSED, From patchwork Wed Apr 8 10:44:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267879 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=sPdbJH7h; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cq5d2bz9sQx for ; Wed, 8 Apr 2020 20:46:15 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728256AbgDHKqP (ORCPT ); Wed, 8 Apr 2020 06:46:15 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:37902 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqP (ORCPT ); Wed, 8 Apr 2020 06:46:15 -0400 Received: by mail-pf1-f193.google.com with SMTP id c21so2232142pfo.5 for ; Wed, 08 Apr 2020 03:46:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HxQQPPzJURVVYK6dlJ1sxPLmKo7D3PdesZjlth/3VHg=; b=sPdbJH7h6jxD8bX8jOzCeA8SM3g66JBiHEWXUdvG0JqYAXLfmAmASKjaG+uA9B2jEz /fIrZqCShVbMXFyRfq3DMTvS8Ucdlx75D0TsO1GxZGEAzYfSNlhOtub3abdVpJiaLDWI JlXahdIgl4K0fagjF6WUYdsHRXmGO7H/0WGwyr8qRBeyZRHMQvPm4nGNuAundJN+GLUU AEijR7qtg5LQIHKTGIxNNm3lQnY4GI2hQoDE3/0nBR6cdQRtXWrSTI4r0O7cf+LSvOmB yN3V/VnOi7hQqLo81BgzFOnLI08abjHkNnOSuw0vQURZoTASkfXNo9+i9AywrA6J19ap TSMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HxQQPPzJURVVYK6dlJ1sxPLmKo7D3PdesZjlth/3VHg=; b=gPFE9QiLWsVQmFRlm1CUfkAlcJsIzbtucA7s/PyhX8plhJL6pEGEIiF5sXQC4+LADK vNSiUZYeyZqHMWp6txKHc3+QNVZ9nh3hcEApO1DLURnK/JKzVnCnHN9G3Rax2Rmcv5CL 48hG5TKyGyZVOJ6RcXUhjMYFgvRoL6VdGvinU9uhKroicwE6XKtG/iGdGvp+kwN6fqf4 97vVxuPeghsHKWJmezrCWSP5J71yCcxJlnXajPwwEvV1/37p/BGgFl9WfQZ65EtMOLdN 3mJLwcMkOeVTnL/1LWzZqcJabQR2KEPcuauQIsjLa71duyHbAwMCj27mGRNVal1P92vP obLA== X-Gm-Message-State: AGi0Pua2Iuh6OfSBu+JiMTu/tsEh1p5NeCCXuehWWbkdLNgvmnF4EgBb 8byqBov/uu1gFJfWK59H2cCQiEM4jPc= X-Google-Smtp-Source: APiQypLgYppJS/EiybjTjZVfK0CmAVukRmWwRFJmkmeNUswDsYRjiKfPf+I8BSwuz5R0OSkCLL4RDg== X-Received: by 2002:aa7:9104:: with SMTP id 4mr7254170pfh.168.1586342773787; Wed, 08 Apr 2020 03:46:13 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:13 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 19/46] e2fsck: merge dir_info after thread finishes Date: Wed, 8 Apr 2020 19:44:47 +0900 Message-Id: <1586342714-12536-20-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/dirinfo.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++ e2fsck/e2fsck.h | 2 ++ e2fsck/pass1.c | 19 ++++++++++++++ 3 files changed, 87 insertions(+) diff --git a/e2fsck/dirinfo.c b/e2fsck/dirinfo.c index c5183261..fab10d89 100644 --- a/e2fsck/dirinfo.c +++ b/e2fsck/dirinfo.c @@ -155,6 +155,72 @@ static int e2fsck_dir_info_min_larger_equal(struct dir_info_db *dir_info, return index; } +/* + * Merge two sorted dir info to @dest + */ +void e2fsck_merge_dir_info(e2fsck_t ctx, struct dir_info_db *src, + struct dir_info_db *dest) +{ + int size_dir_info = sizeof(struct dir_info); + int size = dest->size; + struct dir_info *src_array = src->array; + struct dir_info *dest_array = dest->array; + int src_count = src->count; + int dest_count = dest->count; + int total_count = src_count + dest_count; + struct dir_info *array; + struct dir_info *array_ptr; + int src_index = 0; + int dest_index = 0; + + if (src->count == 0) + return; + + if (size < total_count) + size = total_count; + + if (size < src->size) + size = size; + + array = e2fsck_allocate_memory(ctx, size * size_dir_info, + "directory map"); + array_ptr = array; + /* + * This can be improved by binary search and memcpy, but codes + * would be complexer. And if the groups distributed to each + * thread are stided, this implementation won't be too bad comparing + * to the optimiztion. + */ + while (src_index < src_count || dest_index < dest_count) { + if (src_index >= src_count) { + memcpy(array_ptr, &dest_array[dest_index], + (dest_count - dest_index) * size_dir_info); + break; + } + if (dest_index >= dest_count) { + memcpy(array_ptr, &src_array[src_index], + (src_count - src_index) * size_dir_info); + break; + } + if (src_array[src_index].ino < dest_array[dest_index].ino) { + *array_ptr = src_array[src_index]; + src_index++; + } else { + assert(src_array[src_index].ino > + dest_array[dest_index].ino); + *array_ptr = dest_array[dest_index]; + dest_index++; + } + array_ptr++; + } + + if (dest->array) + ext2fs_free_mem(&dest->array); + dest->array = array; + dest->size = size; + dest->count = total_count; +} + /* * * Insert an inode into the sorted array. The array should have at least one diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 48afc8f3..f8e98f73 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -507,6 +507,8 @@ extern void read_bad_blocks_file(e2fsck_t ctx, const char *bad_blocks_file, /* dirinfo.c */ extern void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent); +void e2fsck_merge_dir_info(e2fsck_t ctx, struct dir_info_db *src, + struct dir_info_db *dest); extern void e2fsck_free_dir_info(e2fsck_t ctx); extern int e2fsck_get_num_dirinfo(e2fsck_t ctx); extern struct dir_info_iter *e2fsck_dir_info_iter_begin(e2fsck_t ctx); diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 85d18c55..56004c9b 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2431,6 +2431,22 @@ out_context: return retval; } +static void e2fsck_pass1_merge_dir_info(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + if (thread_ctx->dir_info == NULL) + return; + + if (global_ctx->dir_info == NULL) { + /* TODO: tdb needs to be handled properly */ + global_ctx->dir_info = thread_ctx->dir_info; + thread_ctx->dir_info = NULL; + return; + } + + e2fsck_merge_dir_info(global_ctx, thread_ctx->dir_info, + global_ctx->dir_info); +} + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2439,6 +2455,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_filsys global_fs = global_ctx->fs; FILE *global_logf = global_ctx->logf; FILE *global_problem_logf = global_ctx->problem_logf; + struct dir_info_db *dir_info = global_ctx->dir_info; ext2fs_inode_bitmap inode_used_map = global_ctx->inode_used_map; ext2fs_inode_bitmap inode_dir_map = global_ctx->inode_dir_map; ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; @@ -2470,6 +2487,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->block_dup_map = block_dup_map; global_ctx->block_ea_map = block_ea_map; global_ctx->block_metadata_map = block_metadata_map; + global_ctx->dir_info = dir_info; + e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); /* Keep the global singal flags*/ global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | From patchwork Wed Apr 8 10:44:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267880 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=sCS64mnD; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cs3xjkz9sQx for ; Wed, 8 Apr 2020 20:46:17 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728259AbgDHKqR (ORCPT ); Wed, 8 Apr 2020 06:46:17 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:36462 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqR (ORCPT ); Wed, 8 Apr 2020 06:46:17 -0400 Received: by mail-pl1-f195.google.com with SMTP id g2so2379563plo.3 for ; Wed, 08 Apr 2020 03:46:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZyNjlIWNHpE31ut2Vmi9YHAxUycDDQCE45FYLqLhvWI=; b=sCS64mnDEvuSFJjuUIBr1uxJ733903AnE32HVF3Ot2omgSwiqMidPAieYVjs0G9doL Ph5X9jc/1jR4Z6+pQsmO57SCwe9XYnTMEGinFJ9N6FEsVCRVn1Hzzv3E2/0M3ABOypGO 7s8taaMIVbuLfUMRMNnujIYrLOoFh+YlUuTn4P/HgmVvpyl5C/T+cYe8yF0gnmatibIr hXNe4NkWxgyy4Asc2A23OCgbzdyIO/FGbmngS/W8MfpADxWTYsgtYpbnHL96fW0axDPS G67C6xrd73kQxRmPSxxkRGlvq1rjBVPOfoc5fyk0UBQBFqdHj/WtOYS6ABcyD/vYN3V7 8FuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZyNjlIWNHpE31ut2Vmi9YHAxUycDDQCE45FYLqLhvWI=; b=C7zlL/AUSwT4TgRl1kzkxOeE8GYZZh2f9/H9ccgAPVtBRwcqAT7M+9sCJfiw0CH2q6 CTdRsI7bQSB3VpXpU/ylijGUcADQaoArDn0N7zE2fpjbB8vt09iteiLbMPkvaQtY535U bsRQa9Mx9JBNItNUW5Em+t+IKKraWaGf01vRqBhkIvON5HJ2kKjuNmCRLIkgPVcHTpsO iCFv4DDK/Ohg11fVVEYA69Pk2O0OnOdWgYvyIG/FVmQjZck4NOj1Mz++AVYC37wYpAoD pxzzeQ8yZGRBdOcY14XhbEEuqP8e+RVAVre6ezDiHhXOKWIUA5l/31YeGfs0n3x9NwRy cNWQ== X-Gm-Message-State: AGi0PuYnn/xPa/oqvFZusK//svrkSxXMguJC8VmtBcx9huabWSe+BjYd jIG0R1RVERCFFny/UcPu63nOQT42Oxg= X-Google-Smtp-Source: APiQypLRfOAdJEvbP298f1A+j4Oq8D//e+pkm0dnXmVqEInKiekU0keJwoby/ueqo+YhSI3P8UboXg== X-Received: by 2002:a17:90a:a40d:: with SMTP id y13mr4560676pjp.82.1586342775873; Wed, 08 Apr 2020 03:46:15 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:15 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 20/46] e2fsck: rbtree bitmap for dir Date: Wed, 8 Apr 2020 19:44:48 +0900 Message-Id: <1586342714-12536-21-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Only rbtree supports merge, so use it for all bitmaps. Change-Id: I863687ce275f9c891cd2d18c115cb75c6e24f4e4 Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- lib/ext2fs/gen_bitmap64.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/lib/ext2fs/gen_bitmap64.c b/lib/ext2fs/gen_bitmap64.c index ccba4427..78384e20 100644 --- a/lib/ext2fs/gen_bitmap64.c +++ b/lib/ext2fs/gen_bitmap64.c @@ -86,7 +86,6 @@ static void warn_bitmap(ext2fs_generic_bitmap_64 bitmap, #define INC_STAT(map, name) ;; #endif - errcode_t ext2fs_alloc_generic_bmap(ext2_filsys fs, errcode_t magic, int type, __u64 start, __u64 end, __u64 real_end, @@ -109,11 +108,7 @@ errcode_t ext2fs_alloc_generic_bmap(ext2_filsys fs, errcode_t magic, ops = &ext2fs_blkmap64_rbtree; break; case EXT2FS_BMAP64_AUTODIR: - retval = ext2fs_get_num_dirs(fs, &num_dirs); - if (retval || num_dirs > (fs->super->s_inodes_count / 320)) - ops = &ext2fs_blkmap64_bitarray; - else - ops = &ext2fs_blkmap64_rbtree; + ops = &ext2fs_blkmap64_rbtree; break; default: return EINVAL; From patchwork Wed Apr 8 10:44:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267881 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=hJ+HoF2W; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cx0pXtz9sQx for ; Wed, 8 Apr 2020 20:46:21 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728260AbgDHKqU (ORCPT ); Wed, 8 Apr 2020 06:46:20 -0400 Received: from mail-pg1-f193.google.com ([209.85.215.193]:35630 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqU (ORCPT ); Wed, 8 Apr 2020 06:46:20 -0400 Received: by mail-pg1-f193.google.com with SMTP id k5so3163656pga.2 for ; Wed, 08 Apr 2020 03:46:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NeYaeqeHKbr3ZeQnojCuWLsnAWyh32SFKAIBBhWg3Tw=; b=hJ+HoF2W37XF/se/2BM2PElP1tXHxLiHYlYhonKPO2vqp5KGeHsQKuwhs4pW2ztUTt U/Fee6DI0vsMiUu1Aj7bSL01qTEBDj0dBWVnJ+uIh9zXeDE18nrIhjfqHbz6UX/NJ8Nn zZJnem4akHuU9LPV30PXBh4cyv/62w48+PQ+Y9CZ3yTxza3ZKkBj4Qg/AjuSFRqGGbh6 eiZ00AXlsZH2qMO2gxLGTs5y18I1KdIImTjRld2BSXbCumnUVfQVFNJZ3WX6bUXQEGTO Z2WcBp1mtzzS0mTGhc5PRz7Dlm4eF6qSW/fMCiWw6vmDEbwpUOg8RoDjyVzFa5wr2X0a ttiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NeYaeqeHKbr3ZeQnojCuWLsnAWyh32SFKAIBBhWg3Tw=; b=KB7Rouzs17Jsp10kZc71W68V6ZVemlmlw3I41Myb7pEcOMckxEVqlWOMYLmSyFXw/N vAeKaAViFwPHz7XNEDu8wMhhWiIzcwg0BhVGrCueVYLJH69lRUoP0PlCXX3JzyqrLa0z PKAX7bFxZwRLpprbD0xgtWhbmzMttq0m6zhIQLEmC5gPKoLChkWaabto6gzAhWi7hOuH lQczWQeuk9u1VCkECZOFXG3bKb1BOz19cSHEyyu6lsRg9tfYhpljpcyF/qzyQAQncXvS t4wrrFqWfNMx0RwFoNpcg3AgLHnPfRTMml3Xzkvsq7re4ocAIkq4XMTUiGdPV51doaH7 A/Lg== X-Gm-Message-State: AGi0PuYqIWPY3k8/IE6/HurGi5nXC+L4oKh9daN3pn4uV/efgz356wrM Keh/UPZUobVoZ0ufg2U1fdfA9iXBJjo= X-Google-Smtp-Source: APiQypIYfppn4soSmyCTuhiMihlJkBcNd3yp+B7kf8WNFvI4vw4Hjrl+GskqcWwJxqJQxNwVHu9l7g== X-Received: by 2002:a62:2ad0:: with SMTP id q199mr7397721pfq.48.1586342777936; Wed, 08 Apr 2020 03:46:17 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:17 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 21/46] e2fsck: merge badblocks after thread finishes Date: Wed, 8 Apr 2020 19:44:49 +0900 Message-Id: <1586342714-12536-22-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 8 ++++- lib/ext2fs/badblocks.c | 75 ++++++++++++++++++++++++++++++++++++++---- lib/ext2fs/ext2fs.h | 2 ++ lib/ext2fs/ext2fsP.h | 1 - 4 files changed, 77 insertions(+), 9 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 56004c9b..79a9eddf 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2283,16 +2283,19 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) io_channel dest_image_io; ext2fs_inode_bitmap inode_map; ext2fs_block_bitmap block_map; + ext2_badblocks_list badblocks; dest_io = dest->io; dest_image_io = dest->image_io; inode_map = dest->inode_map; block_map = dest->block_map; + badblocks = dest->badblocks; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->io = dest_io; dest->image_io = dest_image_io; dest->inode_map = inode_map; dest->block_map = block_map; + dest->badblocks = badblocks; /* * PASS1_MERGE_FS_BITMAP might return directly from this function, * so please do NOT leave any garbage behind after returning. @@ -2309,7 +2312,10 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) } if (src->badblocks) { - retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); + if (dest->badblocks == NULL) + retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); + else + retval = ext2fs_badblocks_merge(src->badblocks, dest->badblocks); if (retval) goto out_dblist; } diff --git a/lib/ext2fs/badblocks.c b/lib/ext2fs/badblocks.c index 0f23983b..568634ff 100644 --- a/lib/ext2fs/badblocks.c +++ b/lib/ext2fs/badblocks.c @@ -11,6 +11,7 @@ #include "config.h" #include +#include #include #if HAVE_UNISTD_H #include @@ -56,6 +57,65 @@ static errcode_t make_u32_list(int size, int num, __u32 *list, return 0; } +/* + * Merge list from src to dest + */ +static errcode_t merge_u32_list(ext2_u32_list src, ext2_u32_list dest) +{ + errcode_t retval; + int src_count = src->num; + int dest_count = dest->num; + int size = src->size + dest->size; + int size_entry = sizeof(blk_t); + blk_t *array; + blk_t *array_ptr; + blk_t *src_array = src->list; + blk_t *dest_array = dest->list; + int src_index = 0; + int dest_index = 0; + + if (src->num == 0) + return 0; + + retval = ext2fs_get_array(size, size_entry, &array); + if (retval) + return retval; + + array_ptr = array; + /* + * This can be improved by binary search and memcpy, but codes would + * be complexer. And if number of bad blocks is small, the optimization + * won't improve performance a lot. + */ + while (src_index < src_count || dest_index < dest_count) { + if (src_index >= src_count) { + memcpy(array_ptr, &dest_array[dest_index], + (dest_count - dest_index) * size_entry); + break; + } + if (dest_index >= dest_count) { + memcpy(array_ptr, &src_array[src_index], + (src_count - src_index) * size_entry); + break; + } + if (src_array[src_index] < dest_array[dest_index]) { + *array_ptr = src_array[src_index]; + src_index++; + } else { + assert(src_array[src_index] > dest_array[dest_index]); + *array_ptr = dest_array[dest_index]; + dest_index++; + } + array_ptr++; + } + + ext2fs_free_mem(&dest->list); + dest->list = array; + dest->num = src_count + dest_count; + dest->size = size; + return 0; +} + /* * This procedure creates an empty u32 list. @@ -79,13 +139,7 @@ errcode_t ext2fs_badblocks_list_create(ext2_badblocks_list *ret, int size) */ errcode_t ext2fs_u32_copy(ext2_u32_list src, ext2_u32_list *dest) { - errcode_t retval; - - retval = make_u32_list(src->size, src->num, src->list, dest); - if (retval) - return retval; - (*dest)->badblocks_flags = src->badblocks_flags; - return 0; + return make_u32_list(src->size, src->num, src->list, dest); } errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src, @@ -95,6 +149,13 @@ errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src, (ext2_u32_list *) dest); } +errcode_t ext2fs_badblocks_merge(ext2_badblocks_list src, + ext2_badblocks_list dest) +{ + return merge_u32_list((ext2_u32_list) src, + (ext2_u32_list) dest); +} + /* * This procedure frees a badblocks list. * diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 2cc6d76e..1404e14a 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -814,6 +814,8 @@ extern int ext2fs_badblocks_list_iterate(ext2_badblocks_iterate iter, extern void ext2fs_badblocks_list_iterate_end(ext2_badblocks_iterate iter); extern errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src, ext2_badblocks_list *dest); +extern errcode_t ext2fs_badblocks_merge(ext2_badblocks_list src, + ext2_badblocks_list dest); extern int ext2fs_badblocks_equal(ext2_badblocks_list bb1, ext2_badblocks_list bb2); extern int ext2fs_u32_list_count(ext2_u32_list bb); diff --git a/lib/ext2fs/ext2fsP.h b/lib/ext2fs/ext2fsP.h index ad8b7d52..02df759a 100644 --- a/lib/ext2fs/ext2fsP.h +++ b/lib/ext2fs/ext2fsP.h @@ -34,7 +34,6 @@ struct ext2_struct_u32_list { int num; int size; __u32 *list; - int badblocks_flags; }; struct ext2_struct_u32_iterate { From patchwork Wed Apr 8 10:44:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267882 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=JHcKOGJV; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cy13Nsz9sQx for ; Wed, 8 Apr 2020 20:46:22 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728262AbgDHKqV (ORCPT ); Wed, 8 Apr 2020 06:46:21 -0400 Received: from mail-pj1-f66.google.com ([209.85.216.66]:51853 "EHLO mail-pj1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqV (ORCPT ); Wed, 8 Apr 2020 06:46:21 -0400 Received: by mail-pj1-f66.google.com with SMTP id n4so1004559pjp.1 for ; Wed, 08 Apr 2020 03:46:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wKveNAyEUA5dlR6L+7rclZEcmVj1eEMaTYNtMs1Jm60=; b=JHcKOGJVrjG8eW5YqhQLr9lHQ+02LEKUKb88jjc8REcOCuc4RvO6yZUcXGQsYI2+0J zBLgDO8yCpBgtl0iOu8IH187UMzbigJ5GgRPI3RA0X1DUnAptcpnlKrdoJcmuCyCCW2O FpyPj+u39LkZfg3HiNKSZzD50iNJdOYjXxZKf29FcL83hzRda23LN3V0MjlknOlL7TpU j1fUAEdkM7rXgwMehPfV5iYDbLaqv1QBes+djWIk27cPAYZMU2e7dMOlz7m+P2c38vtH VKXUk5UqIa8DNYvSJJEsAPqsdYhwxlYc/BTUUueoN2iyK5Lk5u23H25DHXDqO+YBZVC9 lEhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wKveNAyEUA5dlR6L+7rclZEcmVj1eEMaTYNtMs1Jm60=; b=gyEd8zqYLja6Y4ni3sQO7oii5NCtRHJQPyHwVeF5eIDRWR7lstuOuJ2Snr/cEA0KqP AlxqAJqc5Rj6//jSOKqqZxwRwn80TagyC2uUjsClB3RjPGBjexI4GL2+IvegJxFgx/ZO UQh7vyVEOsVAVjVxLorRdRdkr8kSSaXGoMkV7tpJB4MwiXJeTvL4Cr0dAaShb5PiCyB7 0kS1xHWihYiiSTZmQkBrY2UrYgVwbjXYZ4rFYnbTTfvavfyPJMmjbO+XZv9Gc6+dHoKB sumCawKcpka41PEbWSXmTG3OegHHzcmyRbHQXi1YfFdiaz6QPINIoWEfhRvRo0ynHi9r 7bbA== X-Gm-Message-State: AGi0PubtkL3w0i0g2O/QJ0S8hR79JRRowcCpPVbDQRGpMxVbp70ZJiVB TSAPMwlyNuSfijk9gmx+7TdxfB0dGDg= X-Google-Smtp-Source: APiQypI8SnIxt+s/6dzYbeqVznqvVoTtz+2s3B6oswS/BaaO91ROekF2phd2ab/BiE4P9ir7pabVuA== X-Received: by 2002:a17:902:b60d:: with SMTP id b13mr6852672pls.324.1586342780077; Wed, 08 Apr 2020 03:46:20 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:19 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 22/46] e2fsck: merge icounts after thread finishes Date: Wed, 8 Apr 2020 19:44:50 +0900 Message-Id: <1586342714-12536-23-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 35 +++++++++++++++- lib/ext2fs/ext2fs.h | 1 + lib/ext2fs/icount.c | 100 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 135 insertions(+), 1 deletion(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 79a9eddf..3501e2f7 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2453,6 +2453,28 @@ static void e2fsck_pass1_merge_dir_info(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->dir_info); } +#define PASS1_MERGE_CTX_ICOUNT(_dest, _src, _field) \ +do { \ + if (_src->_field) { \ + if (_dest->_field == NULL) { \ + _dest->_field = _src->_field; \ + _src->_field = NULL; \ + } else { \ + errcode_t _ret; \ + _ret = ext2fs_icount_merge(_src->_field, _dest->_field); \ + if (_ret) \ + return _ret; \ + } \ + } \ +} while (0) + +static errcode_t e2fsck_pass1_merge_icounts(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + PASS1_MERGE_CTX_ICOUNT(global_ctx, thread_ctx, inode_count); + PASS1_MERGE_CTX_ICOUNT(global_ctx, thread_ctx, inode_link_info); + return 0; +} + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2472,7 +2494,9 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; ext2fs_block_bitmap block_metadata_map = global_ctx->block_metadata_map; ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; - + ext2_icount_t inode_count = global_ctx->inode_count; + ext2_icount_t inode_link_info = global_ctx->inode_link_info; + #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2495,6 +2519,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->block_metadata_map = block_metadata_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); + global_ctx->inode_count = inode_count; + global_ctx->inode_link_info = inode_link_info; /* Keep the global singal flags*/ global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | @@ -2510,6 +2536,11 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->logf = global_logf; global_ctx->problem_logf = global_problem_logf; global_ctx->global_ctx = NULL; + retval = e2fsck_pass1_merge_icounts(global_ctx, thread_ctx); + if (retval) { + com_err(global_ctx->program_name, 0, _("while merging icounts\n")); + return retval; + } /* * PASS1_COPY_CTX_BITMAP might return directly from this function, @@ -2551,6 +2582,8 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) PASS1_FREE_CTX_BITMAP(thread_ctx, block_dup_map); PASS1_FREE_CTX_BITMAP(thread_ctx, block_ea_map); PASS1_FREE_CTX_BITMAP(thread_ctx, block_metadata_map); + ext2fs_free_icount(thread_ctx->inode_count); + ext2fs_free_icount(thread_ctx->inode_link_info); ext2fs_free_mem(&thread_ctx); return retval; diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 1404e14a..d4f6031a 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -1507,6 +1507,7 @@ extern errcode_t ext2fs_icount_decrement(ext2_icount_t icount, ext2_ino_t ino, __u16 *ret); extern errcode_t ext2fs_icount_store(ext2_icount_t icount, ext2_ino_t ino, __u16 count); +extern errcode_t ext2fs_icount_merge(ext2_icount_t src, ext2_icount_t dest); extern ext2_ino_t ext2fs_get_icount_size(ext2_icount_t icount); errcode_t ext2fs_icount_validate(ext2_icount_t icount, FILE *); diff --git a/lib/ext2fs/icount.c b/lib/ext2fs/icount.c index 888a90b2..a72b53b3 100644 --- a/lib/ext2fs/icount.c +++ b/lib/ext2fs/icount.c @@ -13,6 +13,7 @@ #if HAVE_UNISTD_H #include #endif +#include #include #include #include @@ -701,6 +702,105 @@ errcode_t ext2fs_icount_store(ext2_icount_t icount, ext2_ino_t ino, return 0; } +errcode_t ext2fs_icount_merge_full_map(ext2_icount_t src, ext2_icount_t dest) +{ + /* TODO: add the support for full map */ + return EOPNOTSUPP; +} + +errcode_t ext2fs_icount_merge_el(ext2_icount_t src, ext2_icount_t dest) +{ + int src_count = src->count; + int dest_count = dest->count; + int size = src->size + dest->size; + int size_entry = sizeof(struct ext2_icount_el); + struct ext2_icount_el *array; + struct ext2_icount_el *array_ptr; + struct ext2_icount_el *src_array = src->list; + struct ext2_icount_el *dest_array = dest->list; + int src_index = 0; + int dest_index = 0; + errcode_t retval; + + if (src_count == 0) + return 0; + + retval = ext2fs_get_array(size, size_entry, &array); + if (retval) + return retval; + + array_ptr = array; + /* + * This can be improved by binary search and memcpy, but codes would + * be complexer. And if number of bad blocks is small, the optimization + * won't improve performance a lot. + */ + while (src_index < src_count || dest_index < dest_count) { + if (src_index >= src_count) { + memcpy(array_ptr, &dest_array[dest_index], + (dest_count - dest_index) * size_entry); + break; + } + if (dest_index >= dest_count) { + memcpy(array_ptr, &src_array[src_index], + (src_count - src_index) * size_entry); + break; + } + if (src_array[src_index].ino < dest_array[dest_index].ino) { + *array_ptr = src_array[src_index]; + src_index++; + } else { + assert(src_array[src_index].ino > + dest_array[dest_index].ino); + *array_ptr = dest_array[dest_index]; + dest_index++; + } + array_ptr++; + } + + ext2fs_free_mem(&dest->list); + dest->list = array; + dest->count = src_count + dest_count; + dest->size = size; + return 0; +} + +errcode_t ext2fs_icount_merge(ext2_icount_t src, ext2_icount_t dest) +{ + errcode_t retval; + + if (src->fullmap && !dest->fullmap) + return EINVAL; + + if (!src->fullmap && dest->fullmap) + return EINVAL; + + if (src->multiple && !dest->multiple) + return EINVAL; + + if (!src->multiple && dest->multiple) + return EINVAL; + + if (src->fullmap) + return ext2fs_icount_merge_full_map(src, dest); + + retval = ext2fs_merge_bitmap(src->single, dest->single); + if (retval) + return retval; + + if (src->multiple) { + retval = ext2fs_merge_bitmap(src->multiple, dest->multiple); + if (retval) + return retval; + } + + retval = ext2fs_icount_merge_el(src, dest); + if (retval) + return retval; + + return 0; +} + ext2_ino_t ext2fs_get_icount_size(ext2_icount_t icount) { if (!icount || icount->magic != EXT2_ET_MAGIC_ICOUNT) From patchwork Wed Apr 8 10:44:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267883 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=eMaOsg34; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Cz6Z0Nz9sQx for ; Wed, 8 Apr 2020 20:46:23 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728264AbgDHKqX (ORCPT ); Wed, 8 Apr 2020 06:46:23 -0400 Received: from mail-pg1-f196.google.com ([209.85.215.196]:45378 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqX (ORCPT ); Wed, 8 Apr 2020 06:46:23 -0400 Received: by mail-pg1-f196.google.com with SMTP id 128so534078pge.12 for ; Wed, 08 Apr 2020 03:46:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FMSn/bpiVdxm7dqeeMs1n0Fund38tS3pgY6V3KwE368=; b=eMaOsg34OG8dAGtclbrycjkU+OK97D8Z8Dtx/FmS044SgRh5ruFDvF418CPr1Tscum 2hwdYPeVjVV/0bfYYxVpi+cDATPchAv2J0OpeEjn1x6wMTtrs8qhXugReludsqcwqO+U +cMMWn1B97YhB+XsYQcn6Y8bK1kBw4eDj6VLs4QBGLgx3uQU8tJsBLaGUIDlNa1NJJ8g xHCwOT1tb/4Qe1VedCdU4wqYkvE6ZX38PfaeEx65aN1XmO34f+t4hLUI9FUp+TBhD4bJ 7AJ+WJzseI81jLPdkSVCkCPOAsJrsrcL33kxLkdZIDEFj7dd5Q/neIX6cs0s7HhzdZPk 3t2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FMSn/bpiVdxm7dqeeMs1n0Fund38tS3pgY6V3KwE368=; b=Tb7lYqJym4wNQ9Wf/Btd6K8fgX/SW9D0hz7hUPYMeBnFFESxxecKbNIrCqki/eAC+v j3b+1u7isNLjLGPc33UCZ3HgCw3GIhvkrV5Q61HP6BTqeTw9V5iziBq38t6B8oOzg56r i7cb3Jl1XqPZn1wQoTNM/qTHdCDdO8juY4IQroLcH+yfANbwZBUIUCeYBS1QseEp5NGS +cGY/gpHyrsR4uObqn198HrHYCpP6ad4ezHQ8H84H4X9bHSYG71/V5Go72UqdZYLtpp0 NL8Jmn1pjGxNWVovoS00HiB/PZ2ENvkmbmOhxZ0GGGInpBkU5Yql2Sm7YsgLPdZNBRml 8I6A== X-Gm-Message-State: AGi0PuaMpDHEaR7JWnWk6hyt5S8DThQwtF29s/+xUmULXshUHN8HE4Zf z/rjTc0BP+5KGQ+KDLoEeAlG9bnolD8= X-Google-Smtp-Source: APiQypIasGoo2mn/UlhUjBP4KAZQBkZZmoIHGXbb3jiAibqsk4UMXfq/1mkfOdO5/xAcnxx0D25RMQ== X-Received: by 2002:a62:ce8a:: with SMTP id y132mr6880613pfg.163.1586342782201; Wed, 08 Apr 2020 03:46:22 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:21 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 23/46] e2fsck: merge dblist after thread finishes Date: Wed, 8 Apr 2020 19:44:51 +0900 Message-Id: <1586342714-12536-24-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 18 +++++++++++++----- lib/ext2fs/dblist.c | 36 ++++++++++++++++++++++++++++++++++++ lib/ext2fs/ext2fs.h | 1 + 3 files changed, 50 insertions(+), 5 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 3501e2f7..1f47cbff 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2284,18 +2284,21 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) ext2fs_inode_bitmap inode_map; ext2fs_block_bitmap block_map; ext2_badblocks_list badblocks; + ext2_dblist dblist; dest_io = dest->io; dest_image_io = dest->image_io; inode_map = dest->inode_map; block_map = dest->block_map; badblocks = dest->badblocks; + dblist = dest->dblist; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->io = dest_io; dest->image_io = dest_image_io; dest->inode_map = inode_map; dest->block_map = block_map; dest->badblocks = badblocks; + dest->dblist = dblist; /* * PASS1_MERGE_FS_BITMAP might return directly from this function, * so please do NOT leave any garbage behind after returning. @@ -2304,11 +2307,16 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) PASS1_MERGE_FS_BITMAP(dest, src, block_map); if (src->dblist) { - retval = ext2fs_copy_dblist(src->dblist, &dest->dblist); - if (retval) - return retval; - /* The ext2fs_copy_dblist() uses the src->fs as the fs */ - dest->dblist->fs = dest; + if (dest->dblist) { + retval = ext2fs_merge_dblist(src->dblist, dest->dblist); + if (retval) + return retval; + } else { + /* The ext2fs_copy_dblist() uses the src->fs as the fs */ + dest->dblist = src->dblist; + dest->dblist->fs = dest; + src->dblist = NULL; + } } if (src->badblocks) { diff --git a/lib/ext2fs/dblist.c b/lib/ext2fs/dblist.c index bbdb221d..64caf843 100644 --- a/lib/ext2fs/dblist.c +++ b/lib/ext2fs/dblist.c @@ -119,6 +119,42 @@ errcode_t ext2fs_copy_dblist(ext2_dblist src, ext2_dblist *dest) return 0; } +/* + * Merge a directory block list @src to @dest + */ +errcode_t ext2fs_merge_dblist(ext2_dblist src, ext2_dblist dest) +{ + int src_count = src->count; + int dest_count = dest->count; + int size = src->size + dest->size; + int size_entry = sizeof(struct ext2_db_entry2); + struct ext2_db_entry2 *array, *array2; + errcode_t retval; + + if (src_count == 0) + return 0; + + if (src->sorted || dest->sorted) + return EINVAL; + + retval = ext2fs_get_array(size, size_entry, &array); + if (retval) + return retval; + + array2 = array; + + memcpy(array, src->list, src_count * size_entry); + array += src_count; + memcpy(array, dest->list, dest_count * size_entry); + ext2fs_free_mem(&dest->list); + + dest->list = array2; + dest->count = src_count + dest_count; + dest->size = size; + + return 0; +} + /* * Close a directory block list * diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index d4f6031a..6c872ed1 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -1120,6 +1120,7 @@ extern errcode_t ext2fs_add_dir_block(ext2_dblist dblist, ext2_ino_t ino, blk_t blk, int blockcnt); extern errcode_t ext2fs_add_dir_block2(ext2_dblist dblist, ext2_ino_t ino, blk64_t blk, e2_blkcnt_t blockcnt); +extern errcode_t ext2fs_merge_dblist(ext2_dblist src, ext2_dblist dest); extern void ext2fs_dblist_sort(ext2_dblist dblist, EXT2_QSORT_TYPE (*sortfunc)(const void *, const void *)); From patchwork Wed Apr 8 10:44:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267884 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=WEpgVESF; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1D16DwSz9sQx for ; Wed, 8 Apr 2020 20:46:25 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728268AbgDHKqZ (ORCPT ); Wed, 8 Apr 2020 06:46:25 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:43482 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqZ (ORCPT ); Wed, 8 Apr 2020 06:46:25 -0400 Received: by mail-pl1-f194.google.com with SMTP id z6so882210plk.10 for ; Wed, 08 Apr 2020 03:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=q0g/zyyg3cWNQoMPaaZkiDSHcWfec0cZrjPhzZaKF2I=; b=WEpgVESF1tELt8bIVgSAiStyuydvQd013mG9Wo6+nmpVX2omz9d+PACB+YMcKVMrts 7GNMHAq/d1VBRSedoEQxqHW/OQUxzSeiGI6FkXnqaQertLV56MJur7Jr2ZepQlE6JEm4 kj0zxjLhEohV8qtt52sVRQ+AddwOYnBDQCk1Z/4gZQdfIhcDz3YZ8EgOmmkOOILJalbY 60myYCs9CGoWS0bsE6BgI7fVzmbAQeHdWdCJfgj+PJHmjKlbV3v1rFCgwgoIbkmfgyLu vFArI5kZPQVNOnXIOU8oQVr4jNoTSwiiSHBOHDLbepP7ZfnXZBV8PuXJ64gNJsoEBBl9 ONdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=q0g/zyyg3cWNQoMPaaZkiDSHcWfec0cZrjPhzZaKF2I=; b=mPh8WKGDhJuuU+/S6H04H2ohhTOKQWIVg4lHZkeJwm/51ByFGrBgQ4OhLqvWVuqs2G 6kRBNzE3YrrgsvLxhPVkcnZPXD6hoobgyNx5JwS/r0yckZH9i280/XpGOzxFlvVl9FnV YlQClLeTloyQ6fKQ6iLGXQ8rsQuWfLOZkJYHxvevwhcqFLcyTwKEQSFpEdOQS1f+Rngg 6CH21jLmDs8Xu5A71GzKPTwoQA3Tf79H0Z1e68B0bl7j4gNAayODJYdDtpGPB10CGdR9 NCIgzBqDSfOOn2Ba+chGA+dEXETzO7M3FMvlODUnb4kJmu0u1u1VXIZ3y91CewByjiJY GQmg== X-Gm-Message-State: AGi0PuY1xTOv7DZa9M183NIGa+NSosimptoEye1NrsO1Kkp51xH3uKCg 7DFwAXcDsn7o8SA6LqYcvMI8nNVhL6Q= X-Google-Smtp-Source: APiQypI24txSJ5VGpbi+IijhkfEIiR428oglxXRqRo/NpXeqi6IWTJqyEV0Olzew8lOQuEuM/bw2xA== X-Received: by 2002:a17:90b:1901:: with SMTP id mp1mr4630395pjb.196.1586342784280; Wed, 08 Apr 2020 03:46:24 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:23 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 24/46] e2fsck: add debug codes for multiple threds Date: Wed, 8 Apr 2020 19:44:52 +0900 Message-Id: <1586342714-12536-25-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi These debug codes are added to run the multiple pass1 check thread one by one in order. If all the codes are correct, fsck of multiple threads should have exactly the same outcome with single thread. Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 14 ++++++++++++++ e2fsck/pass1.c | 31 ++++++++++++++++++++++++++++++- 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index f8e98f73..61761684 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -451,6 +451,17 @@ struct e2fsck_struct { }; +#ifdef DEBUG_THREADS +/* + * Enabling DEBUG_THREADS would cause the parall fsck threads run sequentially + */ +struct e2fsck_thread_debug { + pthread_mutex_t etd_mutex; + pthread_cond_t etd_cond; + int etd_finished_threads; +}; +#endif + struct e2fsck_thread_info { /* ID returned by pthread_create() */ pthread_t eti_thread_id; @@ -460,6 +471,9 @@ struct e2fsck_thread_info { int eti_started; /* Context used for this thread */ e2fsck_t eti_thread_ctx; +#ifdef DEBUG_THREADS + struct e2fsck_thread_debug *eti_debug; +#endif }; /* Data structures to evaluate whether an extent tree needs rebuilding. */ diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 1f47cbff..f8115679 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1210,7 +1210,8 @@ void _e2fsck_pass1(e2fsck_t ctx) ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs); pass1_readahead(ctx, &ra_group, &ino_threshold); - if (!(ctx->options & E2F_OPT_PREEN)) + if (!(ctx->options & E2F_OPT_PREEN) && + ((!ctx->global_ctx) || (ctx->thread_info.et_thread_index == 0))) fix_problem(ctx, PR_1_PASS_HEADER, &pctx); if (ext2fs_has_feature_dir_index(fs->super) && @@ -2635,6 +2636,17 @@ static void *e2fsck_pass1_thread(void *arg) { struct e2fsck_thread_info *info = arg; e2fsck_t thread_ctx = info->eti_thread_ctx; +#ifdef DEBUG_THREADS + struct e2fsck_thread_debug *thread_debug = info->eti_debug; +#endif + +#ifdef DEBUG_THREADS + pthread_mutex_lock(&thread_debug->etd_mutex); + while (info->eti_thread_index > thread_debug->etd_finished_threads) { + pthread_cond_wait(&thread_debug->etd_cond, &thread_debug->etd_mutex); + } + pthread_mutex_unlock(&thread_debug->etd_mutex); +#endif #ifdef HAVE_SETJMP_H /* @@ -2659,6 +2671,14 @@ out: thread_ctx->thread_info.et_group_start, thread_ctx->thread_info.et_group_end, thread_ctx->thread_info.et_inode_number); + +#ifdef DEBUG_THREADS + pthread_mutex_lock(&thread_debug->etd_mutex); + thread_debug->etd_finished_threads++; + pthread_cond_broadcast(&thread_debug->etd_cond); + pthread_mutex_unlock(&thread_debug->etd_mutex); +#endif + return NULL; } @@ -2672,6 +2692,12 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, struct e2fsck_thread_info *tmp_pinfo; int i; e2fsck_t thread_ctx; +#ifdef DEBUG_THREADS + struct e2fsck_thread_debug thread_debug = + {PTHREAD_MUTEX_INITIALIZER, PTHREAD_COND_INITIALIZER, 0}; + + thread_debug.etd_finished_threads = 0; +#endif retval = pthread_attr_init(&attr); if (retval) { @@ -2692,6 +2718,9 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, for (i = 0; i < num_threads; i++) { tmp_pinfo = &infos[i]; tmp_pinfo->eti_thread_index = i; +#ifdef DEBUG_THREADS + tmp_pinfo->eti_debug = &thread_debug; +#endif retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, i, num_threads); if (retval) { From patchwork Wed Apr 8 10:44:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267885 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Yg4sC1+1; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1D36lXQz9sQx for ; Wed, 8 Apr 2020 20:46:27 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728271AbgDHKq1 (ORCPT ); Wed, 8 Apr 2020 06:46:27 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:43597 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKq1 (ORCPT ); Wed, 8 Apr 2020 06:46:27 -0400 Received: by mail-pg1-f194.google.com with SMTP id s4so3135692pgk.10 for ; Wed, 08 Apr 2020 03:46:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Mzk7fcdLkWxvQN5L82i/kFI5nSfFfxWIleCKv+NfelE=; b=Yg4sC1+1HDKuSVUqGp0INEW1RcEOtvkWWxi3LDQWlyOdracKDK7kHQx9YXQD07I/BN kQZKilMUGhzNoAUrZXdP7i48UyqgdTfIovKCneoeadwdBnKOszj3HaoP/epeYm4dl8QQ /0Q4HRWaLt8CActziYbrmx8x018R0zi74S6AD+kKd6Iq3MNW4qhIbVS96mk4ybP0BSPm 3u5Ecuk9Jug+qZZjQvbN0Aeiu6xdP8FjypH1+PHbtzHSfPHtGzD8IUfWln7rugVVKX/k tSrXTks0rfwog/vgzDEqoB5p5p5/tX/1ycINx5xo00JSvmK3vdzrEjQCAXRp6zN94Xsy a2xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Mzk7fcdLkWxvQN5L82i/kFI5nSfFfxWIleCKv+NfelE=; b=OCNYmppWVEUNpTc0F6/Sz81oWgTj/+HG4fm8POgrevzKsW/033FXjQWD/3HX3pT9RI pl7ZvkenpKPWdzs6DCIOmHJ16f9vROM2aZn3hign5VZu4bRRoAI59g+yMYnVnYIST+0b lhDdHyPhis2fkJ1Adxy4SmT0KhybE6bJhP9Ha5fZYLDPr/0AVfaQy/p8Ir9lY0oY3wSc AqiqXfJ02p8jlBB0lh3vTybKY7W/+YdvG9pSkW56DpvFB0w5cPz/Lv+W+BiF05Gg2mRa 89lXS2ok9XDe8+CILI9v+SOhwexyEwOpOlA/wTSXwKitngSoHbAgBXw+4+Y4633y54Gh Lsyw== X-Gm-Message-State: AGi0PuYPPD4uAO2wTG/bX5upraoECtKaB92bBf1LtHygrE8XnJxDI2YR fCNg2uVf8wApiJJ1f0hC1P7gfDBefoA= X-Google-Smtp-Source: APiQypJz3JxNmbG+fMKrI6JK9XoJBCPlnRTesa8DZQHkwK9YnG3V+9v8HrZX/sVt6apMYTZTQyzlaw== X-Received: by 2002:a62:b603:: with SMTP id j3mr5806474pff.208.1586342786403; Wed, 08 Apr 2020 03:46:26 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:25 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 25/46] e2fsck: merge counts when threads finish Date: Wed, 8 Apr 2020 19:44:53 +0900 Message-Id: <1586342714-12536-26-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index f8115679..78924b24 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2171,6 +2171,11 @@ do { \ } \ } while (0) +#define PASS1_MERGE_CTX_COUNT(_dest, _src, _field) \ +do { \ + _dest->_field = _field + _src->_field; \ +} while (0) + static errcode_t pass1_open_io_channel(ext2_filsys fs, const char *io_options, io_manager manager, int flags) @@ -2505,6 +2510,23 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; ext2_icount_t inode_count = global_ctx->inode_count; ext2_icount_t inode_link_info = global_ctx->inode_link_info; + __u32 fs_directory_count = global_ctx->fs_directory_count; + __u32 fs_regular_count = global_ctx->fs_regular_count; + __u32 fs_blockdev_count = global_ctx->fs_blockdev_count; + __u32 fs_chardev_count = global_ctx->fs_chardev_count; + __u32 fs_links_count = global_ctx->fs_links_count; + __u32 fs_symlinks_count = global_ctx->fs_symlinks_count; + __u32 fs_fast_symlinks_count = global_ctx->fs_fast_symlinks_count; + __u32 fs_fifo_count = global_ctx->fs_fifo_count; + __u32 fs_total_count = global_ctx->fs_total_count; + __u32 fs_badblocks_count = global_ctx->fs_badblocks_count; + __u32 fs_sockets_count = global_ctx->fs_sockets_count; + __u32 fs_ind_count = global_ctx->fs_ind_count; + __u32 fs_dind_count = global_ctx->fs_dind_count; + __u32 fs_tind_count = global_ctx->fs_tind_count; + __u32 fs_fragmented = global_ctx->fs_fragmented; + __u32 fs_fragmented_dir = global_ctx->fs_fragmented_dir; + __u32 large_files = global_ctx->large_files; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2530,6 +2552,23 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); global_ctx->inode_count = inode_count; global_ctx->inode_link_info = inode_link_info; + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_directory_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_regular_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_blockdev_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_chardev_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_links_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_symlinks_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fast_symlinks_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fifo_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_total_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_badblocks_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_sockets_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_ind_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_dind_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_tind_count); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fragmented); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fragmented_dir); + PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, large_files); /* Keep the global singal flags*/ global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | From patchwork Wed Apr 8 10:44:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267886 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Jhj6WRjX; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1D61Gx6z9sQx for ; Wed, 8 Apr 2020 20:46:30 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728274AbgDHKq3 (ORCPT ); Wed, 8 Apr 2020 06:46:29 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:45047 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKq3 (ORCPT ); Wed, 8 Apr 2020 06:46:29 -0400 Received: by mail-pg1-f195.google.com with SMTP id n13so1717829pgp.11 for ; Wed, 08 Apr 2020 03:46:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=v5Qw9EoH4P+/bTsQks+TZOohVb6qCEbjRC01KIIttQE=; b=Jhj6WRjX5waZ5B1zNbrOEhzxEFOwxQdaIK0P1Hax361/V83URLo6jA90Xj/3m5im52 SJ/B0zXGU8wa2VEGv2Gvo2hWfh+DUI9L5d18SK5lZZ0U8EPIwzapqn6XOtW+DKCSWkiM n9vyyWdq044TWfIK36Pn2XRu7WpiWKnpZQ4dJw6SYZHc7LMp+GKGNaVyk2OPkeUNcMQo eZNWi/HRwCQrTrF08d6D8dgLkK1JIo95r8IHx8HzUHb5hjPInr66qG9bh8R25GfLn4dg rQ2YIRl9iiHeThPLVN+AYSNlixyT2FLQJErgPq/owTAEtrNzcxC05BPAD0hd1GxeHCOu HP5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=v5Qw9EoH4P+/bTsQks+TZOohVb6qCEbjRC01KIIttQE=; b=jk6SKSKX9VVHDzyHi9Z/7z9QgG7Hm2tNULH2KEHzccsqqnMOsZSBr9KBwEfLPGaJl6 8oL0x1YKjTNE37otjHSRjI0J8BUEv9IdXtlvM3f+tpTC5AoZmBdiMW3dXupMfeBRecge 12/UU9Qz8a+t8uvQJ0zNxNzlgcgdseK1WbbfaANO/Nl9uY+229iNU4n6361tmF25BHZB Zk7Q8qzmmaenKHNE+iZeJ15t2Xvb0APEWWt+pMUPkNYe7xU3k6cLJHmL9iUph2y2kEnR gzVIKF7ZrawqQ7Ej9o2vXl/seRwLQOKh7cXLY67gIOmVOIMOvKvUtOrUg3vNP+A5l4Zu MIzA== X-Gm-Message-State: AGi0PuaXZvrrL9Pi9dQNsMfXhqsN4XSqjhotPk3/aSVdy575xYzFdJQX lwjhFH1Vneie2GQJkn4OhpZR+cxeXxs= X-Google-Smtp-Source: APiQypLSz0RjIiSii7I1uuRkUdnJ8OzWRVEHIVPrZFCRJUdfQnqRED69qiaUcLVqwQ5mILvODrwo7g== X-Received: by 2002:a62:2e42:: with SMTP id u63mr7222600pfu.69.1586342788497; Wed, 08 Apr 2020 03:46:28 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:27 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 26/46] e2fsck: merge fs flags when threads finish Date: Wed, 8 Apr 2020 19:44:54 +0900 Message-Id: <1586342714-12536-27-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Signed-off-by: Li Xi Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 78924b24..4bd1f8be 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2291,6 +2291,7 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) ext2fs_block_bitmap block_map; ext2_badblocks_list badblocks; ext2_dblist dblist; + int flags; dest_io = dest->io; dest_image_io = dest->image_io; @@ -2298,6 +2299,7 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) block_map = dest->block_map; badblocks = dest->badblocks; dblist = dest->dblist; + flags = dest->flags; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->io = dest_io; dest->image_io = dest_image_io; @@ -2305,6 +2307,9 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) dest->block_map = block_map; dest->badblocks = badblocks; dest->dblist = dblist; + dest->flags = src->flags | flags; + if (!(src->flags & EXT2_FLAG_VALID) || !(flags & EXT2_FLAG_VALID)) + ext2fs_unmark_valid(dest); /* * PASS1_MERGE_FS_BITMAP might return directly from this function, * so please do NOT leave any garbage behind after returning. From patchwork Wed Apr 8 10:44:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267887 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=YWCPWCTp; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1D83kSKz9sQx for ; Wed, 8 Apr 2020 20:46:32 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728277AbgDHKqc (ORCPT ); Wed, 8 Apr 2020 06:46:32 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:34370 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqc (ORCPT ); Wed, 8 Apr 2020 06:46:32 -0400 Received: by mail-pf1-f196.google.com with SMTP id v23so2880788pfm.1 for ; Wed, 08 Apr 2020 03:46:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/hhjg0I3z+KfU2knk+zX2J3FcwBTvj22ehCcd4jD/2U=; b=YWCPWCTpuXWqR66VLWnSx55kAHtzIshOPTfjgxXa1xix7Ejiz0FxINrbqdTcFHTJze yPV+0sqnamitC8CTmeB5lmgBF0lPegcGjresCraBzUxz4dr1gD2J5C7rj9Bx+Jh8HDc2 C0PsJNoHnSRP6Ayqqpj8VgZLLLgRN0BLQm8kI44S39tAtwrrcaWD1pR6WaR6pVf5HvJ+ soZX44Ylv7Zje8DMYYIcbFyC7f1waxrcRm+ca8bygus6PnvIBpiiTue1yW5o1FPY+TBp Sv6GtYQK7z+RQusI+9bG8w/s4R4qUDfr5gqTZaNo9TpnVftiHGgnWfUgT/f5rDQN8SnN XQtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/hhjg0I3z+KfU2knk+zX2J3FcwBTvj22ehCcd4jD/2U=; b=n3mcgsrOk/q3IeQYhQp1sARTuE4vLDnNvJhvS5v9j1SL0f1AaI4o45Atmiy5t9UltI KObu4IO+wTJBOLj6b9+y6jX7aDwZKxcukFxDhRVxI9nGMl7QjmFJ+DBDsRBpQcLopcYt 7f+TvoPZYN85Cmj5gUBhR7Fq3odK3E7K5LTjTSv2hUZ81W/E1XD7yMrLJVJVKKxhYbpN SXeYPqJEq8cY1Pu8sweC+oE6q4vfUWtqmL3cyAQHcq6VhCBuVEnyBEUuRcfvP7OZKwLW bAs+paXvi+Zar8dOXPGb2uVH940i8fv6Xx/lYtgZ8taVezNXv3L+WyQHk/mmj++gwTGG ldGA== X-Gm-Message-State: AGi0PuZhC6puSaH4Ru3wzM+g+5fxozruJAtpselFkQFE28KLHkT4GUZ+ OrxqQbQPS81pbSL0z+MvRowI8sm0h5M= X-Google-Smtp-Source: APiQypJuZBZHaas1Rk4jABvg0XkX7Bqocnc9dcssLVmhyIg7Zel9w13N1ATnXEcuNmVcU5zcsuR2eQ== X-Received: by 2002:aa7:85c4:: with SMTP id z4mr7174377pfn.121.1586342790713; Wed, 08 Apr 2020 03:46:30 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:30 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 27/46] e2fsck: merge dx_dir_info Date: Wed, 8 Apr 2020 19:44:55 +0900 Message-Id: <1586342714-12536-28-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Change-Id: I250de2d510e3c71974f6c853d5f6ff01229d5573 Signed-off-by: Wang Shilong --- e2fsck/dx_dirinfo.c | 67 +++++++++++++++++++++++++++++++++++++++++++++ e2fsck/e2fsck.h | 1 + e2fsck/pass1.c | 25 +++++++++++++++++ 3 files changed, 93 insertions(+) diff --git a/e2fsck/dx_dirinfo.c b/e2fsck/dx_dirinfo.c index c0b0e9a4..cf9849dc 100644 --- a/e2fsck/dx_dirinfo.c +++ b/e2fsck/dx_dirinfo.c @@ -80,6 +80,73 @@ void e2fsck_add_dx_dir(e2fsck_t ctx, ext2_ino_t ino, struct ext2_inode *inode, } +/* + * Merge two sorted dir info to @dest + */ +void e2fsck_merge_dx_dir(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + struct dx_dir_info *src_array = thread_ctx->dx_dir_info; + struct dx_dir_info *dest_array = global_ctx->dx_dir_info; + int size_dx_info = sizeof(struct dx_dir_info); + int size = global_ctx->dx_dir_info_size; + int src_count = thread_ctx->dx_dir_info_count; + int dest_count = global_ctx->dx_dir_info_count; + int total_count = src_count + dest_count; + struct dx_dir_info *array; + struct dx_dir_info *array_ptr; + int src_index = 0, dest_index = 0; + int i; + + if (thread_ctx->dx_dir_info_count == 0) + return; + + if (size < total_count) + size = total_count; + + if (size < thread_ctx->dx_dir_info_size > size) + size = size; + + array = e2fsck_allocate_memory(global_ctx, size * size_dx_info, + "directory map"); + array_ptr = array; + /* + * This can be improved by binary search and memcpy, but codes + * would be complexer. And if the groups distributed to each + * thread are stided, this implementation won't be too bad comparing + * to the optimiztion. + */ + while (src_index < src_count || dest_index < dest_count) { + if (src_index >= src_count) { + memcpy(array_ptr, &dest_array[dest_index], + (dest_count - dest_index) * size_dx_info); + break; + } + if (dest_index >= dest_count) { + memcpy(array_ptr, &src_array[src_index], + (src_count - src_index) * size_dx_info); + break; + } + if (src_array[src_index].ino < dest_array[dest_index].ino) { + *array_ptr = src_array[src_index]; + src_index++; + } else { + /* + assert(src_array[src_index].ino > + dest_array[dest_index].ino); + */ + *array_ptr = dest_array[dest_index]; + dest_index++; + } + array_ptr++; + } + + if (global_ctx->dx_dir_info) + ext2fs_free_mem(&global_ctx->dx_dir_info); + global_ctx->dx_dir_info = array; + global_ctx->dx_dir_info_size = size; + global_ctx->dx_dir_info_count = total_count; +} + /* * get_dx_dir_info() --- given an inode number, try to find the directory * information entry for it. diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 61761684..0b449b69 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -545,6 +545,7 @@ extern struct dx_dir_info *e2fsck_get_dx_dir_info(e2fsck_t ctx, ext2_ino_t ino); extern void e2fsck_free_dx_dir_info(e2fsck_t ctx); extern int e2fsck_get_num_dx_dirinfo(e2fsck_t ctx); extern struct dx_dir_info *e2fsck_dx_dir_info_iter(e2fsck_t ctx, int *control); +extern void e2fsck_merge_dx_dir(e2fsck_t global_ctx, e2fsck_t thread_ctx); /* ea_refcount.c */ typedef __u64 ea_key_t; diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 4bd1f8be..6789b701 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2472,6 +2472,24 @@ static void e2fsck_pass1_merge_dir_info(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->dir_info); } +static void e2fsck_pass1_merge_dx_dir(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + if (thread_ctx->dx_dir_info == NULL) + return; + + if (global_ctx->dx_dir_info == NULL) { + /* TODO: tdb needs to be handled properly */ + global_ctx->dx_dir_info = thread_ctx->dx_dir_info; + global_ctx->dx_dir_info_size = thread_ctx->dx_dir_info_size; + global_ctx->dx_dir_info_count = thread_ctx->dx_dir_info_count; + thread_ctx->dx_dir_info = NULL; + return; + } + + e2fsck_merge_dx_dir(global_ctx, thread_ctx); +} + + #define PASS1_MERGE_CTX_ICOUNT(_dest, _src, _field) \ do { \ if (_src->_field) { \ @@ -2503,6 +2521,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx FILE *global_logf = global_ctx->logf; FILE *global_problem_logf = global_ctx->problem_logf; struct dir_info_db *dir_info = global_ctx->dir_info; + struct dx_dir_info *dx_dir_info = global_ctx->dx_dir_info; ext2fs_inode_bitmap inode_used_map = global_ctx->inode_used_map; ext2fs_inode_bitmap inode_dir_map = global_ctx->inode_dir_map; ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; @@ -2532,6 +2551,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx __u32 fs_fragmented = global_ctx->fs_fragmented; __u32 fs_fragmented_dir = global_ctx->fs_fragmented_dir; __u32 large_files = global_ctx->large_files; + int dx_dir_info_size = global_ctx->dx_dir_info_size; + int dx_dir_info_count = global_ctx->dx_dir_info_count; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2555,6 +2576,10 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->block_metadata_map = block_metadata_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); + global_ctx->dx_dir_info = dx_dir_info; + global_ctx->dx_dir_info_count = dx_dir_info_count; + global_ctx->dx_dir_info_size = dx_dir_info_size; + e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); global_ctx->inode_count = inode_count; global_ctx->inode_link_info = inode_link_info; PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_directory_count); From patchwork Wed Apr 8 10:44:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267888 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=MjJB/HSR; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DD1S3nz9sQx for ; Wed, 8 Apr 2020 20:46:36 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728280AbgDHKqf (ORCPT ); Wed, 8 Apr 2020 06:46:35 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:34895 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728250AbgDHKqf (ORCPT ); Wed, 8 Apr 2020 06:46:35 -0400 Received: by mail-pf1-f194.google.com with SMTP id a13so2237161pfa.2 for ; Wed, 08 Apr 2020 03:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=o6jxCY3/7VptX/dKB1Kquoqkng2DESomuLljR5WkpYA=; b=MjJB/HSRzoiyPp3P8DwioZ/G+IoUOtQM5hFxYPBCEpg7ovdJJRFbY7zCQR1QYhEdPq BZTJjob60mk3CFlz4chQcj1C6O/CfYrMJAD3pP7CcgFQnn9pwIQ82OPR+SzJSig05nmX X4jhQQfHfYp5D4+HGAsDwHqR5tM2lMW+UD+jt2MnegxKMB1cttIs4ZSkqZAV170UEpji 3ETQ8WWPY8AHw6pWKG6dY31u7YzrBMc4vITsTfMgPwlyQ7waBVBnyNdgVQ4qXBrXlxTT h7mfooXTEXcg/puHMkQeUOS696snM8qxmWhS1pMwi2obQjk3dLiH4N4qdM6ELRvRLSsz u7fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=o6jxCY3/7VptX/dKB1Kquoqkng2DESomuLljR5WkpYA=; b=pcgOUVkoBaD+rK8LOPpCPufIwAqWwNKiIiZ8nmcYc30A7OuEQPK9CUlbA7pyiNeqzD 0h4SFXo9i/BiDBvbOyPPhpksnm67KeJEN9n45ZE72lY5DcaOgUtew52rY9jkA4cZ6Qyg KkVb2+hTp6C25yIQXqVZVt9Cl9PkbRTxZlCzk+zQNkhvq4uL4aIHGEoGJ5EtfbU4USOW 1oNBocBt8AK4euQPxydCzMQpGNc24Cy/3XpGKUhO5E1F+Z5aAgMk55o5g72wWsjctO7M x8O4CupVw0bLGw5R3ldjPm5qO5y/yuU0UEpWhjOH1nOPN8/mGsBafABLnS51cpkXFlcU EOWg== X-Gm-Message-State: AGi0PuZCYMc+zCsbsboIHRS2Uao+14rSjwsfOaCiF7JRBsA+YOsNXFw/ Ktwmqn6UVHH/MAtQ7ZF7oY6/STfPy4Y= X-Google-Smtp-Source: APiQypI8IqLzvHxtrY6IgXv+wOZMqyJAOJnuMNwSTWFcidFnUPNWeCz04Y5/Uar+qHdcZdfuwQOgCA== X-Received: by 2002:a65:5642:: with SMTP id m2mr6591232pgs.191.1586342792777; Wed, 08 Apr 2020 03:46:32 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:32 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 28/46] e2fsck: make threads splitting aware of flex_bg Date: Wed, 8 Apr 2020 19:44:56 +0900 Message-Id: <1586342714-12536-29-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Flex_bg might be enabled, if this is enabled it makes more sense to split based on this. Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 6789b701..0044d7e8 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2854,6 +2854,23 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) struct e2fsck_thread_info *infos = NULL; int num_threads = 1; errcode_t retval; + unsigned flexbg_size = 1; + int max_threads; + + if (ext2fs_has_feature_flex_bg(global_ctx->fs->super)) + flexbg_size = 1 << global_ctx->fs->super->s_log_groups_per_flex; + + max_threads = global_ctx->fs->group_desc_count / flexbg_size; + if (max_threads == 0) + num_threads = 1; + else if (max_threads % num_threads) { + int times = max_threads / num_threads; + + if (times == 0) + num_threads = 1; + else + num_threads = max_threads / times; + } init_ext2_max_sizes(); retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); From patchwork Wed Apr 8 10:44:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267889 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=JML3aw99; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DD58zRz9sSb for ; Wed, 8 Apr 2020 20:46:36 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728281AbgDHKqg (ORCPT ); Wed, 8 Apr 2020 06:46:36 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:45222 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728278AbgDHKqf (ORCPT ); Wed, 8 Apr 2020 06:46:35 -0400 Received: by mail-pf1-f196.google.com with SMTP id r14so2217153pfl.12 for ; Wed, 08 Apr 2020 03:46:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8ZTvSriwJqZF9t50h+dKtDaZ6PR4kqR3n0K1agSL248=; b=JML3aw99UtbB5Mx8ELGFgLW4WL9G3JQsjKm2MW6l07QhuGczuimaggJYltDPg9jhmA UYKOzuZENO0vg2HcQTn4C70FYY6nnivRLzsN9lyZfBEv3ctcmvqspkBtyUoGbTGnJkDj SCeoXqr7oaRAIreJRCiTOI0vMY29Xl4ndinTfXNRfSP2UWhfmlz0P1tWIh8aPEYfC3OA TqzRfcXn9Di3vOtW6b6hlJoYFlQPFQbTnpRmHkaIKLLiUeIuRr6A1SegW6Ttugn/86Yf hPOmmn3tw/bjYt0A1zAQbHAFYSgBydIQ1h8d75lh7LVremx8AwbFfCVL9Kf6wD22X0OQ v41A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8ZTvSriwJqZF9t50h+dKtDaZ6PR4kqR3n0K1agSL248=; b=WyOccR4hh3G8iGEGfxDf+DQnhMwH6+nj1D8iU7e0dMBnLML45eE++c/exstqaHJMO7 5goI/iFwNrhNEm0OQQiCPcj2IzoxfskYW6jQJaFrLZwMEaMjqT32WfTSbpnW2eewtNrT noUNbXEeDDUfdQR51BbCFHLGyXuggqT00Yt220YXQaAaqXbirXmNOxmVpVlkjHYUTUUN +8DkrmIQbiMEFDwhSYvaiFi5zPPf1IyK7Gu//3vu2qtriJGDp8aFUs8l5OVAmSDp8azh iweVfgp1+/4Ms47+w1R+e8vUnUgwgx48JFqluw4AZYDLhYOHycg7nzdGnvZvM4XAwfHY mGDQ== X-Gm-Message-State: AGi0PuZn1J+5k94o7LZsTSNNtzyg6eYVptCsrZzRZCZ+7KmKjpaPF8k9 qXxPwtvB9hlf0nLFDXr/etCN3Zx0E8w= X-Google-Smtp-Source: APiQypKD8O7nTXyRi7x6BsUmZzaNqnsURXPV0Bzqr3ETsdvRPgiD2tAN8SMzztsCB21TfeiFWSxeOw== X-Received: by 2002:aa7:9a5d:: with SMTP id x29mr6697999pfj.284.1586342794930; Wed, 08 Apr 2020 03:46:34 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:34 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 29/46] e2fsck: merge dirs_to_hash when threads finish Date: Wed, 8 Apr 2020 19:44:57 +0900 Message-Id: <1586342714-12536-30-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong This will fix t_dangerous test failure with 2 threads. Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 0044d7e8..52598838 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2512,6 +2512,27 @@ static errcode_t e2fsck_pass1_merge_icounts(e2fsck_t global_ctx, e2fsck_t thread return 0; } +static int e2fsck_pass1_merge_dirs_to_hash(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + int retval = 0; + + if (thread_ctx->dirs_to_hash) { + if (!global_ctx->dirs_to_hash) + retval = ext2fs_badblocks_copy(thread_ctx->dirs_to_hash, + &global_ctx->dirs_to_hash); + else + retval = ext2fs_badblocks_merge(thread_ctx->dirs_to_hash, + global_ctx->dirs_to_hash); + + if (retval) + return retval; + + ext2fs_badblocks_list_free(thread_ctx->dirs_to_hash); + thread_ctx->dirs_to_hash = 0; + } + return retval; +} + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2553,6 +2574,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx __u32 large_files = global_ctx->large_files; int dx_dir_info_size = global_ctx->dx_dir_info_size; int dx_dir_info_count = global_ctx->dx_dir_info_count; + ext2_u32_list dirs_to_hash = global_ctx->dirs_to_hash; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2619,6 +2641,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx com_err(global_ctx->program_name, 0, _("while merging icounts\n")); return retval; } + global_ctx->dirs_to_hash = dirs_to_hash; + retval = e2fsck_pass1_merge_dirs_to_hash(global_ctx, thread_ctx); + if (retval) { + com_err(global_ctx->program_name, 0, _("while merging dirs to hash\n")); + return retval; + } /* * PASS1_COPY_CTX_BITMAP might return directly from this function, From patchwork Wed Apr 8 10:44:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267890 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Y0qr02YK; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DG38Yrz9sQx for ; Wed, 8 Apr 2020 20:46:38 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728284AbgDHKqi (ORCPT ); Wed, 8 Apr 2020 06:46:38 -0400 Received: from mail-pj1-f65.google.com ([209.85.216.65]:38942 "EHLO mail-pj1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728278AbgDHKqh (ORCPT ); Wed, 8 Apr 2020 06:46:37 -0400 Received: by mail-pj1-f65.google.com with SMTP id z3so997181pjr.4 for ; Wed, 08 Apr 2020 03:46:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5K19j3YqbknZ5V2CBpORy72AU7J3ZCigowjjlSslcyc=; b=Y0qr02YK8kEmMngukYuCn8cFGTtb9YXjEPlDDbq7ZeRljIu6wHeWLBanbmkZ/e0QVK hFvXk3aKxcihWJUGjWky3vCnXG2PVRzsD8SEj2/tqyeYhn5aLk2YwWroLL2df+QC/c4V Rx0lmAP/P+INpOzmIxpAxns5/LgJUzTecswcp3BOsbt5PesHN/3j+esT3S8HxkxLq4lZ p4diTSX2a7P7+FDyxEnSWoQVEX4WzCxb4MRJWkIw0uD/p8N61ms7I3WpLvgFdQ5eCGqI mv6oJsYDPiJN0bBga3N9+VvikwOxO+zH6m0zLM8Nh/aQp/wp+z2GFFvdvNPLk7JCAA4D fY3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5K19j3YqbknZ5V2CBpORy72AU7J3ZCigowjjlSslcyc=; b=JEfFI+Z02mwJd462efG3rdpOWqKO3fR3bvlKlyEdR3jMtoymKtPCGBfs6k79BHXI1w lFKLBKqkV1Cpxsv8GvbENCpjWdd7QfQOwdS2CBuUBPcXQlFJiBCI02jUnfun8bjziHbK B0h/0wlWfCwi6IO2c7kB0fK38KEGWdmYMzH+fGMg7imEnQRSq3Qupq99+tbsdyvOZfO/ F9mr24grtkX4Y/Ot9Evx+n3Mid4fuFFneZjI4oNrKOUlm0OjtUk1VN8XNC82XHR+krpO se9g9/1FE3zPe7l3V5wqopT1OWOBBlPa43TWprT03/xgYWW2TLyRAqxe7aLhLgNlFx8p h3Rw== X-Gm-Message-State: AGi0PuYC0QF3689PzV/9w81x7W5C3n/fzILd6q1ht6XUraCy+FpbAk30 3NpwQVPREmoGiuU6rVTlpZWifsWlz/w= X-Google-Smtp-Source: APiQypJfgVxTUZLCzI40DtjtgPhn2wmhYeqjY4C0Kqk5CTOu9Lnl0b12jWHgUcRFeor76gV6iuWp7Q== X-Received: by 2002:a17:90a:77cb:: with SMTP id e11mr4831677pjs.0.1586342796988; Wed, 08 Apr 2020 03:46:36 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:36 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 30/46] e2fsck: merge context flags properly Date: Wed, 8 Apr 2020 19:44:58 +0900 Message-Id: <1586342714-12536-31-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong e2fsck might restart after pass1, so we should keep flags if possible, this patch try to fix f_illitable_flexbg failure Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 52598838..eb102679 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2622,9 +2622,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fragmented_dir); PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, large_files); - /* Keep the global singal flags*/ - global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | - (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); + global_ctx->flags |= flags; retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); if (retval) { From patchwork Wed Apr 8 10:44:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267891 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=cFzp8UcN; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DK3BXSz9sSG for ; Wed, 8 Apr 2020 20:46:41 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728285AbgDHKql (ORCPT ); Wed, 8 Apr 2020 06:46:41 -0400 Received: from mail-pj1-f67.google.com ([209.85.216.67]:34093 "EHLO mail-pj1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKqk (ORCPT ); Wed, 8 Apr 2020 06:46:40 -0400 Received: by mail-pj1-f67.google.com with SMTP id q16so2071319pje.1 for ; Wed, 08 Apr 2020 03:46:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/3etSYCb49GIEUsMi08WiJoCwzgbOtVly/u/pEyahZI=; b=cFzp8UcNtVaIDfdyz4IiecdUmaGJ9a4XvIatZNG+eBymS8WAKou0asBh9RBGdAgoHQ 4fGoKUr9mYkv+R3GIS1PNwF2vdA4t8gEREUlMIRxVKrshRO8sKSzftCX9wuTWdW1m3Ta bW1MDGDBDeOBPo8uOdvfgcY3NIg87ntuQsm9gcK/ui8VGNu13i6jA6rF8B42booF+Q3a Ct9NtJ1uaiUC5M8Gm40e7l6YXh28kWLBZJJDAiSjHDhmnvc9462mS4O6eeeo8EFf2zhz fOO4nZ4ZpReuoEYVohUa8caJWNAB4ImAnr93xxmyoXWKtnk7y03GEVKp+IGI438HNCpu Q5gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/3etSYCb49GIEUsMi08WiJoCwzgbOtVly/u/pEyahZI=; b=af6hoj58kV++rfsGiVUcNsplyobPDN1U4ZKyg/gDHtLt0J2sw2gfrZpx2R94PHhpji HYACIFLZdag0PpvrY/d7ciUVy3REagWpqYvHMlKZNEkIejtPrQIZ2fF01csmvtfUvTi2 T4daHmWWZjUb2hv998n/gDTqyD6FtF0bpQ+W0PBjm84tKLqwaLmNNRZlapZLi8gxxL14 CiapIsL/2UheGQBNkolLhRfEc8pMWXXcgJ6nEFERif1Ideyf+yn/Td9JIuPoh1KozvRx xIdJyRzgs2JLA1gvC1VMvhKlnwWlGUbZtbtuK5g9VZsppaFpzpbA3b89bKaFsu9PI3Aa 94ow== X-Gm-Message-State: AGi0PuZG8hjgNwIXUwaOEb57xwI85FMpSEL+J2B2lrxlZEsB0agc9RVb 5+rdOqunokozOmxZQrBb6lfkqrcQQqE= X-Google-Smtp-Source: APiQypLK1aG8hGol98gumfQH1T6sKnOMIeU05ByVdr5kHyEC2Dr7jur/XX2P5jys/2R+f3zV8mzoDw== X-Received: by 2002:a17:902:eb13:: with SMTP id l19mr6605721plb.156.1586342799064; Wed, 08 Apr 2020 03:46:39 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:38 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 31/46] e2fsck: split and merge quota context Date: Wed, 8 Apr 2020 19:44:59 +0900 Message-Id: <1586342714-12536-32-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Every threads calculate its own quota accounting, merge them after threads finish. Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 23 +++++++++++++++++++++++ lib/support/mkquota.c | 19 +++++++++++++++++++ lib/support/quotaio.h | 2 ++ 3 files changed, 44 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index eb102679..c7b9cf72 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -56,6 +56,7 @@ #include #include "problem.h" +#include "support/dict.h" #ifdef NO_INLINE_FUNCS #define _INLINE_ @@ -2447,6 +2448,11 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, log_out(thread_context, _("Scan group range [%d, %d)\n"), tinfo->et_group_start, tinfo->et_group_end); thread_context->fs = thread_fs; + retval = quota_init_context(&thread_context->qctx, thread_fs, 0); + if (retval) { + com_err(global_ctx->program_name, retval, "while init quota context"); + goto out_fs; + } *thread_ctx = thread_context; return 0; out_fs: @@ -2533,6 +2539,20 @@ static int e2fsck_pass1_merge_dirs_to_hash(e2fsck_t global_ctx, e2fsck_t thread_ return retval; } +static void e2fsck_pass1_merge_quota_ctx(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + dict_t *dict; + enum quota_type qtype; + + for (qtype = 0; qtype < MAXQUOTAS; qtype++) { + dict = thread_ctx->qctx->quota_dict[qtype]; + if (dict) + quota_merge_and_update_usage( + global_ctx->qctx->quota_dict[qtype], dict); + } + quota_release_context(&thread_ctx->qctx); +} + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2575,6 +2595,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx int dx_dir_info_size = global_ctx->dx_dir_info_size; int dx_dir_info_count = global_ctx->dx_dir_info_count; ext2_u32_list dirs_to_hash = global_ctx->dirs_to_hash; + quota_ctx_t qctx = global_ctx->qctx; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2645,6 +2666,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx com_err(global_ctx->program_name, 0, _("while merging dirs to hash\n")); return retval; } + global_ctx->qctx = qctx; + e2fsck_pass1_merge_quota_ctx(global_ctx, thread_ctx); /* * PASS1_COPY_CTX_BITMAP might return directly from this function, diff --git a/lib/support/mkquota.c b/lib/support/mkquota.c index 6f7ae6d6..745106b0 100644 --- a/lib/support/mkquota.c +++ b/lib/support/mkquota.c @@ -639,6 +639,25 @@ out: return err; } +errcode_t quota_merge_and_update_usage(dict_t *dest, dict_t *src) +{ + dnode_t *n; + struct dquot *src_dq, *dest_dq; + + for (n = dict_first(src); n; n = dict_next(src, n)) { + src_dq = dnode_get(n); + if (!src_dq) + continue; + dest_dq = get_dq(dest, src_dq->dq_id); + if (dest_dq == NULL) + return -ENOMEM; + dest_dq->dq_dqb.dqb_curspace += src_dq->dq_dqb.dqb_curspace; + dest_dq->dq_dqb.dqb_curinodes += src_dq->dq_dqb.dqb_curinodes; + } + + return 0; +} + /* * Compares the measured quota in qctx->quota_dict with that in the quota inode * on disk and updates the limits in qctx->quota_dict. 'usage_inconsistent' is diff --git a/lib/support/quotaio.h b/lib/support/quotaio.h index 60689700..6077268a 100644 --- a/lib/support/quotaio.h +++ b/lib/support/quotaio.h @@ -40,6 +40,7 @@ #include "ext2fs/ext2_fs.h" #include "ext2fs/ext2fs.h" #include "dqblk_v2.h" +#include "support/dict.h" typedef int64_t qsize_t; /* Type in which we store size limitations */ @@ -233,6 +234,7 @@ int quota_file_exists(ext2_filsys fs, enum quota_type qtype); void quota_set_sb_inum(ext2_filsys fs, ext2_ino_t ino, enum quota_type qtype); errcode_t quota_compare_and_update(quota_ctx_t qctx, enum quota_type qtype, int *usage_inconsistent); +errcode_t quota_merge_and_update_usage(dict_t *dest, dict_t *src); int parse_quota_opts(const char *opts, int (*func)(char *)); /* parse_qtype.c */ From patchwork Wed Apr 8 10:45:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267892 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=sj5VTPoR; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DN3xYlz9sQx for ; Wed, 8 Apr 2020 20:46:44 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728287AbgDHKqo (ORCPT ); Wed, 8 Apr 2020 06:46:44 -0400 Received: from mail-pg1-f193.google.com ([209.85.215.193]:34607 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKqo (ORCPT ); Wed, 8 Apr 2020 06:46:44 -0400 Received: by mail-pg1-f193.google.com with SMTP id l14so3166543pgb.1 for ; Wed, 08 Apr 2020 03:46:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wqEZgfLU5Xq6IY63Buss+q7miZSgJx5KNKvmjxetdkU=; b=sj5VTPoRhdT6vF0q8vKBcnSd5iMeKfpihb1wSvWSDgFax8hR7rLJSdRSwBuYu4cS7K txAFzTTa46EvrGy7wINTDDlPik0QiKxzC5SH67bIKbX2q0TlG1G6fUI1xxWVtyToz9Ex 3E1WyN/oWq1jr2i7z7qzGcilNXDM9h8mNQd4UtNW2JYeSLbKxbIOcpH/wlPYIlksdk7e ycKYTqMQaSE2Oll0wKGvtsX857rR2ehS4CBSAprq/hy+2+QIzLu5bSNXuKtLFa4AAsaj uvfyzufHfa6I95b4qHGugOEjClEjNoUGWYNXWABCdazBEs5PIzVzt5ODGGzNLnVH7wKV RdAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wqEZgfLU5Xq6IY63Buss+q7miZSgJx5KNKvmjxetdkU=; b=E+eleuxd7YP5P1VgKyvPmCZaUeSw+svq8Da0dO/ZntldLF2kRSRWBTCxbjfUeb6QDJ cknr26l/0aNBEKh1bsypUOoDD0nU8O95Bur1EyT0qQMY0lNS0CvvSL4wQMn/O2D4NI+Y BVDvuhumxTLXqEwXUR96ePxGHmZzcwn3illjB+pwk8LATLOCN+t+AaPN3AmY9GAtPh9w PmS7QYpfzxoFQgB/gpRLKHTiWaX3wLdCohdASsLwOJ/eWHkemkEcYZH9dUpD4Wesoz1T 7PVapN5HZjuzjlN2QparObHYwKfg8fO9YjoHj6IOipic1lGkZC8SjMT6Q3cq7E7bCjdh tZJg== X-Gm-Message-State: AGi0PuYst6Tmo1AwbBvP11WXSdZDTmaPMeYBWqwQLgInJ7IUcaG/ZIHW CfvoYC9TKPpPCWZbh654NEy8fdm0o0Y= X-Google-Smtp-Source: APiQypKv54uNQYr/8Lpbwovpy8vcP19ptqaNMIzWlA5v65vbjix2RfAkzQGnIawQ66Nvxtf+Ngwu/g== X-Received: by 2002:a63:2254:: with SMTP id t20mr6558245pgm.121.1586342801300; Wed, 08 Apr 2020 03:46:41 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:40 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 32/46] e2fsck: serialize fix operations Date: Wed, 8 Apr 2020 19:45:00 +0900 Message-Id: <1586342714-12536-33-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Allow different threads to fix at the same time could be dangerous and eror-prone now, and most of time parallel scanning and checking is important. So this patch try to add a mutex to serialize fix operations during pass1. And the good benefit of this, we don't need block allocations and free, superblock updates protection any more, since only fix operations during pass1 could touch them. Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 3 +- e2fsck/pass1.c | 164 +++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 156 insertions(+), 11 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 0b449b69..20d61651 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -448,7 +448,8 @@ struct e2fsck_struct { __u32 fs_ext_attr_inodes; __u32 fs_ext_attr_blocks; __u32 extent_depth_count[MAX_EXTENT_DEPTH_COUNT]; - + /* serialize fix operation for multiple threads */ + pthread_mutex_t fs_fix_mutex; }; #ifdef DEBUG_THREADS diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index c7b9cf72..f3b52103 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -132,6 +132,24 @@ static void process_inodes(e2fsck_t ctx, char *block_buf, static __u64 ext2_max_sizes[EXT2_MAX_BLOCK_LOG_SIZE - EXT2_MIN_BLOCK_LOG_SIZE + 1]; +static void e2fsck_pass1_fix_lock(e2fsck_t ctx) +{ + e2fsck_t global_ctx = ctx->global_ctx; + if (!global_ctx) + global_ctx = ctx; + + pthread_mutex_lock(&global_ctx->fs_fix_mutex); +} + +static void e2fsck_pass1_fix_unlock(e2fsck_t ctx) +{ + e2fsck_t global_ctx = ctx->global_ctx; + if (!global_ctx) + global_ctx = ctx; + + pthread_mutex_unlock(&global_ctx->fs_fix_mutex); +} + /* * Free all memory allocated by pass1 in preparation for restarting * things. @@ -282,8 +300,10 @@ static void check_extents_inlinedata(e2fsck_t ctx, if (!fix_problem(ctx, PR_1_SPECIAL_EXTENTS_IDATA, pctx)) return; + e2fsck_pass1_fix_lock(ctx); pctx->inode->i_flags &= ~BAD_SPECIAL_FLAGS; e2fsck_write_inode(ctx, pctx->ino, pctx->inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); } #undef BAD_SPECIAL_FLAGS @@ -300,8 +320,10 @@ static void check_immutable(e2fsck_t ctx, struct problem_context *pctx) if (!fix_problem(ctx, PR_1_SET_IMMUTABLE, pctx)) return; + e2fsck_pass1_fix_lock(ctx); pctx->inode->i_flags &= ~BAD_SPECIAL_FLAGS; e2fsck_write_inode(ctx, pctx->ino, pctx->inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); } /* @@ -318,8 +340,10 @@ static void check_size(e2fsck_t ctx, struct problem_context *pctx) if (!fix_problem(ctx, PR_1_SET_NONZSIZE, pctx)) return; + e2fsck_pass1_fix_lock(ctx); ext2fs_inode_size_set(ctx->fs, inode, 0); e2fsck_write_inode(ctx, pctx->ino, pctx->inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); } /* @@ -388,9 +412,11 @@ static problem_t check_large_ea_inode(e2fsck_t ctx, if (!(inode.i_flags & EXT4_EA_INODE_FL)) { pctx->num = entry->e_value_inum; if (fix_problem(ctx, PR_1_ATTR_SET_EA_INODE_FL, pctx)) { + e2fsck_pass1_fix_lock(ctx); inode.i_flags |= EXT4_EA_INODE_FL; ext2fs_write_inode(ctx->fs, entry->e_value_inum, &inode); + e2fsck_pass1_fix_unlock(ctx); } else { return PR_1_ATTR_NO_EA_INODE_FL; } @@ -551,11 +577,13 @@ fix: } /* simply remove all possible EA(s) */ + e2fsck_pass1_fix_lock(ctx); *((__u32 *)header) = 0UL; e2fsck_write_inode_full(ctx, pctx->ino, pctx->inode, EXT2_INODE_SIZE(sb), "pass1"); ea_ibody_quota->blocks = 0; ea_ibody_quota->inodes = 0; + e2fsck_pass1_fix_unlock(ctx); } static int check_inode_extra_negative_epoch(__u32 xtime, __u32 extra) { @@ -606,12 +634,14 @@ static void check_inode_extra_space(e2fsck_t ctx, struct problem_context *pctx, inode->i_extra_isize & 3)) { if (!fix_problem(ctx, PR_1_EXTRA_ISIZE, pctx)) return; + e2fsck_pass1_fix_lock(ctx); if (inode->i_extra_isize < min || inode->i_extra_isize > max) inode->i_extra_isize = sb->s_want_extra_isize; else inode->i_extra_isize = (inode->i_extra_isize + 3) & ~3; e2fsck_write_inode_full(ctx, pctx->ino, pctx->inode, EXT2_INODE_SIZE(sb), "pass1"); + e2fsck_pass1_fix_unlock(ctx); } /* check if there is no place for an EA header */ @@ -640,6 +670,7 @@ static void check_inode_extra_space(e2fsck_t ctx, struct problem_context *pctx, if (!fix_problem(ctx, PR_1_EA_TIME_OUT_OF_RANGE, pctx)) return; + e2fsck_pass1_fix_lock(ctx); if (CHECK_INODE_EXTRA_NEGATIVE_EPOCH(inode, atime)) inode->i_atime_extra &= ~EXT4_EPOCH_MASK; if (CHECK_INODE_EXTRA_NEGATIVE_EPOCH(inode, ctime)) @@ -650,6 +681,7 @@ static void check_inode_extra_space(e2fsck_t ctx, struct problem_context *pctx, inode->i_mtime_extra &= ~EXT4_EPOCH_MASK; e2fsck_write_inode_full(ctx, pctx->ino, pctx->inode, EXT2_INODE_SIZE(sb), "pass1"); + e2fsck_pass1_fix_unlock(ctx); } } @@ -807,10 +839,12 @@ static void check_is_really_dir(e2fsck_t ctx, struct problem_context *pctx, isdir: if (fix_problem(ctx, PR_1_TREAT_AS_DIRECTORY, pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_mode = (inode->i_mode & 07777) | LINUX_S_IFDIR; e2fsck_write_inode_full(ctx, pctx->ino, inode, EXT2_INODE_SIZE(ctx->fs->super), "check_is_really_dir"); + e2fsck_pass1_fix_unlock(ctx); } } @@ -881,8 +915,11 @@ static errcode_t recheck_bad_inode_checksum(ext2_filsys fs, ext2_ino_t ino, if (!fix_problem(ctx, PR_1_INODE_ONLY_CSUM_INVALID, pctx)) return 0; + + e2fsck_pass1_fix_lock(ctx); retval = ext2fs_write_inode_full(fs, ino, (struct ext2_inode *)&inode, sizeof(inode)); + e2fsck_pass1_fix_unlock(ctx); return retval; } @@ -892,15 +929,19 @@ static void reserve_block_for_root_repair(e2fsck_t ctx) errcode_t err; ext2_filsys fs = ctx->fs; + e2fsck_pass1_fix_lock(ctx); ctx->root_repair_block = 0; if (ext2fs_test_inode_bitmap2(ctx->inode_used_map, EXT2_ROOT_INO)) - return; + goto out; err = ext2fs_new_block2(fs, 0, ctx->block_found_map, &blk); if (err) - return; + goto out; ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); ctx->root_repair_block = blk; +out: + e2fsck_pass1_fix_unlock(ctx); + return; } static void reserve_block_for_lnf_repair(e2fsck_t ctx) @@ -911,15 +952,19 @@ static void reserve_block_for_lnf_repair(e2fsck_t ctx) static const char name[] = "lost+found"; ext2_ino_t ino; + e2fsck_pass1_fix_lock(ctx); ctx->lnf_repair_block = 0; if (!ext2fs_lookup(fs, EXT2_ROOT_INO, name, sizeof(name)-1, 0, &ino)) - return; + goto out; err = ext2fs_new_block2(fs, 0, ctx->block_found_map, &blk); if (err) - return; + goto out; ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); ctx->lnf_repair_block = blk; +out: + e2fsck_pass1_fix_unlock(ctx); + return; } static errcode_t get_inline_data_ea_size(ext2_filsys fs, ext2_ino_t ino, @@ -1018,8 +1063,10 @@ static int fix_inline_data_extents_file(e2fsck_t ctx, if (ext2fs_extent_header_verify(inode->i_block, sizeof(inode->i_block)) == 0 && fix_problem(ctx, PR_1_CLEAR_INLINE_DATA_FOR_EXTENT, pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_flags &= ~EXT4_INLINE_DATA_FL; dirty = 1; + e2fsck_pass1_fix_unlock(ctx); goto out; } @@ -1033,8 +1080,10 @@ static int fix_inline_data_extents_file(e2fsck_t ctx, if (EXT2_I_SIZE(inode) < EXT4_MIN_INLINE_DATA_SIZE + max_inline_ea_size && fix_problem(ctx, PR_1_CLEAR_EXTENT_FOR_INLINE_DATA, pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_flags &= ~EXT4_EXTENTS_FL; dirty = 1; + e2fsck_pass1_fix_unlock(ctx); goto out; } @@ -1044,6 +1093,7 @@ static int fix_inline_data_extents_file(e2fsck_t ctx, */ if (could_be_block_map(fs, inode) && fix_problem(ctx, PR_1_CLEAR_EXTENT_INLINE_DATA_FLAGS, pctx)) { + e2fsck_pass1_fix_lock(ctx); #ifdef WORDS_BIGENDIAN int i; @@ -1053,18 +1103,24 @@ static int fix_inline_data_extents_file(e2fsck_t ctx, inode->i_flags &= ~(EXT4_EXTENTS_FL | EXT4_INLINE_DATA_FL); dirty = 1; + e2fsck_pass1_fix_unlock(ctx); goto out; } /* Oh well, just clear the busted inode. */ if (fix_problem(ctx, PR_1_CLEAR_EXTENT_INLINE_DATA_INODE, pctx)) { + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, 0, "pass1"); + e2fsck_pass1_fix_unlock(ctx); return -1; } out: - if (dirty) + if (dirty) { + e2fsck_pass1_fix_lock(ctx); e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); + } return 0; } @@ -1408,7 +1464,9 @@ void _e2fsck_pass1(e2fsck_t ctx) &pctx)) { errcode_t err; + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, 0, "pass1"); + e2fsck_pass1_fix_unlock(ctx); ext2fs_badblocks_list_free(ctx->fs->badblocks); ctx->fs->badblocks = NULL; err = ext2fs_read_bb_inode(ctx->fs, @@ -1457,7 +1515,9 @@ void _e2fsck_pass1(e2fsck_t ctx) inode->i_links_count > 0 && fix_problem(ctx, PR_1_INODE_IS_GARBAGE, &pctx)) { pctx.errcode = 0; + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, 0, "pass1"); + e2fsck_pass1_fix_unlock(ctx); } failed_csum = pctx.errcode != 0; @@ -1481,10 +1541,12 @@ void _e2fsck_pass1(e2fsck_t ctx) if (inode->i_dtime && low_dtime_check && inode->i_dtime < ctx->fs->super->s_inodes_count) { if (fix_problem(ctx, PR_1_LOW_DTIME, &pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_dtime = inode->i_links_count ? 0 : ctx->now; e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } } @@ -1503,9 +1565,11 @@ void _e2fsck_pass1(e2fsck_t ctx) if (!inode->i_dtime && inode->i_mode) { if (fix_problem(ctx, PR_1_ZERO_DTIME, &pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_dtime = ctx->now; e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } } @@ -1542,11 +1606,15 @@ void _e2fsck_pass1(e2fsck_t ctx) pctx.errcode = get_inline_data_ea_size(fs, ino, &size); if (!pctx.errcode && fix_problem(ctx, PR_1_INLINE_DATA_FEATURE, &pctx)) { + e2fsck_pass1_fix_lock(ctx); ext2fs_set_feature_inline_data(sb); ext2fs_mark_super_dirty(fs); + e2fsck_pass1_fix_unlock(ctx); inlinedata_fs = 1; } else if (fix_problem(ctx, PR_1_INLINE_DATA_SET, &pctx)) { + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, 0, "pass1"); + e2fsck_pass1_fix_unlock(ctx); /* skip FINISH_INODE_LOOP */ continue; } @@ -1588,10 +1656,12 @@ void _e2fsck_pass1(e2fsck_t ctx) /* broken EA or no system.data EA; truncate */ if (fix_problem(ctx, PR_1_INLINE_DATA_NO_ATTR, &pctx)) { + e2fsck_pass1_fix_lock(ctx); err = ext2fs_inode_size_set(fs, inode, 0); if (err) { pctx.errcode = err; ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_fix_unlock(ctx); goto endit; } inode->i_flags &= ~EXT4_INLINE_DATA_FL; @@ -1599,6 +1669,7 @@ void _e2fsck_pass1(e2fsck_t ctx) sizeof(inode->i_block)); e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } break; @@ -1631,12 +1702,16 @@ void _e2fsck_pass1(e2fsck_t ctx) if ((ext2fs_extent_header_verify(inode->i_block, sizeof(inode->i_block)) == 0) && fix_problem(ctx, PR_1_EXTENT_FEATURE, &pctx)) { + e2fsck_pass1_fix_lock(ctx); ext2fs_set_feature_extents(sb); ext2fs_mark_super_dirty(fs); extent_fs = 1; + e2fsck_pass1_fix_unlock(ctx); } else if (fix_problem(ctx, PR_1_EXTENTS_SET, &pctx)) { clear_inode: + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, 0, "pass1"); + e2fsck_pass1_fix_unlock(ctx); if (ino == EXT2_BAD_INO) ext2fs_mark_inode_bitmap2(ctx->inode_used_map, ino); @@ -1671,12 +1746,14 @@ void _e2fsck_pass1(e2fsck_t ctx) if ((ext2fs_extent_header_verify(ehp, sizeof(inode->i_block)) == 0) && (fix_problem(ctx, PR_1_UNSET_EXTENT_FL, &pctx))) { + e2fsck_pass1_fix_lock(ctx); inode->i_flags |= EXT4_EXTENTS_FL; #ifdef WORDS_BIGENDIAN memcpy(inode->i_block, tmp_block, sizeof(inode->i_block)); #endif e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } } @@ -1689,9 +1766,11 @@ void _e2fsck_pass1(e2fsck_t ctx) (inode->i_flags & EXT4_INLINE_DATA_FL) || inode->i_file_acl) && fix_problem(ctx, PR_1_INVALID_BAD_INODE, &pctx)) { + e2fsck_pass1_fix_lock(ctx); memset(inode, 0, sizeof(struct ext2_inode)); e2fsck_write_inode(ctx, ino, inode, "clear bad inode"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } @@ -1750,9 +1829,11 @@ void _e2fsck_pass1(e2fsck_t ctx) */ if (inode->i_dtime && inode->i_links_count) { if (fix_problem(ctx, PR_1_ROOT_DTIME, &pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_dtime = 0; e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } } @@ -1762,9 +1843,11 @@ void _e2fsck_pass1(e2fsck_t ctx) if (!LINUX_S_ISREG(inode->i_mode) && fix_problem(ctx, PR_1_JOURNAL_BAD_MODE, &pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_mode = LINUX_S_IFREG; e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } check_blocks(ctx, &pctx, block_buf, NULL); @@ -1778,8 +1861,10 @@ void _e2fsck_pass1(e2fsck_t ctx) memset(inode, 0, inode_size); ext2fs_icount_store(ctx->inode_link_info, ino, 0); + e2fsck_pass1_fix_lock(ctx); e2fsck_write_inode_full(ctx, ino, inode, inode_size, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } } else if (quota_inum_is_reserved(fs, ino)) { @@ -1789,9 +1874,11 @@ void _e2fsck_pass1(e2fsck_t ctx) if (!LINUX_S_ISREG(inode->i_mode) && fix_problem(ctx, PR_1_QUOTA_BAD_MODE, &pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_mode = LINUX_S_IFREG; e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } check_blocks(ctx, &pctx, block_buf, NULL); @@ -1802,11 +1889,13 @@ void _e2fsck_pass1(e2fsck_t ctx) inode->i_blocks || inode->i_block[0]) && fix_problem(ctx, PR_1_QUOTA_INODE_NOT_CLEAR, &pctx)) { - memset(inode, 0, inode_size); ext2fs_icount_store(ctx->inode_link_info, ino, 0); + e2fsck_pass1_fix_lock(ctx); + memset(inode, 0, inode_size); e2fsck_write_inode_full(ctx, ino, inode, inode_size, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } } else if (ino < EXT2_FIRST_INODE(fs->super)) { @@ -1826,9 +1915,11 @@ void _e2fsck_pass1(e2fsck_t ctx) } if (problem) { if (fix_problem(ctx, problem, &pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_mode = 0; e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } } @@ -1889,9 +1980,11 @@ void _e2fsck_pass1(e2fsck_t ctx) ino); } else { if (fix_problem(ctx, PR_1_SET_IMAGIC, &pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_flags &= ~EXT2_IMAGIC_FL; e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } } @@ -1908,8 +2001,10 @@ void _e2fsck_pass1(e2fsck_t ctx) LINUX_S_ISLNK(inode->i_mode) && !ext2fs_inode_has_valid_blocks2(fs, inode) && fix_problem(ctx, PR_1_FAST_SYMLINK_EXTENT_FL, &pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_flags &= ~EXT4_EXTENTS_FL; e2fsck_write_inode(ctx, ino, inode, "pass1"); + e2fsck_pass1_fix_unlock(ctx); failed_csum = 0; } @@ -2042,8 +2137,11 @@ void _e2fsck_pass1(e2fsck_t ctx) ctx->ea_block_quota_inodes = 0; } - if (ctx->invalid_bitmaps) + if (ctx->invalid_bitmaps) { + e2fsck_pass1_fix_lock(ctx); handle_fs_bad_blocks(ctx); + e2fsck_pass1_fix_unlock(ctx); + } /* We don't need the block_ea_map any more */ if (ctx->block_ea_map) { @@ -2056,7 +2154,9 @@ void _e2fsck_pass1(e2fsck_t ctx) if (ctx->flags & E2F_FLAG_RESIZE_INODE) { clear_problem_context(&pctx); + e2fsck_pass1_fix_lock(ctx); pctx.errcode = ext2fs_create_resize_inode(fs); + e2fsck_pass1_fix_unlock(ctx); if (pctx.errcode) { if (!fix_problem(ctx, PR_1_RESIZE_INODE_CREATE, &pctx)) { @@ -2068,9 +2168,11 @@ void _e2fsck_pass1(e2fsck_t ctx) if (!pctx.errcode) { e2fsck_read_inode(ctx, EXT2_RESIZE_INO, inode, "recreate inode"); + e2fsck_pass1_fix_lock(ctx); inode->i_mtime = ctx->now; e2fsck_write_inode(ctx, EXT2_RESIZE_INO, inode, "recreate inode"); + e2fsck_pass1_fix_unlock(ctx); } ctx->flags &= ~E2F_FLAG_RESIZE_INODE; } @@ -2092,7 +2194,9 @@ void _e2fsck_pass1(e2fsck_t ctx) clear_problem_context(&pctx); fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); } + e2fsck_pass1_fix_lock(ctx); e2fsck_pass1_dupblocks(ctx, block_buf); + e2fsck_pass1_fix_unlock(ctx); } ctx->flags |= E2F_FLAG_ALLOC_OK; ext2fs_free_mem(&inodes_to_process); @@ -2906,6 +3010,7 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) unsigned flexbg_size = 1; int max_threads; + pthread_mutex_init(&global_ctx->fs_fix_mutex, NULL); if (ext2fs_has_feature_flex_bg(global_ctx->fs->super)) flexbg_size = 1 << global_ctx->fs->super->s_log_groups_per_flex; @@ -3210,10 +3315,12 @@ static void adjust_extattr_refcount(e2fsck_t ctx, ext2_refcount_t refcount, should_be = header->h_refcount + adjust_sign * (int)count; pctx.num = should_be; if (fix_problem(ctx, PR_1_EXTATTR_REFCOUNT, &pctx)) { + e2fsck_pass1_fix_lock(ctx); header->h_refcount = should_be; pctx.errcode = ext2fs_write_ext_attr3(fs, blk, block_buf, pctx.ino); + e2fsck_pass1_fix_unlock(ctx); if (pctx.errcode) { fix_problem(ctx, PR_1_EXTATTR_WRITE_ABORT, &pctx); @@ -3442,8 +3549,10 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, */ if (failed_csum && fix_problem(ctx, PR_1_EA_BLOCK_ONLY_CSUM_INVALID, pctx)) { + e2fsck_pass1_fix_lock(ctx); pctx->errcode = ext2fs_write_ext_attr3(fs, blk, block_buf, pctx->ino); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) return 0; } @@ -3489,8 +3598,10 @@ refcount_fail: clear_extattr: if (region) region_free(region); + e2fsck_pass1_fix_lock(ctx); ext2fs_file_acl_block_set(fs, inode, 0); e2fsck_write_inode(ctx, ino, inode, "check_ext_attr"); + e2fsck_pass1_fix_unlock(ctx); return 0; } @@ -3720,10 +3831,12 @@ static void scan_extent_node(e2fsck_t ctx, struct problem_context *pctx, if (try_repairs && is_dir && problem == 0 && (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) && fix_problem(ctx, PR_1_UNINIT_DBLOCK, pctx)) { + e2fsck_pass1_fix_lock(ctx); extent.e_flags &= ~EXT2_EXTENT_FLAGS_UNINIT; pb->inode_modified = 1; pctx->errcode = ext2fs_extent_replace(ehandle, 0, &extent); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) return; failed_csum = 0; @@ -3767,13 +3880,17 @@ report_problem: } e2fsck_read_bitmaps(ctx); pb->inode_modified = 1; + e2fsck_pass1_fix_lock(ctx); pctx->errcode = ext2fs_extent_delete(ehandle, 0); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) { pctx->str = "ext2fs_extent_delete"; return; } + e2fsck_pass1_fix_lock(ctx); pctx->errcode = ext2fs_extent_fix_parents(ehandle); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode && pctx->errcode != EXT2_ET_NO_CURRENT_NODE) { pctx->str = "ext2fs_extent_fix_parents"; @@ -3837,9 +3954,11 @@ report_problem: pctx->num = e_info.curr_level - 1; problem = PR_1_EXTENT_INDEX_START_INVALID; if (fix_problem(ctx, problem, pctx)) { + e2fsck_pass1_fix_lock(ctx); pb->inode_modified = 1; pctx->errcode = ext2fs_extent_fix_parents(ehandle); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) { pctx->str = "ext2fs_extent_fix_parents"; return; @@ -3903,15 +4022,19 @@ report_problem: pctx->blk = extent.e_lblk; pctx->blk2 = new_lblk; if (fix_problem(ctx, PR_1_COLLAPSE_DBLOCK, pctx)) { + e2fsck_pass1_fix_lock(ctx); extent.e_lblk = new_lblk; pb->inode_modified = 1; pctx->errcode = ext2fs_extent_replace(ehandle, 0, &extent); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) { pctx->errcode = 0; goto alloc_later; } + e2fsck_pass1_fix_lock(ctx); pctx->errcode = ext2fs_extent_fix_parents(ehandle); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) goto failed_add_dir_block; pctx->errcode = ext2fs_extent_goto(ehandle, @@ -4007,8 +4130,10 @@ alloc_later: /* Failed csum but passes checks? Ask to fix checksum. */ if (failed_csum && fix_problem(ctx, PR_1_EXTENT_ONLY_CSUM_INVALID, pctx)) { + e2fsck_pass1_fix_lock(ctx); pb->inode_modified = 1; pctx->errcode = ext2fs_extent_replace(ehandle, 0, &extent); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) return; } @@ -4033,9 +4158,12 @@ static void check_blocks_extents(e2fsck_t ctx, struct problem_context *pctx, eh = (struct ext3_extent_header *) &inode->i_block[0]; retval = ext2fs_extent_header_verify(eh, sizeof(inode->i_block)); if (retval) { - if (fix_problem(ctx, PR_1_MISSING_EXTENT_HEADER, pctx)) + if (fix_problem(ctx, PR_1_MISSING_EXTENT_HEADER, pctx)) { + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, 0, "check_blocks_extents"); + e2fsck_pass1_fix_unlock(ctx); + } pctx->errcode = 0; return; } @@ -4043,9 +4171,12 @@ static void check_blocks_extents(e2fsck_t ctx, struct problem_context *pctx, /* ...since this function doesn't fail if i_block is zeroed. */ pctx->errcode = ext2fs_extent_open2(fs, ino, inode, &ehandle); if (pctx->errcode) { - if (fix_problem(ctx, PR_1_READ_EXTENT, pctx)) + if (fix_problem(ctx, PR_1_READ_EXTENT, pctx)) { + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, 0, "check_blocks_extents"); + e2fsck_pass1_fix_unlock(ctx); + } pctx->errcode = 0; return; } @@ -4082,8 +4213,10 @@ static void check_blocks_extents(e2fsck_t ctx, struct problem_context *pctx, fix_problem(ctx, PR_1_EXTENT_ITERATE_FAILURE, pctx)) { pb->num_blocks = 0; inode->i_blocks = 0; + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, E2F_FLAG_RESTART, "check_blocks_extents"); + e2fsck_pass1_fix_unlock(ctx); pctx->errcode = 0; } ext2fs_extent_free(ehandle); @@ -4259,8 +4392,10 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, } if (pb.clear) { + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, E2F_FLAG_RESTART, "check_blocks"); + e2fsck_pass1_fix_unlock(ctx); return; } @@ -4276,7 +4411,9 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, if (!pb.num_blocks && pb.is_dir && !(inode->i_flags & EXT4_INLINE_DATA_FL)) { if (fix_problem(ctx, PR_1_ZERO_LENGTH_DIR, pctx)) { + e2fsck_pass1_fix_lock(ctx); e2fsck_clear_inode(ctx, ino, inode, 0, "check_blocks"); + e2fsck_pass1_fix_unlock(ctx); ctx->fs_directory_count--; return; } @@ -4355,6 +4492,7 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, pctx->num = (pb.last_block + 1) * fs->blocksize; pctx->group = bad_size; if (fix_problem(ctx, PR_1_BAD_I_SIZE, pctx)) { + e2fsck_pass1_fix_lock(ctx); if (LINUX_S_ISDIR(inode->i_mode)) pctx->num &= 0xFFFFFFFFULL; ext2fs_inode_size_set(fs, inode, pctx->num); @@ -4365,6 +4503,7 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, inode->i_flags &= ~EXT4_INLINE_DATA_FL; } dirty_inode++; + e2fsck_pass1_fix_unlock(ctx); } pctx->num = 0; } @@ -4378,8 +4517,10 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, (inode->osd2.linux2.l_i_blocks_hi != 0)))) { pctx->num = pb.num_blocks; if (fix_problem(ctx, PR_1_BAD_I_BLOCKS, pctx)) { + e2fsck_pass1_fix_lock(ctx); inode->i_blocks = pb.num_blocks; inode->osd2.linux2.l_i_blocks_hi = pb.num_blocks >> 32; + e2fsck_pass1_fix_unlock(ctx); dirty_inode++; } pctx->num = 0; @@ -4408,8 +4549,11 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, e2fsck_rehash_dir_later(ctx, ino); out: - if (dirty_inode) + if (dirty_inode) { + e2fsck_pass1_fix_lock(ctx); e2fsck_write_inode(ctx, ino, inode, "check_blocks"); + e2fsck_pass1_fix_unlock(ctx); + } } #if 0 From patchwork Wed Apr 8 10:45:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267893 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=WMS8hY4K; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DR3T0Zz9sQx for ; Wed, 8 Apr 2020 20:46:47 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728100AbgDHKqr (ORCPT ); Wed, 8 Apr 2020 06:46:47 -0400 Received: from mail-pj1-f65.google.com ([209.85.216.65]:34095 "EHLO mail-pj1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727026AbgDHKqq (ORCPT ); Wed, 8 Apr 2020 06:46:46 -0400 Received: by mail-pj1-f65.google.com with SMTP id q16so2071360pje.1 for ; Wed, 08 Apr 2020 03:46:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kNC6DmJMmMRsRohb1ZFlDLLq7TMQSB5L7qoWyrub6m4=; b=WMS8hY4K4NveDvMO5Q90hyeSJddOuNbPgMK+hnzDvJ4BzRZ6MVJ+DqML3b3Udaahbb DIWnKxLEjBNzOUJ45GcKwvH94KVZz09k2c5VLUo8xogQy3UFMzn2xSc8Wy+IGcLwjdfE 9a7oqhHYeHLg7uJFphzpZksOOveiQFKRqA7M5SQjKThFYWXTh1XBhsHcYBQCTAzdpZQs frcX8TTqAMqahQdwn9unuhWr8Fi8s7BX7EtcTaR8FYkLITE9SqUh+qsnhXiaRde0fp+z lqeLUMg/otnJr7y3znA33YSO37ojx0u6VUNaodr/htsFpvI/4lvo2mzPRZQ0yO1DQf8b Gg5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=kNC6DmJMmMRsRohb1ZFlDLLq7TMQSB5L7qoWyrub6m4=; b=KONydhVU99P0eOHJqJdghPJwCsnratV7K7NtC0fjTk5OKm7KYkdI9MbkgTzfeN5t6I 7k6eA0aUvS2/1UDH7sQlbatEOK9v6iVGgImaeoYxaeAMZroCiHXqhRUclP+5jP2RosYb GycpFl482kJnMAys4W6Ld1DVCUTye964YaqWckrdh9k7MDOXvtUMT7dgTViCafvNromy ofBvelZpyTe7nEN4bXt5rckb6bHSQSLHtu/SLyRMq0kI1QoPs1B1MkJTdvbgthcs3+8T 8UN/M0sNvqINIGsaPpYz+1z9TWqlPrVxzEAKG5epFW6E8j1evvqN5LMFEgmkuXX4mjxS Z1yg== X-Gm-Message-State: AGi0Puaxb4Hd5H03xZUISo+ic6xguEKRgFDOV8M+P1F7nX0+qacEZ1IT Fe7r6m1sOG/3c3wx/A3USATQOnBb9oM= X-Google-Smtp-Source: APiQypJZ3k0UQs/K8qSWi/T7SNjo2slZ2KiLBGXFjWCPkAzDJSFRLV63wBa4eY0gxqb/FudnijFIvg== X-Received: by 2002:a17:902:704b:: with SMTP id h11mr6269573plt.51.1586342803461; Wed, 08 Apr 2020 03:46:43 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:42 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 33/46] e2fsck: move some fixes out of parallel pthreads Date: Wed, 8 Apr 2020 19:45:01 +0900 Message-Id: <1586342714-12536-34-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong We could only use @found_map_block to find free blocks after we have collectd all used blocks, so something like handle_fs_bad_blocks(), ext2fs_create_resize_inode(), e2fsck_pass1_dupblocks() really should be handled after all threads has been finished. Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 2 + e2fsck/pass1.c | 314 +++++++++++++++++---------- tests/f_multithread/expect.1 | 2 +- tests/f_multithread_logfile/expect.1 | 2 +- tests/f_multithread_no/expect.1 | 2 +- 5 files changed, 199 insertions(+), 123 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 20d61651..19df6cd3 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -450,6 +450,8 @@ struct e2fsck_struct { __u32 extent_depth_count[MAX_EXTENT_DEPTH_COUNT]; /* serialize fix operation for multiple threads */ pthread_mutex_t fs_fix_mutex; + /* protect block_found_map, block_dup_map */ + pthread_mutex_t fs_block_map_mutex; }; #ifdef DEBUG_THREADS diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index f3b52103..793a2944 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -132,24 +132,35 @@ static void process_inodes(e2fsck_t ctx, char *block_buf, static __u64 ext2_max_sizes[EXT2_MAX_BLOCK_LOG_SIZE - EXT2_MIN_BLOCK_LOG_SIZE + 1]; +#define e2fsck_get_lock_context(ctx) \ + e2fsck_t global_ctx = ctx->global_ctx; \ + if (!global_ctx) \ + global_ctx = ctx; \ + static void e2fsck_pass1_fix_lock(e2fsck_t ctx) { - e2fsck_t global_ctx = ctx->global_ctx; - if (!global_ctx) - global_ctx = ctx; - + e2fsck_get_lock_context(ctx); pthread_mutex_lock(&global_ctx->fs_fix_mutex); } static void e2fsck_pass1_fix_unlock(e2fsck_t ctx) { - e2fsck_t global_ctx = ctx->global_ctx; - if (!global_ctx) - global_ctx = ctx; - + e2fsck_get_lock_context(ctx); pthread_mutex_unlock(&global_ctx->fs_fix_mutex); } +static inline void e2fsck_pass1_block_map_lock(e2fsck_t ctx) +{ + e2fsck_get_lock_context(ctx); + pthread_mutex_lock(&global_ctx->fs_block_map_mutex); +} + +static inline void e2fsck_pass1_block_map_unlock(e2fsck_t ctx) +{ + e2fsck_get_lock_context(ctx); + pthread_mutex_unlock(&global_ctx->fs_block_map_mutex); +} + /* * Free all memory allocated by pass1 in preparation for restarting * things. @@ -789,11 +800,15 @@ static void check_is_really_dir(e2fsck_t ctx, struct problem_context *pctx, if (i >= 4) not_device++; + e2fsck_pass1_block_map_lock(ctx); if (blk < ctx->fs->super->s_first_data_block || blk >= ext2fs_blocks_count(ctx->fs->super) || ext2fs_fast_test_block_bitmap2(ctx->block_found_map, - blk)) + blk)) { + e2fsck_pass1_block_map_unlock(ctx); return; /* Invalid block, can't be dir */ + } + e2fsck_pass1_block_map_unlock(ctx); } blk = inode->i_block[0]; } @@ -929,19 +944,15 @@ static void reserve_block_for_root_repair(e2fsck_t ctx) errcode_t err; ext2_filsys fs = ctx->fs; - e2fsck_pass1_fix_lock(ctx); ctx->root_repair_block = 0; if (ext2fs_test_inode_bitmap2(ctx->inode_used_map, EXT2_ROOT_INO)) - goto out; + return; err = ext2fs_new_block2(fs, 0, ctx->block_found_map, &blk); if (err) - goto out; + return; ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); ctx->root_repair_block = blk; -out: - e2fsck_pass1_fix_unlock(ctx); - return; } static void reserve_block_for_lnf_repair(e2fsck_t ctx) @@ -952,18 +963,15 @@ static void reserve_block_for_lnf_repair(e2fsck_t ctx) static const char name[] = "lost+found"; ext2_ino_t ino; - e2fsck_pass1_fix_lock(ctx); ctx->lnf_repair_block = 0; if (!ext2fs_lookup(fs, EXT2_ROOT_INO, name, sizeof(name)-1, 0, &ino)) - goto out; + return; err = ext2fs_new_block2(fs, 0, ctx->block_found_map, &blk); if (err) - goto out; + return; ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); ctx->lnf_repair_block = blk; -out: - e2fsck_pass1_fix_unlock(ctx); return; } @@ -1229,6 +1237,115 @@ static int e2fsck_should_abort(e2fsck_t ctx) return 0; } +/* + * We need call mark_table_blocks() before multiple + * thread start, since all known system blocks should be + * marked and checked later. + */ +static int _e2fsck_pass1_prepare(e2fsck_t ctx) +{ + struct problem_context pctx; + ext2_filsys fs = ctx->fs; + + clear_problem_context(&pctx); + if (!(ctx->options & E2F_OPT_PREEN)) + fix_problem(ctx, PR_1_PASS_HEADER, &pctx); + + pctx.errcode = e2fsck_allocate_subcluster_bitmap(ctx->fs, + _("in-use block map"), EXT2FS_BMAP64_RBTREE, + "block_found_map", &ctx->block_found_map); + if (pctx.errcode) { + pctx.num = 1; + fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); + ctx->flags |= E2F_FLAG_ABORT; + return pctx.errcode; + } + pctx.errcode = e2fsck_allocate_block_bitmap(ctx->fs, + _("metadata block map"), EXT2FS_BMAP64_RBTREE, + "block_metadata_map", &ctx->block_metadata_map); + if (pctx.errcode) { + pctx.num = 1; + fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); + ctx->flags |= E2F_FLAG_ABORT; + return pctx.errcode; + } + + mark_table_blocks(ctx); + pctx.errcode = ext2fs_convert_subcluster_bitmap(ctx->fs, + &ctx->block_found_map); + if (pctx.errcode) { + fix_problem(ctx, PR_1_CONVERT_SUBCLUSTER, &pctx); + ctx->flags |= E2F_FLAG_ABORT; + return pctx.errcode; + } + + if (ext2fs_has_feature_mmp(fs->super) && + fs->super->s_mmp_block > fs->super->s_first_data_block && + fs->super->s_mmp_block < ext2fs_blocks_count(fs->super)) + ext2fs_mark_block_bitmap2(ctx->block_found_map, + fs->super->s_mmp_block); + + return 0; +} + +static void _e2fsck_pass1_post(e2fsck_t ctx) +{ + struct problem_context pctx; + ext2_filsys fs = ctx->fs; + char *block_buf; + + reserve_block_for_root_repair(ctx); + reserve_block_for_lnf_repair(ctx); + + if (ctx->invalid_bitmaps) + handle_fs_bad_blocks(ctx); + + if (ctx->flags & E2F_FLAG_RESIZE_INODE) { + struct ext2_inode *inode; + int inode_size = EXT2_INODE_SIZE(fs->super); + inode = e2fsck_allocate_memory(ctx, inode_size, + "scratch inode"); + + clear_problem_context(&pctx); + pctx.errcode = ext2fs_create_resize_inode(fs); + if (pctx.errcode) { + if (!fix_problem(ctx, PR_1_RESIZE_INODE_CREATE, + &pctx)) { + ctx->flags |= E2F_FLAG_ABORT; + ext2fs_free_mem(&inode); + return; + } + pctx.errcode = 0; + } + if (!pctx.errcode) { + e2fsck_read_inode(ctx, EXT2_RESIZE_INO, inode, + "recreate inode"); + inode->i_mtime = ctx->now; + e2fsck_write_inode(ctx, EXT2_RESIZE_INO, inode, + "recreate inode"); + } + ctx->flags &= ~E2F_FLAG_RESIZE_INODE; + ext2fs_free_mem(&inode); + } + + if (ctx->flags & E2F_FLAG_RESTART) + return; + + if (ctx->block_dup_map) { + if (ctx->options & E2F_OPT_PREEN) { + clear_problem_context(&pctx); + fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); + } + block_buf = + (char *)e2fsck_allocate_memory(ctx, + ctx->fs->blocksize * 3, + "block interate buffer"); + e2fsck_pass1_dupblocks(ctx, block_buf); + ext2fs_free_mem(&block_buf); + } +} + + void _e2fsck_pass1(e2fsck_t ctx) { int i; @@ -1267,10 +1384,6 @@ void _e2fsck_pass1(e2fsck_t ctx) ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs); pass1_readahead(ctx, &ra_group, &ino_threshold); - if (!(ctx->options & E2F_OPT_PREEN) && - ((!ctx->global_ctx) || (ctx->thread_info.et_thread_index == 0))) - fix_problem(ctx, PR_1_PASS_HEADER, &pctx); - if (ext2fs_has_feature_dir_index(fs->super) && !(ctx->options & E2F_OPT_NO)) { if (ext2fs_u32_list_create(&ctx->dirs_to_hash, 50)) @@ -1318,24 +1431,6 @@ void _e2fsck_pass1(e2fsck_t ctx) ctx->flags |= E2F_FLAG_ABORT; return; } - pctx.errcode = e2fsck_allocate_subcluster_bitmap(fs, - _("in-use block map"), EXT2FS_BMAP64_RBTREE, - "block_found_map", &ctx->block_found_map); - if (pctx.errcode) { - pctx.num = 1; - fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); - ctx->flags |= E2F_FLAG_ABORT; - return; - } - pctx.errcode = e2fsck_allocate_block_bitmap(fs, - _("metadata block map"), EXT2FS_BMAP64_RBTREE, - "block_metadata_map", &ctx->block_metadata_map); - if (pctx.errcode) { - pctx.num = 1; - fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); - ctx->flags |= E2F_FLAG_ABORT; - return; - } pctx.errcode = e2fsck_setup_icount(ctx, "inode_link_info", 0, NULL, &ctx->inode_link_info); if (pctx.errcode) { @@ -1377,14 +1472,6 @@ void _e2fsck_pass1(e2fsck_t ctx) } } - mark_table_blocks(ctx); - pctx.errcode = ext2fs_convert_subcluster_bitmap(fs, - &ctx->block_found_map); - if (pctx.errcode) { - fix_problem(ctx, PR_1_CONVERT_SUBCLUSTER, &pctx); - ctx->flags |= E2F_FLAG_ABORT; - goto endit; - } block_buf = (char *) e2fsck_allocate_memory(ctx, fs->blocksize * 3, "block interate buffer"); if (EXT2_INODE_SIZE(fs->super) == EXT2_GOOD_OLD_INODE_SIZE) @@ -1418,12 +1505,6 @@ void _e2fsck_pass1(e2fsck_t ctx) fs->super->s_mkfs_time < fs->super->s_inodes_count)) low_dtime_check = 0; - if (ext2fs_has_feature_mmp(fs->super) && - fs->super->s_mmp_block > fs->super->s_first_data_block && - fs->super->s_mmp_block < ext2fs_blocks_count(fs->super)) - ext2fs_mark_block_bitmap2(ctx->block_found_map, - fs->super->s_mmp_block); - /* Set up ctx->lost_and_found if possible */ (void) e2fsck_get_lost_and_found(ctx, 0); @@ -1774,8 +1855,10 @@ void _e2fsck_pass1(e2fsck_t ctx) failed_csum = 0; } + e2fsck_pass1_block_map_lock(ctx); pctx.errcode = ext2fs_copy_bitmap(ctx->block_found_map, &pb.fs_meta_blocks); + e2fsck_pass1_block_map_unlock(ctx); if (pctx.errcode) { pctx.num = 4; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); @@ -2107,9 +2190,6 @@ void _e2fsck_pass1(e2fsck_t ctx) ext2fs_close_inode_scan(scan); scan = NULL; - reserve_block_for_root_repair(ctx); - reserve_block_for_lnf_repair(ctx); - /* * If any extended attribute blocks' reference counts need to * be adjusted, either up (ctx->refcount_extra), or down @@ -2137,11 +2217,6 @@ void _e2fsck_pass1(e2fsck_t ctx) ctx->ea_block_quota_inodes = 0; } - if (ctx->invalid_bitmaps) { - e2fsck_pass1_fix_lock(ctx); - handle_fs_bad_blocks(ctx); - e2fsck_pass1_fix_unlock(ctx); - } /* We don't need the block_ea_map any more */ if (ctx->block_ea_map) { @@ -2152,31 +2227,6 @@ void _e2fsck_pass1(e2fsck_t ctx) /* We don't need the encryption policy => ID map any more */ destroy_encryption_policy_map(ctx); - if (ctx->flags & E2F_FLAG_RESIZE_INODE) { - clear_problem_context(&pctx); - e2fsck_pass1_fix_lock(ctx); - pctx.errcode = ext2fs_create_resize_inode(fs); - e2fsck_pass1_fix_unlock(ctx); - if (pctx.errcode) { - if (!fix_problem(ctx, PR_1_RESIZE_INODE_CREATE, - &pctx)) { - ctx->flags |= E2F_FLAG_ABORT; - goto endit; - } - pctx.errcode = 0; - } - if (!pctx.errcode) { - e2fsck_read_inode(ctx, EXT2_RESIZE_INO, inode, - "recreate inode"); - e2fsck_pass1_fix_lock(ctx); - inode->i_mtime = ctx->now; - e2fsck_write_inode(ctx, EXT2_RESIZE_INO, inode, - "recreate inode"); - e2fsck_pass1_fix_unlock(ctx); - } - ctx->flags &= ~E2F_FLAG_RESIZE_INODE; - } - if (ctx->flags & E2F_FLAG_RESTART) { /* * Only the master copy of the superblock and block @@ -2189,15 +2239,6 @@ void _e2fsck_pass1(e2fsck_t ctx) goto endit; } - if (ctx->block_dup_map) { - if (ctx->options & E2F_OPT_PREEN) { - clear_problem_context(&pctx); - fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); - } - e2fsck_pass1_fix_lock(ctx); - e2fsck_pass1_dupblocks(ctx, block_buf); - e2fsck_pass1_fix_unlock(ctx); - } ctx->flags |= E2F_FLAG_ALLOC_OK; ext2fs_free_mem(&inodes_to_process); endit: @@ -2501,10 +2542,10 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, assert(global_ctx->inode_reg_map == NULL); assert(global_ctx->inodes_to_rebuild == NULL); - assert(global_ctx->block_found_map == NULL); assert(global_ctx->block_dup_map == NULL); + assert(global_ctx->block_found_map != NULL); + assert(global_ctx->block_metadata_map != NULL); assert(global_ctx->block_ea_map == NULL); - assert(global_ctx->block_metadata_map == NULL); retval = ext2fs_get_mem(sizeof(struct e2fsck_struct), &thread_context); if (retval) { @@ -2672,10 +2713,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; - ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; - ext2fs_block_bitmap block_metadata_map = global_ctx->block_metadata_map; ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; ext2_icount_t inode_count = global_ctx->inode_count; ext2_icount_t inode_link_info = global_ctx->inode_link_info; @@ -2717,10 +2756,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->inode_imagic_map = inode_imagic_map; global_ctx->inodes_to_rebuild = inodes_to_rebuild; global_ctx->inode_reg_map = inode_reg_map; - global_ctx->block_found_map = block_found_map; - global_ctx->block_dup_map = block_dup_map; global_ctx->block_ea_map = block_ea_map; - global_ctx->block_metadata_map = block_metadata_map; + global_ctx->block_dup_map = block_dup_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); global_ctx->dx_dir_info = dx_dir_info; @@ -2783,10 +2820,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_imagic_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_reg_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inodes_to_rebuild); - PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_found_map); - PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_dup_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_ea_map); - PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_metadata_map); return 0; } @@ -2809,10 +2843,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) PASS1_FREE_CTX_BITMAP(thread_ctx, inode_imagic_map); PASS1_FREE_CTX_BITMAP(thread_ctx, inode_reg_map); PASS1_FREE_CTX_BITMAP(thread_ctx, inodes_to_rebuild); - PASS1_FREE_CTX_BITMAP(thread_ctx, block_found_map); - PASS1_FREE_CTX_BITMAP(thread_ctx, block_dup_map); PASS1_FREE_CTX_BITMAP(thread_ctx, block_ea_map); - PASS1_FREE_CTX_BITMAP(thread_ctx, block_metadata_map); ext2fs_free_icount(thread_ctx->inode_count); ext2fs_free_icount(thread_ctx->inode_link_info); ext2fs_free_mem(&thread_ctx); @@ -3010,7 +3041,12 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) unsigned flexbg_size = 1; int max_threads; + retval = _e2fsck_pass1_prepare(global_ctx); + if (retval) + goto out_abort; + pthread_mutex_init(&global_ctx->fs_fix_mutex, NULL); + pthread_mutex_init(&global_ctx->fs_block_map_mutex, NULL); if (ext2fs_has_feature_flex_bg(global_ctx->fs->super)) flexbg_size = 1 << global_ctx->fs->super->s_log_groups_per_flex; @@ -3049,6 +3085,7 @@ out_abort: void e2fsck_pass1(e2fsck_t ctx) { e2fsck_pass1_multithread(ctx); + _e2fsck_pass1_post(ctx); } #undef FINISH_INODE_LOOP @@ -3234,7 +3271,12 @@ static void alloc_imagic_map(e2fsck_t ctx) */ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) { - struct problem_context pctx; + struct problem_context pctx; + e2fsck_t global_ctx; + + global_ctx = ctx->global_ctx; + if (!global_ctx) + global_ctx = ctx; clear_problem_context(&pctx); @@ -3243,11 +3285,15 @@ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) !(ctx->options & E2F_OPT_UNSHARE_BLOCKS)) { return; } - if (!ctx->block_dup_map) { + /** + * this should be safe because this operation has + * been serialized by mutex. + */ + if (!global_ctx->block_dup_map) { pctx.errcode = e2fsck_allocate_block_bitmap(ctx->fs, _("multiply claimed block map"), EXT2FS_BMAP64_RBTREE, "block_dup_map", - &ctx->block_dup_map); + &global_ctx->block_dup_map); if (pctx.errcode) { pctx.num = 3; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, @@ -3257,7 +3303,7 @@ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) return; } } - ext2fs_fast_mark_block_bitmap2(ctx->block_dup_map, block); + ext2fs_fast_mark_block_bitmap2(global_ctx->block_dup_map, block); } else { ext2fs_fast_mark_block_bitmap2(ctx->block_found_map, block); } @@ -3270,14 +3316,16 @@ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) static _INLINE_ void mark_blocks_used(e2fsck_t ctx, blk64_t block, unsigned int num) { - if (ext2fs_test_block_bitmap_range2(ctx->block_found_map, block, num)) + e2fsck_pass1_block_map_lock(ctx); + if (ext2fs_test_block_bitmap_range2(ctx->block_found_map, block, num)) { ext2fs_mark_block_bitmap_range2(ctx->block_found_map, block, num); - else { + } else { unsigned int i; for (i = 0; i < num; i += EXT2FS_CLUSTER_RATIO(ctx->fs)) mark_block_used(ctx, block + i); } + e2fsck_pass1_block_map_unlock(ctx); } /* @@ -3591,7 +3639,9 @@ refcount_fail: inc_ea_inode_refs(ctx, pctx, first, end); ea_refcount_store(ctx->refcount, blk, header->h_refcount - 1); + e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); + e2fsck_pass1_block_map_unlock(ctx); ext2fs_fast_mark_block_bitmap2(ctx->block_ea_map, blk); return 1; @@ -3976,7 +4026,9 @@ report_problem: pctx->str = "EXT2_EXTENT_UP"; return; } + e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); + e2fsck_pass1_block_map_unlock(ctx); pb->num_blocks++; goto next; } @@ -4083,6 +4135,7 @@ alloc_later: pb->last_block, extent.e_pblk, extent.e_lblk)) { + e2fsck_pass1_block_map_lock(ctx); for (i = 0; i < extent.e_len; i++) { pctx->blk = extent.e_lblk + i; pctx->blk2 = extent.e_pblk + i; @@ -4090,6 +4143,7 @@ alloc_later: mark_block_used(ctx, extent.e_pblk + i); mark_block_used(ctx, extent.e_pblk + i); } + e2fsck_pass1_block_map_unlock(ctx); } /* @@ -4732,6 +4786,7 @@ static int process_block(ext2_filsys fs, *block_nr = 0; return 0; } + if (!p->suppress && (p->num_illegal_blocks % 12) == 0) { if (fix_problem(ctx, PR_1_TOO_MANY_BAD_BLOCKS, pctx)) { p->clear = 1; @@ -4773,8 +4828,11 @@ static int process_block(ext2_filsys fs, * being in use; all of the other blocks are handled * by mark_table_blocks()). */ - if (blockcnt == BLOCK_COUNT_DIND) + if (blockcnt == BLOCK_COUNT_DIND) { + e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); + e2fsck_pass1_block_map_unlock(ctx); + } p->num_blocks++; } else if (!(ctx->fs->cluster_ratio_bits && p->previous_block && @@ -4782,15 +4840,19 @@ static int process_block(ext2_filsys fs, EXT2FS_B2C(ctx->fs, p->previous_block)) && (blk & EXT2FS_CLUSTER_MASK(ctx->fs)) == ((unsigned) blockcnt & EXT2FS_CLUSTER_MASK(ctx->fs)))) { + e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); + e2fsck_pass1_block_map_unlock(ctx); p->num_blocks++; } else if (has_unaligned_cluster_map(ctx, p->previous_block, p->last_block, blk, blockcnt)) { pctx->blk = blockcnt; pctx->blk2 = blk; fix_problem(ctx, PR_1_MISALIGNED_CLUSTER, pctx); + e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); mark_block_used(ctx, blk); + e2fsck_pass1_block_map_unlock(ctx); } if (blockcnt >= 0) p->last_block = blockcnt; @@ -4857,10 +4919,12 @@ static int process_bad_block(ext2_filsys fs, } if (blockcnt < 0) { + e2fsck_pass1_block_map_lock(ctx); if (ext2fs_test_block_bitmap2(p->fs_meta_blocks, blk)) { p->bbcheck = 1; if (fix_problem(ctx, PR_1_BB_FS_BLOCK, pctx)) { *block_nr = 0; + e2fsck_pass1_block_map_unlock(ctx); return BLOCK_CHANGED; } } else if (ext2fs_test_block_bitmap2(ctx->block_found_map, @@ -4869,12 +4933,17 @@ static int process_bad_block(ext2_filsys fs, if (fix_problem(ctx, PR_1_BBINODE_BAD_METABLOCK, pctx)) { *block_nr = 0; + e2fsck_pass1_block_map_unlock(ctx); return BLOCK_CHANGED; } - if (e2fsck_should_abort(ctx)) + if (e2fsck_should_abort(ctx)) { + e2fsck_pass1_block_map_unlock(ctx); return BLOCK_ABORT; - } else + } + } else { mark_block_used(ctx, blk); + } + e2fsck_pass1_block_map_unlock(ctx); return 0; } #if 0 @@ -4887,10 +4956,13 @@ static int process_bad_block(ext2_filsys fs, * there's an overlap between the filesystem table blocks * (bitmaps and inode table) and the bad block list. */ + e2fsck_pass1_block_map_lock(ctx); if (!ext2fs_test_block_bitmap2(ctx->block_found_map, blk)) { ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); + e2fsck_pass1_block_map_unlock(ctx); return 0; } + e2fsck_pass1_block_map_unlock(ctx); /* * Try to find the where the filesystem block was used... */ @@ -5045,6 +5117,7 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group, fix_problem(ctx, (old_block ? PR_1_RELOC_FROM_TO : PR_1_RELOC_TO), &pctx); pctx.blk2 = 0; + e2fsck_pass1_block_map_lock(ctx); for (i = 0; i < num; i++) { pctx.blk = i; ext2fs_mark_block_bitmap2(ctx->block_found_map, (*new_block)+i); @@ -5065,6 +5138,7 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group, if (pctx.errcode) fix_problem(ctx, PR_1_RELOC_WRITE_ERR, &pctx); } + e2fsck_pass1_block_map_unlock(ctx); ext2fs_free_mem(&buf); } diff --git a/tests/f_multithread/expect.1 b/tests/f_multithread/expect.1 index 8d2acd2b..4db68d9e 100644 --- a/tests/f_multithread/expect.1 +++ b/tests/f_multithread/expect.1 @@ -1,7 +1,7 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) -[Thread 0] Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity diff --git a/tests/f_multithread_logfile/expect.1 b/tests/f_multithread_logfile/expect.1 index 8d2acd2b..4db68d9e 100644 --- a/tests/f_multithread_logfile/expect.1 +++ b/tests/f_multithread_logfile/expect.1 @@ -1,7 +1,7 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) -[Thread 0] Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity diff --git a/tests/f_multithread_no/expect.1 b/tests/f_multithread_no/expect.1 index f85a3382..eda2fcac 100644 --- a/tests/f_multithread_no/expect.1 +++ b/tests/f_multithread_no/expect.1 @@ -1,7 +1,7 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) -[Thread 0] Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity From patchwork Wed Apr 8 10:45:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267894 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ADuNUHw4; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DS1HP6z9sSb for ; Wed, 8 Apr 2020 20:46:48 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728206AbgDHKqr (ORCPT ); Wed, 8 Apr 2020 06:46:47 -0400 Received: from mail-pj1-f68.google.com ([209.85.216.68]:51876 "EHLO mail-pj1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKqr (ORCPT ); Wed, 8 Apr 2020 06:46:47 -0400 Received: by mail-pj1-f68.google.com with SMTP id n4so1004907pjp.1 for ; Wed, 08 Apr 2020 03:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=fYXRiYU0Sfgy3GFZs2RGKrN+nTDw2FLVJUWDpghCSrU=; b=ADuNUHw4YYBzaCXzD78rYz6iihmLTs6csowzTHfu4PxzVLuQsII4EhV0dquz9a3bU7 PMRNXRNheoB0aX8D/gvLy9zykSAsaHyEw0ZwoCTToqjbvRK1nIy3pX5xVeKavciMGOLS VoOC+qc2L08T9ctfDZ7rUI9nFIx0sBwdXSqfca8gkHMdya0nSigz/ucZDs6myh2Qapmu 0RxKBI5wf1gHky83bsrs4Hmvjn1r+nP98CXk8ICJCPYXBQnEFWiNyV9q2sP5BaoAUiea He3OSTKOU5MBjE6uRZYvSJGf3Fgpaw5N6rmi4NbiCkDAPLhsJyfglykpF1yIUEPlOwqo S99A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=fYXRiYU0Sfgy3GFZs2RGKrN+nTDw2FLVJUWDpghCSrU=; b=C81qW014/81jQgu8Yp1J+T9/0sJJBALVgPBXrJFJqxDyQaZVYLk3eGUuZwsv4K5ndR 4V3hvk6J73kTHERZDTCqFEhFUoIkOVdhKJoWQDjphoAEZGrAgUi2rRRmydgycJ6xNfpV 6dUQVfofproDKtd1GU+7byuScY6Ckvu7gcjb7THcPIZuFKOhUih+/aOycr+LoNMnH6ds Sa6iRQ3emXQF9nrTZ33FDWutGZPkyMcEP//PhtM8goDRpL/+RSgx9QlGAVMn2TkkMnsu 7hHAyOfDODsvvDJW4+WjXNaBDqrHCpsfbpdAZSJSasYGUsFSis5K/u8XXRdzwlOkyRcD 6ecQ== X-Gm-Message-State: AGi0PubDL68XZ1e3zj4G7KEwgJWv33+y6D7BdlbIKs02dr4hDUTE2ORF 6yM3btBDN7F0Dgbc53hI42anRbQcopU= X-Google-Smtp-Source: APiQypJ7kLm8DrqxgRyPXYO1G0HTf7vjzQhpWDcuWnVPBoF5U2GXwD/OlPl1a41zk+UkdTj235NoXA== X-Received: by 2002:a17:90a:2751:: with SMTP id o75mr1393318pje.26.1586342805529; Wed, 08 Apr 2020 03:46:45 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:45 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 34/46] e2fsck: split and merge invalid bitmaps Date: Wed, 8 Apr 2020 19:45:02 +0900 Message-Id: <1586342714-12536-35-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 793a2944..348cf46b 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2523,6 +2523,65 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) return retval; } +static void e2fsck_pass1_copy_invalid_bitmaps(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + int i, j; + int grp_start = thread_ctx->thread_info.et_group_start; + int grp_end = thread_ctx->thread_info.et_group_end; + int total = grp_end - grp_start; + + thread_ctx->invalid_inode_bitmap_flag = e2fsck_allocate_memory(global_ctx, + sizeof(int) * total, "invalid_inode_bitmap"); + thread_ctx->invalid_block_bitmap_flag = e2fsck_allocate_memory(global_ctx, + sizeof(int) * total, "invalid_block_bitmap"); + thread_ctx->invalid_inode_table_flag = e2fsck_allocate_memory(global_ctx, + sizeof(int) * total, "invalid_inode_table"); + thread_ctx->invalid_bitmaps = 0; + + for (i = grp_start, j = 0; i < grp_end; i++, j++) { + thread_ctx->invalid_block_bitmap_flag[j] = + global_ctx->invalid_block_bitmap_flag[i]; + thread_ctx->invalid_inode_bitmap_flag[j] = + global_ctx->invalid_inode_bitmap_flag[i]; + thread_ctx->invalid_inode_table_flag[j] = + global_ctx->invalid_inode_table_flag[i]; + if (thread_ctx->invalid_block_bitmap_flag[j]) + thread_ctx->invalid_bitmaps++; + if (thread_ctx->invalid_inode_bitmap_flag[j]) + thread_ctx->invalid_bitmaps++; + if (thread_ctx->invalid_inode_table_flag[j]) + thread_ctx->invalid_bitmaps++; + + } +} + +static void e2fsck_pass1_merge_invalid_bitmaps(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + int i, j; + int grp_start = thread_ctx->thread_info.et_group_start; + int grp_end = thread_ctx->thread_info.et_group_end; + + for (i = grp_start, j = 0; i < grp_end; i++, j++) { + global_ctx->invalid_block_bitmap_flag[i] = + thread_ctx->invalid_block_bitmap_flag[j]; + global_ctx->invalid_inode_bitmap_flag[i] = + thread_ctx->invalid_inode_bitmap_flag[j]; + global_ctx->invalid_inode_table_flag[i] = + thread_ctx->invalid_inode_table_flag[j]; + if (thread_ctx->invalid_block_bitmap_flag[j]) + global_ctx->invalid_bitmaps++; + if (thread_ctx->invalid_inode_bitmap_flag[j]) + global_ctx->invalid_bitmaps++; + if (thread_ctx->invalid_inode_table_flag[j]) + global_ctx->invalid_bitmaps++; + } + ext2fs_free_mem(&thread_ctx->invalid_block_bitmap_flag); + ext2fs_free_mem(&thread_ctx->invalid_inode_bitmap_flag); + ext2fs_free_mem(&thread_ctx->invalid_inode_table_flag); +} + static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx, int thread_index, @@ -2599,6 +2658,7 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, goto out_fs; } *thread_ctx = thread_context; + e2fsck_pass1_copy_invalid_bitmaps(global_ctx, thread_context); return 0; out_fs: ext2fs_free_mem(&thread_fs); @@ -2739,6 +2799,10 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx int dx_dir_info_count = global_ctx->dx_dir_info_count; ext2_u32_list dirs_to_hash = global_ctx->dirs_to_hash; quota_ctx_t qctx = global_ctx->qctx; + int *invalid_block_bitmap_flag = global_ctx->invalid_block_bitmap_flag; + int *invalid_inode_bitmap_flag = global_ctx->invalid_inode_bitmap_flag; + int *invalid_inode_table_flag = global_ctx->invalid_inode_table_flag; + int invalid_bitmaps = global_ctx->invalid_bitmaps; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2809,6 +2873,11 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx } global_ctx->qctx = qctx; e2fsck_pass1_merge_quota_ctx(global_ctx, thread_ctx); + global_ctx->invalid_block_bitmap_flag = invalid_block_bitmap_flag; + global_ctx->invalid_inode_bitmap_flag = invalid_inode_bitmap_flag; + global_ctx->invalid_inode_table_flag = invalid_inode_table_flag; + global_ctx->invalid_bitmaps = invalid_bitmaps; + e2fsck_pass1_merge_invalid_bitmaps(global_ctx, thread_ctx); /* * PASS1_COPY_CTX_BITMAP might return directly from this function, @@ -2859,6 +2928,8 @@ static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, int i; struct e2fsck_thread_info *pinfo; + /* merge invalid bitmaps will recalculate it */ + global_ctx->invalid_bitmaps = 0; for (i = 0; i < num_threads; i++) { pinfo = &infos[i]; From patchwork Wed Apr 8 10:45:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267895 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Drc092r2; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DV1Yjvz9sSG for ; Wed, 8 Apr 2020 20:46:50 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728289AbgDHKqt (ORCPT ); Wed, 8 Apr 2020 06:46:49 -0400 Received: from mail-pj1-f68.google.com ([209.85.216.68]:55010 "EHLO mail-pj1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKqt (ORCPT ); Wed, 8 Apr 2020 06:46:49 -0400 Received: by mail-pj1-f68.google.com with SMTP id np9so998166pjb.4 for ; Wed, 08 Apr 2020 03:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EBGXkTLDO+qkfmVTsY5v+eUq67B/IGtywx0ipqzj7ag=; b=Drc092r2icgCOlBgkEI2ooEy8ExW1ODmdTMB1PKwBfsXr1WZ0dO2cuEhHc0MqIFR3k ypEPVJ9NSsZWKah5e1m9WnMWjATdKRYpUpSNrCIZTF0DAeHRWeDUgaU3qjqGNA99fgRe gSEspEBxvAD+/jyV64oYKsRjPotGWdC/fT+cBp6fjF+jdrFana+M+QhKhIXPKahd3iot YUDlLmOSym3uTADGr/B+zv+qca3sJUoxTXx6w9sS3cyGqS5U9csfEKSfKEnhB4k0umGQ 6zFcb0p8VNEbIeO1OMPfm5sX4LuWY79fq3FhMWGXNbqTR+mVi4y1fRI+N0zVI0++hFSq FgFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EBGXkTLDO+qkfmVTsY5v+eUq67B/IGtywx0ipqzj7ag=; b=eas4gvDmYlMQInCoDx59/QbZ83C7CkXciMgR1R+xVa8E2WTqLJww2pbMSTgN+zm3q0 C4UhXswij5uUEbGCm3XdaOamAAasIKiSuDdtlKG5b+5pG+RjabVW8eh4abNr69O/wxIN zcgkYeQWVxQpQ6x2hYO/nUEmQbsWWbUc/rHQyABYf7Ve+3IvLbYQmPfAQYMUfVNKFvT+ PicoPG6Z2Tt3ZrN8ehIefTDCTGdzJybj3q+2JAt//BG/REoJ07EI6aerWdo62LrHDdJe R/ogc86xBbJG8fS6WN6YJCcPvkdAv+QyJ9eIF6/Ej3NvEf1iR8d33ITdCYV64SfqoiMH p/lA== X-Gm-Message-State: AGi0PuZPULajWd/VPtsZmAzeFW4zEbzZPE4hIW5e33XdCAeBzn/6WyKo FcocrqI8y7G2N1MqP7QwFciI4QFuU0E= X-Google-Smtp-Source: APiQypL5JeF/FXWxIaMrsIhRc20ebhTqVoZX6dPmLYnOag0HuLyfJe8qivlNkSdi4/NU/TImFxgegg== X-Received: by 2002:a17:90a:c78b:: with SMTP id gn11mr4516959pjb.147.1586342807771; Wed, 08 Apr 2020 03:46:47 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:47 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 35/46] e2fsck: fix to protect EA checking Date: Wed, 8 Apr 2020 19:45:03 +0900 Message-Id: <1586342714-12536-36-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong EA related variables are now shared by different threads, but without any protections. So this patch try to fix these by seralizing operations. Optimizations could be done later, since EA blocks could be shared, need be careful to split and merge. Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 2 + e2fsck/pass1.c | 184 ++++++++++++++++++++++++++++-------------------- 2 files changed, 111 insertions(+), 75 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 19df6cd3..4e156f17 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -452,6 +452,8 @@ struct e2fsck_struct { pthread_mutex_t fs_fix_mutex; /* protect block_found_map, block_dup_map */ pthread_mutex_t fs_block_map_mutex; + /* protect ea related structure */ + pthread_mutex_t fs_ea_mutex; }; #ifdef DEBUG_THREADS diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 348cf46b..1e98f8b6 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -161,6 +161,18 @@ static inline void e2fsck_pass1_block_map_unlock(e2fsck_t ctx) pthread_mutex_unlock(&global_ctx->fs_block_map_mutex); } +static inline void e2fsck_pass1_ea_lock(e2fsck_t ctx) +{ + e2fsck_get_lock_context(ctx); + pthread_mutex_lock(&global_ctx->fs_ea_mutex); +} + +static inline void e2fsck_pass1_ea_unlock(e2fsck_t ctx) +{ + e2fsck_get_lock_context(ctx); + pthread_mutex_unlock(&global_ctx->fs_ea_mutex); +} + /* * Free all memory allocated by pass1 in preparation for restarting * things. @@ -439,15 +451,16 @@ static void inc_ea_inode_refs(e2fsck_t ctx, struct problem_context *pctx, struct ext2_ext_attr_entry *first, void *end) { struct ext2_ext_attr_entry *entry; + e2fsck_t global_ctx = ctx->global_ctx ? ctx->global_ctx : ctx; for (entry = first; (void *)entry < end && !EXT2_EXT_IS_LAST_ENTRY(entry); entry = EXT2_EXT_ATTR_NEXT(entry)) { if (!entry->e_value_inum) continue; - if (!ctx->ea_inode_refs) { + if (!global_ctx->ea_inode_refs) { pctx->errcode = ea_refcount_create(0, - &ctx->ea_inode_refs); + &global_ctx->ea_inode_refs); if (pctx->errcode) { pctx->num = 4; fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); @@ -455,8 +468,8 @@ static void inc_ea_inode_refs(e2fsck_t ctx, struct problem_context *pctx, return; } } - ea_refcount_increment(ctx->ea_inode_refs, entry->e_value_inum, - 0); + ea_refcount_increment(global_ctx->ea_inode_refs, + entry->e_value_inum, 0); } } @@ -582,8 +595,10 @@ fix: * EA(s) in automatic fashion -bzzz */ if (problem == 0 || !fix_problem(ctx, problem, pctx)) { + e2fsck_pass1_ea_lock(ctx); inc_ea_inode_refs(ctx, pctx, (struct ext2_ext_attr_entry *)start, end); + e2fsck_pass1_ea_unlock(ctx); return; } @@ -1292,14 +1307,49 @@ static void _e2fsck_pass1_post(e2fsck_t ctx) { struct problem_context pctx; ext2_filsys fs = ctx->fs; - char *block_buf; + char *block_buf = + (char *)e2fsck_allocate_memory(ctx, ctx->fs->blocksize * 3, + "block interate buffer"); reserve_block_for_root_repair(ctx); reserve_block_for_lnf_repair(ctx); + /* + * If any extended attribute blocks' reference counts need to + * be adjusted, either up (ctx->refcount_extra), or down + * (ctx->refcount), then fix them. + */ + if (ctx->refcount) { + adjust_extattr_refcount(ctx, ctx->refcount, block_buf, -1); + ea_refcount_free(ctx->refcount); + ctx->refcount = 0; + } + if (ctx->refcount_extra) { + adjust_extattr_refcount(ctx, ctx->refcount_extra, + block_buf, +1); + ea_refcount_free(ctx->refcount_extra); + ctx->refcount_extra = 0; + } + + if (ctx->ea_block_quota_blocks) { + ea_refcount_free(ctx->ea_block_quota_blocks); + ctx->ea_block_quota_blocks = 0; + } + + if (ctx->ea_block_quota_inodes) { + ea_refcount_free(ctx->ea_block_quota_inodes); + ctx->ea_block_quota_inodes = 0; + } + if (ctx->invalid_bitmaps) handle_fs_bad_blocks(ctx); + /* We don't need the block_ea_map any more */ + if (ctx->block_ea_map) { + ext2fs_free_block_bitmap(ctx->block_ea_map); + ctx->block_ea_map = 0; + } + if (ctx->flags & E2F_FLAG_RESIZE_INODE) { struct ext2_inode *inode; int inode_size = EXT2_INODE_SIZE(fs->super); @@ -1336,10 +1386,6 @@ static void _e2fsck_pass1_post(e2fsck_t ctx) clear_problem_context(&pctx); fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); } - block_buf = - (char *)e2fsck_allocate_memory(ctx, - ctx->fs->blocksize * 3, - "block interate buffer"); e2fsck_pass1_dupblocks(ctx, block_buf); ext2fs_free_mem(&block_buf); } @@ -2190,40 +2236,6 @@ void _e2fsck_pass1(e2fsck_t ctx) ext2fs_close_inode_scan(scan); scan = NULL; - /* - * If any extended attribute blocks' reference counts need to - * be adjusted, either up (ctx->refcount_extra), or down - * (ctx->refcount), then fix them. - */ - if (ctx->refcount) { - adjust_extattr_refcount(ctx, ctx->refcount, block_buf, -1); - ea_refcount_free(ctx->refcount); - ctx->refcount = 0; - } - if (ctx->refcount_extra) { - adjust_extattr_refcount(ctx, ctx->refcount_extra, - block_buf, +1); - ea_refcount_free(ctx->refcount_extra); - ctx->refcount_extra = 0; - } - - if (ctx->ea_block_quota_blocks) { - ea_refcount_free(ctx->ea_block_quota_blocks); - ctx->ea_block_quota_blocks = 0; - } - - if (ctx->ea_block_quota_inodes) { - ea_refcount_free(ctx->ea_block_quota_inodes); - ctx->ea_block_quota_inodes = 0; - } - - - /* We don't need the block_ea_map any more */ - if (ctx->block_ea_map) { - ext2fs_free_block_bitmap(ctx->block_ea_map); - ctx->block_ea_map = 0; - } - /* We don't need the encryption policy => ID map any more */ destroy_encryption_policy_map(ctx); @@ -2774,7 +2786,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; - ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; ext2_icount_t inode_count = global_ctx->inode_count; ext2_icount_t inode_link_info = global_ctx->inode_link_info; @@ -2803,6 +2814,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx int *invalid_inode_bitmap_flag = global_ctx->invalid_inode_bitmap_flag; int *invalid_inode_table_flag = global_ctx->invalid_inode_table_flag; int invalid_bitmaps = global_ctx->invalid_bitmaps; + ext2_refcount_t refcount = global_ctx->refcount; + ext2_refcount_t refcount_extra = global_ctx->refcount_extra; + ext2_refcount_t ea_block_quota_blocks = global_ctx->ea_block_quota_blocks; + ext2_refcount_t ea_block_quota_inodes = global_ctx->ea_block_quota_inodes; + ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; + ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2820,7 +2837,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->inode_imagic_map = inode_imagic_map; global_ctx->inodes_to_rebuild = inodes_to_rebuild; global_ctx->inode_reg_map = inode_reg_map; - global_ctx->block_ea_map = block_ea_map; global_ctx->block_dup_map = block_dup_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); @@ -2830,6 +2846,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); global_ctx->inode_count = inode_count; global_ctx->inode_link_info = inode_link_info; + global_ctx->refcount = refcount; + global_ctx->refcount_extra = refcount_extra; + global_ctx->ea_block_quota_blocks = ea_block_quota_blocks; + global_ctx->ea_block_quota_inodes = ea_block_quota_inodes; + global_ctx->block_ea_map = block_ea_map; + global_ctx->ea_inode_refs = ea_inode_refs; PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_directory_count); PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_regular_count); PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_blockdev_count); @@ -2889,7 +2911,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_imagic_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_reg_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inodes_to_rebuild); - PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_ea_map); return 0; } @@ -2912,7 +2933,6 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) PASS1_FREE_CTX_BITMAP(thread_ctx, inode_imagic_map); PASS1_FREE_CTX_BITMAP(thread_ctx, inode_reg_map); PASS1_FREE_CTX_BITMAP(thread_ctx, inodes_to_rebuild); - PASS1_FREE_CTX_BITMAP(thread_ctx, block_ea_map); ext2fs_free_icount(thread_ctx->inode_count); ext2fs_free_icount(thread_ctx->inode_link_info); ext2fs_free_mem(&thread_ctx); @@ -3118,6 +3138,7 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) pthread_mutex_init(&global_ctx->fs_fix_mutex, NULL); pthread_mutex_init(&global_ctx->fs_block_map_mutex, NULL); + pthread_mutex_init(&global_ctx->fs_ea_mutex, NULL); if (ext2fs_has_feature_flex_bg(global_ctx->fs->super)) flexbg_size = 1 << global_ctx->fs->super->s_log_groups_per_flex; @@ -3434,12 +3455,10 @@ static void adjust_extattr_refcount(e2fsck_t ctx, ext2_refcount_t refcount, should_be = header->h_refcount + adjust_sign * (int)count; pctx.num = should_be; if (fix_problem(ctx, PR_1_EXTATTR_REFCOUNT, &pctx)) { - e2fsck_pass1_fix_lock(ctx); header->h_refcount = should_be; pctx.errcode = ext2fs_write_ext_attr3(fs, blk, block_buf, pctx.ino); - e2fsck_pass1_fix_unlock(ctx); if (pctx.errcode) { fix_problem(ctx, PR_1_EXTATTR_WRITE_ABORT, &pctx); @@ -3466,6 +3485,7 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, __u64 quota_inodes = 0; region_t region = 0; int failed_csum = 0; + e2fsck_t global_ctx = ctx->global_ctx ? ctx->global_ctx : ctx; ea_block_quota->blocks = 0; ea_block_quota->inodes = 0; @@ -3489,26 +3509,30 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, } /* If ea bitmap hasn't been allocated, create it */ - if (!ctx->block_ea_map) { + e2fsck_pass1_ea_lock(ctx); + if (!global_ctx->block_ea_map) { pctx->errcode = e2fsck_allocate_block_bitmap(fs, _("ext attr block map"), EXT2FS_BMAP64_RBTREE, "block_ea_map", - &ctx->block_ea_map); + &global_ctx->block_ea_map); if (pctx->errcode) { pctx->num = 2; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_ea_unlock(ctx); return 0; } } /* Create the EA refcount structure if necessary */ - if (!ctx->refcount) { - pctx->errcode = ea_refcount_create(0, &ctx->refcount); + if (!global_ctx->refcount) { + pctx->errcode = ea_refcount_create(0, + &global_ctx->refcount); if (pctx->errcode) { pctx->num = 1; fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_ea_unlock(ctx); return 0; } } @@ -3519,37 +3543,44 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, #endif /* Have we seen this EA block before? */ - if (ext2fs_fast_test_block_bitmap2(ctx->block_ea_map, blk)) { + if (ext2fs_fast_test_block_bitmap2(global_ctx->block_ea_map, + blk)) { ea_block_quota->blocks = EXT2FS_C2B(fs, 1); ea_block_quota->inodes = 0; - if (ctx->ea_block_quota_blocks) { - ea_refcount_fetch(ctx->ea_block_quota_blocks, blk, - "a_blocks); + if (global_ctx->ea_block_quota_blocks) { + ea_refcount_fetch(global_ctx->ea_block_quota_blocks, + blk, "a_blocks); if (quota_blocks) ea_block_quota->blocks = quota_blocks; } - if (ctx->ea_block_quota_inodes) - ea_refcount_fetch(ctx->ea_block_quota_inodes, blk, - &ea_block_quota->inodes); + if (global_ctx->ea_block_quota_inodes) + ea_refcount_fetch(global_ctx->ea_block_quota_inodes, + blk, &ea_block_quota->inodes); - if (ea_refcount_decrement(ctx->refcount, blk, 0) == 0) + if (ea_refcount_decrement(global_ctx->refcount, + blk, 0) == 0) { + e2fsck_pass1_ea_unlock(ctx); return 1; + } /* Ooops, this EA was referenced more than it stated */ - if (!ctx->refcount_extra) { + if (!global_ctx->refcount_extra) { pctx->errcode = ea_refcount_create(0, - &ctx->refcount_extra); + &global_ctx->refcount_extra); if (pctx->errcode) { pctx->num = 2; fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_ea_unlock(ctx); return 0; } } - ea_refcount_increment(ctx->refcount_extra, blk, 0); + ea_refcount_increment(global_ctx->refcount_extra, blk, 0); + e2fsck_pass1_ea_unlock(ctx); return 1; } + e2fsck_pass1_ea_unlock(ctx); /* * OK, we haven't seen this EA block yet. So we need to @@ -3676,44 +3707,47 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, return 0; } + e2fsck_pass1_ea_lock(ctx); if (quota_blocks != EXT2FS_C2B(fs, 1U)) { - if (!ctx->ea_block_quota_blocks) { + if (!global_ctx->ea_block_quota_blocks) { pctx->errcode = ea_refcount_create(0, - &ctx->ea_block_quota_blocks); + &global_ctx->ea_block_quota_blocks); if (pctx->errcode) { pctx->num = 3; goto refcount_fail; } } - ea_refcount_store(ctx->ea_block_quota_blocks, blk, - quota_blocks); + ea_refcount_store(global_ctx->ea_block_quota_blocks, + blk, quota_blocks); } if (quota_inodes) { - if (!ctx->ea_block_quota_inodes) { + if (!global_ctx->ea_block_quota_inodes) { pctx->errcode = ea_refcount_create(0, - &ctx->ea_block_quota_inodes); + &global_ctx->ea_block_quota_inodes); if (pctx->errcode) { pctx->num = 4; refcount_fail: fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_ea_unlock(ctx); return 0; } } - ea_refcount_store(ctx->ea_block_quota_inodes, blk, - quota_inodes); + ea_refcount_store(global_ctx->ea_block_quota_inodes, + blk, quota_inodes); } ea_block_quota->blocks = quota_blocks; ea_block_quota->inodes = quota_inodes; - inc_ea_inode_refs(ctx, pctx, first, end); - ea_refcount_store(ctx->refcount, blk, header->h_refcount - 1); + inc_ea_inode_refs(global_ctx, pctx, first, end); + ea_refcount_store(global_ctx->refcount, blk, header->h_refcount - 1); e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); e2fsck_pass1_block_map_unlock(ctx); - ext2fs_fast_mark_block_bitmap2(ctx->block_ea_map, blk); + ext2fs_fast_mark_block_bitmap2(global_ctx->block_ea_map, blk); + e2fsck_pass1_ea_unlock(ctx); return 1; clear_extattr: From patchwork Wed Apr 8 10:45:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267896 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=k9WeWuKl; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1DW5719z9sSG for ; Wed, 8 Apr 2020 20:46:51 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728294AbgDHKqv (ORCPT ); Wed, 8 Apr 2020 06:46:51 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:35668 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728290AbgDHKqv (ORCPT ); Wed, 8 Apr 2020 06:46:51 -0400 Received: by mail-pg1-f195.google.com with SMTP id k5so3164154pga.2 for ; Wed, 08 Apr 2020 03:46:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Nj5Zcto19jAG/2+iUH/d35sA52Q6CmVF9ht5DaMNdYM=; b=k9WeWuKlNBsXuibwg7Fsvr58pTJgR0WJCnCLLbXowu3kRDblswG4BWDhorljGg1nGy awvxw7VKIsBZyZ8kxPEEoP5q3jeg9s6qUWxEzkQiHKFs6BhpMxfaklDx89F8crws9oaE 5qpmWYkF9mJa0aZ9/gAN9yYs6hkxpK6US8ENUUBb33U3nDj00Jq4fAgRUGhY/qHl26zh xjKowqaAv67NoWQRXA0upq6Tf6CvcukqBYUwlo7fLyM3yPrLp20R14aQWQQe9nnqPgRZ Q+Uvinwn+DJlgMEjnf7MDusitU8V+pZ/joe2f2ms8JEYXT+wzuNPhj1j9O5lsitF6gjr J7fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Nj5Zcto19jAG/2+iUH/d35sA52Q6CmVF9ht5DaMNdYM=; b=aLlKfLSmI4q90/4YMNA4ytARGixQmFA8zqr16aYloCA0FJzZfyslMcI5Pv850EtYni Bv5/P/vOLinSXee0F5A87P7RbsEKnu6cy/tdYvQQXvIrWaW6VEcgh2fXZG1U5XwXftM5 J7aVvKdImNfyKjoSxb0+cfwQ7Cdt4/YobKf2Jpb3lOubrxi4rT/cuNJzEU56xHvV+rzw ef4RGRTnOvdu/sBWOxHtWk0bjgW2erIAQi4vuw7cT0zjTr0CQ3A6fRwh5gGyBwt0Tmfj ubunyrsHYPoqndT+WwG0D0t58t/O8BT2kIEtKhTMO2Q4bg+svbDIlld5GXn+5KFmz+ZO Ticg== X-Gm-Message-State: AGi0PuaPk3hiPr8TxUudSCwfNfS4bA9KwQ773yHEkKim6lETw48tzvHe 4MIqPijQkB3czFq8YXv6X0qwK5TAtQk= X-Google-Smtp-Source: APiQypITE9yabz3G4hFYKTw0prhnaEfulZIpa5BSHe54Tx/M2FBfaiqkCE2K3MOSUx1lJ6lEynO5FA== X-Received: by 2002:aa7:957d:: with SMTP id x29mr6882762pfq.304.1586342809873; Wed, 08 Apr 2020 03:46:49 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:49 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 36/46] e2fsck: allow admin specify number of threads Date: Wed, 8 Apr 2020 19:45:04 +0900 Message-Id: <1586342714-12536-37-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 1 + e2fsck/pass1.c | 8 ++++++-- e2fsck/unix.c | 3 ++- tests/f_multithread/script | 2 +- tests/f_multithread_completion/script | 2 +- tests/f_multithread_logfile/script | 2 +- tests/f_multithread_no/script | 2 +- tests/f_multithread_preen/script | 2 +- tests/f_multithread_yes/script | 2 +- 9 files changed, 15 insertions(+), 9 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 4e156f17..cfe045a1 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -448,6 +448,7 @@ struct e2fsck_struct { __u32 fs_ext_attr_inodes; __u32 fs_ext_attr_blocks; __u32 extent_depth_count[MAX_EXTENT_DEPTH_COUNT]; + __u32 fs_num_threads; /* serialize fix operation for multiple threads */ pthread_mutex_t fs_fix_mutex; /* protect block_found_map, block_dup_map */ diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 1e98f8b6..7320d85f 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -3127,11 +3127,14 @@ static void init_ext2_max_sizes() static void e2fsck_pass1_multithread(e2fsck_t global_ctx) { struct e2fsck_thread_info *infos = NULL; - int num_threads = 1; + int num_threads = global_ctx->fs_num_threads; errcode_t retval; unsigned flexbg_size = 1; int max_threads; + if (num_threads < 1) + num_threads = 1; + retval = _e2fsck_pass1_prepare(global_ctx); if (retval) goto out_abort; @@ -3149,11 +3152,12 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) int times = max_threads / num_threads; if (times == 0) - num_threads = 1; + num_threads = max_threads; else num_threads = max_threads / times; } + global_ctx->fs_num_threads = num_threads; init_ext2_max_sizes(); retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); if (retval) { diff --git a/e2fsck/unix.c b/e2fsck/unix.c index fff7376c..79800a98 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -848,7 +848,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) phys_mem_kb = get_memory_size() / 1024; ctx->readahead_kb = ~0ULL; - while ((c = getopt(argc, argv, "pamnyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) + while ((c = getopt(argc, argv, "pam:nyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) switch (c) { case 'C': ctx->progress = e2fsck_update_progress; @@ -891,6 +891,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) break; case 'm': ctx->options |= E2F_OPT_MULTITHREAD; + ctx->fs_num_threads = atoi(optarg); break; case 'n': if (ctx->options & (E2F_OPT_YES|E2F_OPT_PREEN)) diff --git a/tests/f_multithread/script b/tests/f_multithread/script index 0fe96cd0..83cd0f03 100644 --- a/tests/f_multithread/script +++ b/tests/f_multithread/script @@ -1,4 +1,4 @@ -FSCK_OPT="-fy -m" +FSCK_OPT="-fy -m1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_completion/script b/tests/f_multithread_completion/script index bf23cd61..0ec13816 100644 --- a/tests/f_multithread_completion/script +++ b/tests/f_multithread_completion/script @@ -1,4 +1,4 @@ -FSCK_OPT="-fy -m -C 1" +FSCK_OPT="-fy -m1 -C 1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_logfile/script b/tests/f_multithread_logfile/script index d7042a03..ae497298 100644 --- a/tests/f_multithread_logfile/script +++ b/tests/f_multithread_logfile/script @@ -1,5 +1,5 @@ LOG_FNAME="f_multithread_logfile_xxx" -FSCK_OPT="-fy -m -y -E log_filename=$LOG_FNAME" +FSCK_OPT="-fy -m1 -y -E log_filename=$LOG_FNAME" SKIP_VERIFY="true" ONE_PASS_ONLY="true" SKIP_CLEANUP="true" diff --git a/tests/f_multithread_no/script b/tests/f_multithread_no/script index b93deb3a..db791e11 100644 --- a/tests/f_multithread_no/script +++ b/tests/f_multithread_no/script @@ -1,4 +1,4 @@ -FSCK_OPT="-fn -m" +FSCK_OPT="-fn -m1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_preen/script b/tests/f_multithread_preen/script index ecb79cd6..8965f4a7 100644 --- a/tests/f_multithread_preen/script +++ b/tests/f_multithread_preen/script @@ -1,4 +1,4 @@ -FSCK_OPT="-fp -m" +FSCK_OPT="-fp -m1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_yes/script b/tests/f_multithread_yes/script index 38891f6a..8b4aa9b8 100644 --- a/tests/f_multithread_yes/script +++ b/tests/f_multithread_yes/script @@ -1,4 +1,4 @@ -FSCK_OPT="-f -m" +FSCK_OPT="-f -m1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck From patchwork Wed Apr 8 10:45:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267898 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Ok+aQUIb; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Df3mLFz9sSG for ; Wed, 8 Apr 2020 20:46:58 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728309AbgDHKq4 (ORCPT ); Wed, 8 Apr 2020 06:46:56 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:36946 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKqz (ORCPT ); Wed, 8 Apr 2020 06:46:55 -0400 Received: by mail-pf1-f196.google.com with SMTP id u65so2232071pfb.4 for ; Wed, 08 Apr 2020 03:46:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=w0AODl7Eb3YSB/hFowqs3E8B7le5vdNgf7Buve3ErFk=; b=Ok+aQUIb3D9Qa1sSFMeAM3UnJhAVaRpMUEFysT5ziLxPe7oX2gQFSrLoQJb6TVtj4N GrXRiMFqLt4Oq86R8krj1/s2+/j+bfQVj3SV3m9q6SKjehAnnQequlM4FO/C66vOjNZC Lo2A7yt+20Gjqhb78K9XJwZcpPmWedbixsSox6A08ej4/h0tv66wI67RarQ0ycwIKnim uQaQ8LkOy09gZo3UpO9TyMjYxgghQMicFbFPV3wX0FfO5QCYH1pnUCKfR9zMBdNo2cNL JBbDkOqtkl55x8DYjZlnO3y1ziNwId4I54djOjJ6cub4zKMjtShQg6EAhVJi9nhXMZ+0 C1Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=w0AODl7Eb3YSB/hFowqs3E8B7le5vdNgf7Buve3ErFk=; b=kJ40qv+CzBtyrsnvD+byNA/Q2kXQ6WQ/Tr/QrJ4Qa1M5AV+XVuzqhp7Q1zAPyN1GP+ tle9FkICkY4i2/Te252cS0ELsgDL6rOKCSR5bDDQyUMyI9UHaDiB+bPDji8VUXBuuvbR bhpmGLg140rHKWold+w2As+to0o05g3S6HKiWKkgVZkP2vxuENRLwxCSiKxawGALaFqy 11lXTfrRfkB46L5RVix5JIUzVdCtYeqC+akVMfmJCLGj5+UVBV68NNZGJVZiIko+Sqhk SoYIQX+kPv9alcE/RKOSwFTlnQ4uX7bZVILKAOadpjKFCnh8RKmG4Dijiz5bStu0NRw3 3A2g== X-Gm-Message-State: AGi0Puaz1/6C0kAQGTvUr6rJnnArQx3Hn5Q/UUliZ91XavOMQjnbR/6Q ILauMSAR8gQyjRJcJDAOYVj7IApJ00Q= X-Google-Smtp-Source: APiQypJQ8g8qKIp5f08+G03wLvkooaV1d/9XGO8REs92kQdpXCgsr40kVd8e/6ggHO18iFo66STElA== X-Received: by 2002:aa7:8bd0:: with SMTP id s16mr7170940pfd.101.1586342812227; Wed, 08 Apr 2020 03:46:52 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:51 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 37/46] e2fsck: kickoff mutex lock for block found map Date: Wed, 8 Apr 2020 19:45:05 +0900 Message-Id: <1586342714-12536-38-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Now @block_found_map is no longer shared by multiple threads, and @block_dup_map need be checked again after threads finish. This patch also fix a bug in bitmap merging codes, this could make following pass skip many checkings and fixes.. Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 1 + e2fsck/pass1.c | 143 ++++++++++++++++-------------- lib/ext2fs/bitmaps.c | 5 +- lib/ext2fs/bitops.h | 2 + lib/ext2fs/blkmap64_rb.c | 38 ++++++-- lib/ext2fs/bmap64.h | 3 +- lib/ext2fs/ext2fs.h | 9 +- lib/ext2fs/gen_bitmap64.c | 49 +++++++++- lib/ext2fs/icount.c | 6 +- tests/f_itable_collision/expect.1 | 3 - 10 files changed, 170 insertions(+), 89 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index cfe045a1..26ef1d81 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -202,6 +202,7 @@ struct resource_track { #define E2F_FLAG_TIME_INSANE 0x2000 /* Time is insane */ #define E2F_FLAG_PROBLEMS_FIXED 0x4000 /* At least one problem was fixed */ #define E2F_FLAG_ALLOC_OK 0x8000 /* Can we allocate blocks? */ +#define E2F_FLAG_DUP_BLOCK 0x20000 /* dup block found during pass1 */ #define E2F_RESET_FLAGS (E2F_FLAG_TIME_INSANE | E2F_FLAG_PROBLEMS_FIXED) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 7320d85f..4654e673 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -712,6 +712,30 @@ static void check_inode_extra_space(e2fsck_t ctx, struct problem_context *pctx, } +static _INLINE_ int is_blocks_used(e2fsck_t ctx, blk64_t block, + unsigned int num) +{ + int retval; + + /* used to avoid duplicate output from below */ + retval = ext2fs_test_block_bitmap_range2_valid(ctx->block_found_map, + block, num); + if (!retval) + return 0; + + retval = ext2fs_test_block_bitmap_range2(ctx->block_found_map, block, num); + if (retval) { + if (ctx->global_ctx) + retval = ext2fs_test_block_bitmap_range2( + ctx->global_ctx->block_found_map, block, num); + if (retval) + return 0; + } + + return 1; +} + + /* * Check to see if the inode might really be a directory, despite i_mode * @@ -815,15 +839,10 @@ static void check_is_really_dir(e2fsck_t ctx, struct problem_context *pctx, if (i >= 4) not_device++; - e2fsck_pass1_block_map_lock(ctx); if (blk < ctx->fs->super->s_first_data_block || blk >= ext2fs_blocks_count(ctx->fs->super) || - ext2fs_fast_test_block_bitmap2(ctx->block_found_map, - blk)) { - e2fsck_pass1_block_map_unlock(ctx); + is_blocks_used(ctx, blk, 1)) return; /* Invalid block, can't be dir */ - } - e2fsck_pass1_block_map_unlock(ctx); } blk = inode->i_block[0]; } @@ -1294,6 +1313,19 @@ static int _e2fsck_pass1_prepare(e2fsck_t ctx) return pctx.errcode; } + pctx.errcode = e2fsck_allocate_block_bitmap(ctx->fs, + _("multiply claimed block map"), + EXT2FS_BMAP64_RBTREE, "block_dup_map", + &ctx->block_dup_map); + if (pctx.errcode) { + pctx.num = 3; + fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, + &pctx); + /* Should never get here */ + ctx->flags |= E2F_FLAG_ABORT; + return pctx.errcode; + } + if (ext2fs_has_feature_mmp(fs->super) && fs->super->s_mmp_block > fs->super->s_first_data_block && fs->super->s_mmp_block < ext2fs_blocks_count(fs->super)) @@ -1382,12 +1414,17 @@ static void _e2fsck_pass1_post(e2fsck_t ctx) return; if (ctx->block_dup_map) { + if (!(ctx->flags & E2F_FLAG_DUP_BLOCK)) { + ext2fs_free_mem(&block_buf); + return; + } if (ctx->options & E2F_OPT_PREEN) { clear_problem_context(&pctx); fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); } e2fsck_pass1_dupblocks(ctx, block_buf); ext2fs_free_mem(&block_buf); + ctx->flags &= ~E2F_FLAG_DUP_BLOCK; } } @@ -1901,10 +1938,8 @@ void _e2fsck_pass1(e2fsck_t ctx) failed_csum = 0; } - e2fsck_pass1_block_map_lock(ctx); - pctx.errcode = ext2fs_copy_bitmap(ctx->block_found_map, + pctx.errcode = ext2fs_copy_bitmap(ctx->global_ctx->block_found_map, &pb.fs_meta_blocks); - e2fsck_pass1_block_map_unlock(ctx); if (pctx.errcode) { pctx.num = 4; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); @@ -2296,7 +2331,7 @@ do { \ _src->_map_field = NULL; \ } else { \ _ret = ext2fs_merge_bitmap(_src->_map_field, \ - _dest->_map_field); \ + _dest->_map_field, NULL);\ if (_ret) \ return _ret; \ } \ @@ -2313,7 +2348,7 @@ do { \ _src->_map_field = NULL; \ } else { \ _ret = ext2fs_merge_bitmap(_src->_map_field, \ - _dest->_map_field); \ + _dest->_map_field, NULL);\ if (_ret) \ return _ret; \ } \ @@ -2613,7 +2648,7 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, assert(global_ctx->inode_reg_map == NULL); assert(global_ctx->inodes_to_rebuild == NULL); - assert(global_ctx->block_dup_map == NULL); + assert(global_ctx->block_dup_map != NULL); assert(global_ctx->block_found_map != NULL); assert(global_ctx->block_metadata_map != NULL); assert(global_ctx->block_ea_map == NULL); @@ -2624,8 +2659,15 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, return retval; } memcpy(thread_context, global_ctx, sizeof(struct e2fsck_struct)); - thread_context->global_ctx = global_ctx; + thread_context->block_dup_map = NULL; + retval = e2fsck_allocate_block_bitmap(global_ctx->fs, + _("in-use block map"), EXT2FS_BMAP64_RBTREE, + "block_found_map", &thread_context->block_found_map); + if (retval) + goto out_context; + + thread_context->global_ctx = global_ctx; retval = ext2fs_get_mem(sizeof(struct struct_ext2_filsys), &thread_fs); if (retval) { com_err(global_ctx->program_name, retval, "while allocating memory"); @@ -2675,6 +2717,8 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, out_fs: ext2fs_free_mem(&thread_fs); out_context: + if (thread_context->block_found_map) + ext2fs_free_mem(&thread_context->block_found_map); ext2fs_free_mem(&thread_context); return retval; } @@ -2712,7 +2756,6 @@ static void e2fsck_pass1_merge_dx_dir(e2fsck_t global_ctx, e2fsck_t thread_ctx) e2fsck_merge_dx_dir(global_ctx, thread_ctx); } - #define PASS1_MERGE_CTX_ICOUNT(_dest, _src, _field) \ do { \ if (_src->_field) { \ @@ -2785,7 +2828,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; - ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; ext2_icount_t inode_count = global_ctx->inode_count; ext2_icount_t inode_link_info = global_ctx->inode_link_info; @@ -2820,6 +2862,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_refcount_t ea_block_quota_inodes = global_ctx->ea_block_quota_inodes; ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; + ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; + ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2838,6 +2882,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->inodes_to_rebuild = inodes_to_rebuild; global_ctx->inode_reg_map = inode_reg_map; global_ctx->block_dup_map = block_dup_map; + global_ctx->block_found_map = block_found_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); global_ctx->dx_dir_info = dx_dir_info; @@ -2895,6 +2940,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx } global_ctx->qctx = qctx; e2fsck_pass1_merge_quota_ctx(global_ctx, thread_ctx); + retval = ext2fs_merge_bitmap(thread_ctx->block_found_map, + global_ctx->block_found_map, + global_ctx->block_dup_map); + e2fsck_pass1_block_map_unlock(global_ctx); + if (retval == EEXIST) + global_ctx->flags |= E2F_FLAG_DUP_BLOCK; global_ctx->invalid_block_bitmap_flag = invalid_block_bitmap_flag; global_ctx->invalid_inode_bitmap_flag = invalid_inode_bitmap_flag; global_ctx->invalid_inode_table_flag = invalid_inode_table_flag; @@ -2933,6 +2984,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) PASS1_FREE_CTX_BITMAP(thread_ctx, inode_imagic_map); PASS1_FREE_CTX_BITMAP(thread_ctx, inode_reg_map); PASS1_FREE_CTX_BITMAP(thread_ctx, inodes_to_rebuild); + PASS1_FREE_CTX_BITMAP(thread_ctx, block_found_map); ext2fs_free_icount(thread_ctx->inode_count); ext2fs_free_icount(thread_ctx->inode_link_info); ext2fs_free_mem(&thread_ctx); @@ -3368,38 +3420,18 @@ static void alloc_imagic_map(e2fsck_t ctx) static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) { struct problem_context pctx; - e2fsck_t global_ctx; - - global_ctx = ctx->global_ctx; - if (!global_ctx) - global_ctx = ctx; + e2fsck_t global_ctx = ctx->global_ctx ? ctx->global_ctx : ctx; clear_problem_context(&pctx); - - if (ext2fs_fast_test_block_bitmap2(ctx->block_found_map, block)) { + if (is_blocks_used(ctx, block, 1)) { if (ext2fs_has_feature_shared_blocks(ctx->fs->super) && !(ctx->options & E2F_OPT_UNSHARE_BLOCKS)) { return; } - /** - * this should be safe because this operation has - * been serialized by mutex. - */ - if (!global_ctx->block_dup_map) { - pctx.errcode = e2fsck_allocate_block_bitmap(ctx->fs, - _("multiply claimed block map"), - EXT2FS_BMAP64_RBTREE, "block_dup_map", - &global_ctx->block_dup_map); - if (pctx.errcode) { - pctx.num = 3; - fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, - &pctx); - /* Should never get here */ - ctx->flags |= E2F_FLAG_ABORT; - return; - } - } + ctx->flags |= E2F_FLAG_DUP_BLOCK; + e2fsck_pass1_block_map_lock(ctx); ext2fs_fast_mark_block_bitmap2(global_ctx->block_dup_map, block); + e2fsck_pass1_block_map_unlock(ctx); } else { ext2fs_fast_mark_block_bitmap2(ctx->block_found_map, block); } @@ -3412,8 +3444,7 @@ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) static _INLINE_ void mark_blocks_used(e2fsck_t ctx, blk64_t block, unsigned int num) { - e2fsck_pass1_block_map_lock(ctx); - if (ext2fs_test_block_bitmap_range2(ctx->block_found_map, block, num)) { + if (!is_blocks_used(ctx, block, num)) { ext2fs_mark_block_bitmap_range2(ctx->block_found_map, block, num); } else { unsigned int i; @@ -3421,7 +3452,6 @@ static _INLINE_ void mark_blocks_used(e2fsck_t ctx, blk64_t block, for (i = 0; i < num; i += EXT2FS_CLUSTER_RATIO(ctx->fs)) mark_block_used(ctx, block + i); } - e2fsck_pass1_block_map_unlock(ctx); } /* @@ -3747,9 +3777,7 @@ refcount_fail: inc_ea_inode_refs(global_ctx, pctx, first, end); ea_refcount_store(global_ctx->refcount, blk, header->h_refcount - 1); - e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); - e2fsck_pass1_block_map_unlock(ctx); ext2fs_fast_mark_block_bitmap2(global_ctx->block_ea_map, blk); e2fsck_pass1_ea_unlock(ctx); return 1; @@ -4135,9 +4163,7 @@ report_problem: pctx->str = "EXT2_EXTENT_UP"; return; } - e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); - e2fsck_pass1_block_map_unlock(ctx); pb->num_blocks++; goto next; } @@ -4244,7 +4270,6 @@ alloc_later: pb->last_block, extent.e_pblk, extent.e_lblk)) { - e2fsck_pass1_block_map_lock(ctx); for (i = 0; i < extent.e_len; i++) { pctx->blk = extent.e_lblk + i; pctx->blk2 = extent.e_pblk + i; @@ -4252,7 +4277,6 @@ alloc_later: mark_block_used(ctx, extent.e_pblk + i); mark_block_used(ctx, extent.e_pblk + i); } - e2fsck_pass1_block_map_unlock(ctx); } /* @@ -4938,9 +4962,7 @@ static int process_block(ext2_filsys fs, * by mark_table_blocks()). */ if (blockcnt == BLOCK_COUNT_DIND) { - e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); - e2fsck_pass1_block_map_unlock(ctx); } p->num_blocks++; } else if (!(ctx->fs->cluster_ratio_bits && @@ -4949,19 +4971,15 @@ static int process_block(ext2_filsys fs, EXT2FS_B2C(ctx->fs, p->previous_block)) && (blk & EXT2FS_CLUSTER_MASK(ctx->fs)) == ((unsigned) blockcnt & EXT2FS_CLUSTER_MASK(ctx->fs)))) { - e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); - e2fsck_pass1_block_map_unlock(ctx); p->num_blocks++; } else if (has_unaligned_cluster_map(ctx, p->previous_block, p->last_block, blk, blockcnt)) { pctx->blk = blockcnt; pctx->blk2 = blk; fix_problem(ctx, PR_1_MISALIGNED_CLUSTER, pctx); - e2fsck_pass1_block_map_lock(ctx); mark_block_used(ctx, blk); mark_block_used(ctx, blk); - e2fsck_pass1_block_map_unlock(ctx); } if (blockcnt >= 0) p->last_block = blockcnt; @@ -5028,31 +5046,25 @@ static int process_bad_block(ext2_filsys fs, } if (blockcnt < 0) { - e2fsck_pass1_block_map_lock(ctx); if (ext2fs_test_block_bitmap2(p->fs_meta_blocks, blk)) { p->bbcheck = 1; if (fix_problem(ctx, PR_1_BB_FS_BLOCK, pctx)) { *block_nr = 0; - e2fsck_pass1_block_map_unlock(ctx); return BLOCK_CHANGED; } - } else if (ext2fs_test_block_bitmap2(ctx->block_found_map, - blk)) { + } else if (is_blocks_used(ctx, blk, 1)) { p->bbcheck = 1; if (fix_problem(ctx, PR_1_BBINODE_BAD_METABLOCK, pctx)) { *block_nr = 0; - e2fsck_pass1_block_map_unlock(ctx); return BLOCK_CHANGED; } if (e2fsck_should_abort(ctx)) { - e2fsck_pass1_block_map_unlock(ctx); return BLOCK_ABORT; } } else { mark_block_used(ctx, blk); } - e2fsck_pass1_block_map_unlock(ctx); return 0; } #if 0 @@ -5065,13 +5077,10 @@ static int process_bad_block(ext2_filsys fs, * there's an overlap between the filesystem table blocks * (bitmaps and inode table) and the bad block list. */ - e2fsck_pass1_block_map_lock(ctx); - if (!ext2fs_test_block_bitmap2(ctx->block_found_map, blk)) { + if (!is_blocks_used(ctx, blk, 1)) { ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); - e2fsck_pass1_block_map_unlock(ctx); return 0; } - e2fsck_pass1_block_map_unlock(ctx); /* * Try to find the where the filesystem block was used... */ @@ -5226,7 +5235,6 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group, fix_problem(ctx, (old_block ? PR_1_RELOC_FROM_TO : PR_1_RELOC_TO), &pctx); pctx.blk2 = 0; - e2fsck_pass1_block_map_lock(ctx); for (i = 0; i < num; i++) { pctx.blk = i; ext2fs_mark_block_bitmap2(ctx->block_found_map, (*new_block)+i); @@ -5247,7 +5255,6 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group, if (pctx.errcode) fix_problem(ctx, PR_1_RELOC_WRITE_ERR, &pctx); } - e2fsck_pass1_block_map_unlock(ctx); ext2fs_free_mem(&buf); } diff --git a/lib/ext2fs/bitmaps.c b/lib/ext2fs/bitmaps.c index baa7c627..01a3b7cb 100644 --- a/lib/ext2fs/bitmaps.c +++ b/lib/ext2fs/bitmaps.c @@ -47,9 +47,10 @@ errcode_t ext2fs_copy_bitmap(ext2fs_generic_bitmap src, } errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src, - ext2fs_generic_bitmap dest) + ext2fs_generic_bitmap dest, + ext2fs_generic_bitmap dup) { - return ext2fs_merge_generic_bmap(src, dest); + return ext2fs_merge_generic_bmap(src, dest, dup); } void ext2fs_set_bitmap_padding(ext2fs_generic_bitmap map) diff --git a/lib/ext2fs/bitops.h b/lib/ext2fs/bitops.h index 505b3c9c..1facc8dd 100644 --- a/lib/ext2fs/bitops.h +++ b/lib/ext2fs/bitops.h @@ -120,6 +120,8 @@ extern int ext2fs_test_block_bitmap_range(ext2fs_block_bitmap bitmap, extern void ext2fs_set_bitmap_padding(ext2fs_generic_bitmap map); extern __u32 ext2fs_get_generic_bitmap_start(ext2fs_generic_bitmap bitmap); extern __u32 ext2fs_get_generic_bitmap_end(ext2fs_generic_bitmap bitmap); +extern int ext2fs_test_block_bitmap_range2_valid(ext2fs_block_bitmap bitmap, + blk64_t block, unsigned int num); /* 64-bit versions */ diff --git a/lib/ext2fs/blkmap64_rb.c b/lib/ext2fs/blkmap64_rb.c index 42a10536..2337302f 100644 --- a/lib/ext2fs/blkmap64_rb.c +++ b/lib/ext2fs/blkmap64_rb.c @@ -969,27 +969,53 @@ static void rb_print_stats(ext2fs_generic_bitmap_64 bitmap EXT2FS_ATTR((unused)) #endif static errcode_t rb_merge_bmap(ext2fs_generic_bitmap_64 src, - ext2fs_generic_bitmap_64 dest) + ext2fs_generic_bitmap_64 dest, + ext2fs_generic_bitmap_64 dup) { - struct ext2fs_rb_private *src_bp, *dest_bp; + struct ext2fs_rb_private *src_bp, *dest_bp, *dup_bp = NULL; struct bmap_rb_extent *src_ext; struct rb_node *src_node; - errcode_t retval = 0; + int retval = 0; + int dup_found = 0; src_bp = (struct ext2fs_rb_private *) src->private; dest_bp = (struct ext2fs_rb_private *) dest->private; + if (dup) + dup_bp = (struct ext2fs_rb_private *) dup->private; src_bp->rcursor = NULL; dest_bp->rcursor = NULL; src_node = ext2fs_rb_first(&src_bp->root); while (src_node) { src_ext = node_to_extent(src_node); - rb_insert_extent(src_ext->start, src_ext->count, dest_bp); - + retval = rb_test_clear_bmap_extent(dest, + src_ext->start + src->start, + src_ext->count); + if (retval) { + rb_insert_extent(src_ext->start, src_ext->count, + dest_bp); + } else { + /* unlikely case, do it one by one block */ + __u64 i; + + for (i = src_ext->start; + i < src_ext->start + src_ext->count; i++) { + retval = rb_test_clear_bmap_extent(dest, i + src->start, 1); + if (retval) { + rb_insert_extent(i, 1, dest_bp); + } else { + if (dup_bp) + rb_insert_extent(i, 1, dup_bp); + dup_found = 1; + } + } + } src_node = ext2fs_rb_next(src_node); } - return retval; + if (dup_found && dup) + return EEXIST; + return 0; } struct ext2_bitmap_ops ext2fs_blkmap64_rbtree = { diff --git a/lib/ext2fs/bmap64.h b/lib/ext2fs/bmap64.h index 09a5886b..68a4bb0a 100644 --- a/lib/ext2fs/bmap64.h +++ b/lib/ext2fs/bmap64.h @@ -73,7 +73,8 @@ struct ext2_bitmap_ops { errcode_t (*copy_bmap)(ext2fs_generic_bitmap_64 src, ext2fs_generic_bitmap_64 dest); errcode_t (*merge_bmap)(ext2fs_generic_bitmap_64 src, - ext2fs_generic_bitmap_64 dest); + ext2fs_generic_bitmap_64 dest, + ext2fs_generic_bitmap_64 dup); errcode_t (*resize_bmap)(ext2fs_generic_bitmap_64 bitmap, __u64 new_end, __u64 new_real_end); diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 6c872ed1..8121b642 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -839,8 +839,10 @@ extern void ext2fs_free_block_bitmap(ext2fs_block_bitmap bitmap); extern void ext2fs_free_inode_bitmap(ext2fs_inode_bitmap bitmap); extern errcode_t ext2fs_copy_bitmap(ext2fs_generic_bitmap src, ext2fs_generic_bitmap *dest); -errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src, - ext2fs_generic_bitmap dest); + +extern errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src, + ext2fs_generic_bitmap dest, + ext2fs_generic_bitmap dup); extern errcode_t ext2fs_write_inode_bitmap(ext2_filsys fs); extern errcode_t ext2fs_write_block_bitmap (ext2_filsys fs); extern errcode_t ext2fs_read_inode_bitmap (ext2_filsys fs); @@ -1439,7 +1441,8 @@ errcode_t ext2fs_resize_generic_bmap(ext2fs_generic_bitmap bmap, __u64 new_end, __u64 new_real_end); errcode_t ext2fs_merge_generic_bmap(ext2fs_generic_bitmap gen_src, - ext2fs_generic_bitmap gen_dest); + ext2fs_generic_bitmap gen_dest, + ext2fs_generic_bitmap gen_dup); errcode_t ext2fs_compare_generic_bmap(errcode_t neq, ext2fs_generic_bitmap bm1, ext2fs_generic_bitmap bm2); diff --git a/lib/ext2fs/gen_bitmap64.c b/lib/ext2fs/gen_bitmap64.c index 78384e20..d354cffa 100644 --- a/lib/ext2fs/gen_bitmap64.c +++ b/lib/ext2fs/gen_bitmap64.c @@ -340,24 +340,32 @@ errcode_t ext2fs_copy_generic_bmap(ext2fs_generic_bitmap gen_src, } errcode_t ext2fs_merge_generic_bmap(ext2fs_generic_bitmap gen_src, - ext2fs_generic_bitmap gen_dest) + ext2fs_generic_bitmap gen_dest, + ext2fs_generic_bitmap gen_dup) { ext2fs_generic_bitmap_64 src = (ext2fs_generic_bitmap_64) gen_src; ext2fs_generic_bitmap_64 dest = (ext2fs_generic_bitmap_64) gen_dest; + ext2fs_generic_bitmap_64 dup = (ext2fs_generic_bitmap_64) gen_dup; if (!src || !dest) return EINVAL; - if (!EXT2FS_IS_64_BITMAP(src) || !EXT2FS_IS_64_BITMAP(dest)) + if (!EXT2FS_IS_64_BITMAP(src) || !EXT2FS_IS_64_BITMAP(dest) || + (dup && !EXT2FS_IS_64_BITMAP(dup))) return EINVAL; - if (src->bitmap_ops != dest->bitmap_ops) + if (src->bitmap_ops != dest->bitmap_ops || + (dup && src->bitmap_ops != dup->bitmap_ops)) + return EINVAL; + + if (src->cluster_bits != dest->cluster_bits || + (dup && dup->cluster_bits != src->cluster_bits)) return EINVAL; if (src->bitmap_ops->merge_bmap == NULL) return EOPNOTSUPP; - return src->bitmap_ops->merge_bmap(src, dest); + return src->bitmap_ops->merge_bmap(src, dest, dup); } errcode_t ext2fs_resize_generic_bmap(ext2fs_generic_bitmap gen_bmap, @@ -712,6 +720,39 @@ int ext2fs_test_block_bitmap_range2(ext2fs_block_bitmap gen_bmap, return bmap->bitmap_ops->test_clear_bmap_extent(bmap, block, num); } +int ext2fs_test_block_bitmap_range2_valid(ext2fs_block_bitmap bitmap, + blk64_t block, unsigned int num) +{ + ext2fs_generic_bitmap_64 bmap = (ext2fs_generic_bitmap_64)bitmap; + __u64 end = block + num; + + if (!bmap) + return 0; + + if (EXT2FS_IS_32_BITMAP(bmap)) { + if ((block & ~0xffffffffULL) || + ((block+num-1) & ~0xffffffffULL)) { + return 0; + } + } + + if (!EXT2FS_IS_64_BITMAP(bmap)) + return 0; + + /* convert to clusters if necessary */ + block >>= bmap->cluster_bits; + end += (1 << bmap->cluster_bits) - 1; + end >>= bmap->cluster_bits; + num = end - block; + + if ((block < bmap->start) || (block > bmap->end) || + (block+num-1 > bmap->end)) + return 0; + + return 1; +} + + void ext2fs_mark_block_bitmap_range2(ext2fs_block_bitmap gen_bmap, blk64_t block, unsigned int num) { diff --git a/lib/ext2fs/icount.c b/lib/ext2fs/icount.c index a72b53b3..ebbc1243 100644 --- a/lib/ext2fs/icount.c +++ b/lib/ext2fs/icount.c @@ -784,12 +784,14 @@ errcode_t ext2fs_icount_merge(ext2_icount_t src, ext2_icount_t dest) if (src->fullmap) return ext2fs_icount_merge_full_map(src, dest); - retval = ext2fs_merge_bitmap(src->single, dest->single); + retval = ext2fs_merge_bitmap(src->single, + dest->single, NULL); if (retval) return retval; if (src->multiple) { - retval = ext2fs_merge_bitmap(src->multiple, dest->multiple); + retval = ext2fs_merge_bitmap(src->multiple, + dest->multiple, NULL); if (retval) return retval; } diff --git a/tests/f_itable_collision/expect.1 b/tests/f_itable_collision/expect.1 index 01c85d4d..7e98baa8 100644 --- a/tests/f_itable_collision/expect.1 +++ b/tests/f_itable_collision/expect.1 @@ -1,6 +1,5 @@ Pass 1: Checking inodes, blocks, and sizes Inode 12 block 37 conflicts with critical metadata, skipping block checks. -Illegal block number passed to ext2fs_test_block_bitmap #268435455 for in-use block map Illegal block number passed to ext2fs_mark_block_bitmap #268435455 for in-use block map Inode 12, i_blocks is 48, should be 56. Fix? yes @@ -27,9 +26,7 @@ Clear inode? yes Restarting e2fsck from the beginning... Pass 1: Checking inodes, blocks, and sizes Inode 12 block 37 conflicts with critical metadata, skipping block checks. -Illegal block number passed to ext2fs_test_block_bitmap #4294967294 for in-use block map Illegal block number passed to ext2fs_mark_block_bitmap #4294967294 for in-use block map -Illegal block number passed to ext2fs_test_block_bitmap #268435455 for in-use block map Illegal block number passed to ext2fs_mark_block_bitmap #268435455 for in-use block map Running additional passes to resolve blocks claimed by more than one inode... From patchwork Wed Apr 8 10:45:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267897 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=U4r769r6; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dd6LmLz9sSb for ; Wed, 8 Apr 2020 20:46:57 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728308AbgDHKq4 (ORCPT ); Wed, 8 Apr 2020 06:46:56 -0400 Received: from mail-pg1-f196.google.com ([209.85.215.196]:37017 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728303AbgDHKqz (ORCPT ); Wed, 8 Apr 2020 06:46:55 -0400 Received: by mail-pg1-f196.google.com with SMTP id r4so3151770pgg.4 for ; Wed, 08 Apr 2020 03:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6Lf08hFBZYUmWyGRfX/iPNL+UZiEF3Xj6CbBsYbLc84=; b=U4r769r6faQNIc6jBZx3L+mMTQlQlXv+0JQeyZVTk9xSTfadtJU/ABmOfcaKZMkEbl unbCjiObTPTS06F/oZG+8huUk+eGM/KFcJiOlQzDuJrjQLg8DKOsPktyJJDnmff2nWHJ eyK2vttFiljo6yoO9IWktm0khu97O3fj2OyOQj8xI0WYHvxj/k4mfYZhaWpSe3esioaR OfBY3NpNA87iN7UQzIdeMLpIpmUk81MpX9EGV+qTpAII+RzUvnH4DAjhPQqa5fBibGqH SuCD95z8PQbLl/fMy7peDNbF6lMvymhnAPUMv62EH4fSI/g92ZxE9YhtNNI7Q/kw2bxb nzKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6Lf08hFBZYUmWyGRfX/iPNL+UZiEF3Xj6CbBsYbLc84=; b=evvPw8TT6/UGC67lBNQ/jrvLhZfAXNQZxsPIqXoD1n/3bO2sA8CziJmxM8acwMC/1Q DHWJc4w85DhfJwaTm7/APDVyMz0iw5FugKL3bBTW9p5IIPA1AOXQu5rL/RwTSgV03cg7 wcf7Qx47n0/0Iiq6FXVwGMO64z2hF8jvD8JjdIHye/vTSki3S8JVk7RM+jOocF8hkqql 3mGzgqwxCgQIgGZ7VbZLbqIz3u3V9xSdsKbQAPDJuSFdXdkbaw6hfuOCqlQQ/82nC0ZP yJjE5/hTRf+7wA/OPLRf++RBGJ75+seChE6RRaQ7ATT3uo/GoqLbCiOp+Apn61wDXcnb rXIw== X-Gm-Message-State: AGi0PuaCWMEiXa4Jjx0N1BuAQdgyx0A8OlU27k6ZQnyKyqvdNLiPlhJf x79VwRcrNGwEcS52DWJeX06VZrtSWg4= X-Google-Smtp-Source: APiQypJqJlGah0ULaGb9t+g84RGc12O0Xz7Z+FimSv/RhbsbgjmXroLCJHWJ1mec2Qx30NdgPRGuGw== X-Received: by 2002:aa7:9f47:: with SMTP id h7mr7154118pfr.292.1586342814287; Wed, 08 Apr 2020 03:46:54 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:53 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 38/46] e2fsck: fix readahead for pfsck of pass1 Date: Wed, 8 Apr 2020 19:45:06 +0900 Message-Id: <1586342714-12536-39-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Several improvments for this patch: 1) move readahead_kb detection to preparing phase. 2) inode readahead should be aware of thread block group boundary. 3) make readahead_kb aware of multiple threads. Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 89 +++++++++++++++++++++++++++++++------------------- 1 file changed, 55 insertions(+), 34 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 4654e673..efd2e72d 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1173,13 +1173,14 @@ static void pass1_readahead(e2fsck_t ctx, dgrp_t *group, ext2_ino_t *next_ino) dgrp_t start = *group, grp; blk64_t blocks_to_read = 0; errcode_t err = EXT2_ET_INVALID_ARGUMENT; + dgrp_t grp_end = ctx->thread_info.et_group_end; if (ctx->readahead_kb == 0) goto out; /* Keep iterating groups until we have enough to readahead */ inodes_per_block = EXT2_INODES_PER_BLOCK(ctx->fs->super); - for (grp = start; grp < ctx->fs->group_desc_count; grp++) { + for (grp = start; grp < grp_end; grp++) { if (ext2fs_bg_flags_test(ctx->fs, grp, EXT2_BG_INODE_UNINIT)) continue; inodes_in_group = ctx->fs->super->s_inodes_per_group - @@ -1271,6 +1272,38 @@ static int e2fsck_should_abort(e2fsck_t ctx) return 0; } +/** + * Even though we could specify number of threads, + * but it might be more than the whole filesystem + * block groups, correct it here. + */ +static void e2fsck_pass1_set_thread_num(e2fsck_t ctx) +{ + unsigned flexbg_size = 1; + ext2_filsys fs = ctx->fs; + int num_threads = ctx->fs_num_threads; + int max_threads; + + if (num_threads < 1) + num_threads = 1; + + if (ext2fs_has_feature_flex_bg(fs->super)) + flexbg_size = 1 << fs->super->s_log_groups_per_flex; + + max_threads = fs->group_desc_count / flexbg_size; + if (max_threads == 0) + num_threads = 1; + else if (max_threads % num_threads) { + int times = max_threads / num_threads; + + if (times == 0) + num_threads = max_threads; + else + num_threads = max_threads / times; + } + ctx->fs_num_threads = num_threads; +} + /* * We need call mark_table_blocks() before multiple * thread start, since all known system blocks should be @@ -1280,6 +1313,20 @@ static int _e2fsck_pass1_prepare(e2fsck_t ctx) { struct problem_context pctx; ext2_filsys fs = ctx->fs; + unsigned long long readahead_kb; + + e2fsck_pass1_set_thread_num(ctx); + /* If we can do readahead, figure out how many groups to pull in. */ + if (!e2fsck_can_readahead(ctx->fs)) + ctx->readahead_kb = 0; + else if (ctx->readahead_kb == ~0ULL) + ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs); + + /* don't use more than 1/10 of memory for threads checking */ + readahead_kb = get_memory_size() / (10 * ctx->fs_num_threads); + /* maybe better disable RA if this is too small? */ + if (ctx->readahead_kb > readahead_kb) + ctx->readahead_kb = readahead_kb; clear_problem_context(&pctx); if (!(ctx->options & E2F_OPT_PREEN)) @@ -1460,13 +1507,7 @@ void _e2fsck_pass1(e2fsck_t ctx) init_resource_track(&rtrack, ctx->fs->io); clear_problem_context(&pctx); - /* If we can do readahead, figure out how many groups to pull in. */ - if (!e2fsck_can_readahead(ctx->fs)) - ctx->readahead_kb = 0; - else if (ctx->readahead_kb == ~0ULL) - ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs); pass1_readahead(ctx, &ra_group, &ino_threshold); - if (ext2fs_has_feature_dir_index(fs->super) && !(ctx->options & E2F_OPT_NO)) { if (ext2fs_u32_list_create(&ctx->dirs_to_hash, 50)) @@ -2993,7 +3034,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) } static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, - int num_threads, e2fsck_t global_ctx) + e2fsck_t global_ctx) { errcode_t rc; errcode_t ret = 0; @@ -3002,7 +3043,7 @@ static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, /* merge invalid bitmaps will recalculate it */ global_ctx->invalid_bitmaps = 0; - for (i = 0; i < num_threads; i++) { + for (i = 0; i < global_ctx->fs_num_threads; i++) { pinfo = &infos[i]; if (!pinfo->eti_started) @@ -3079,7 +3120,7 @@ out: } static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, - int num_threads, e2fsck_t global_ctx) + e2fsck_t global_ctx) { struct e2fsck_thread_info *infos; pthread_attr_t attr; @@ -3094,6 +3135,7 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, thread_debug.etd_finished_threads = 0; #endif + int num_threads = global_ctx->fs_num_threads; retval = pthread_attr_init(&attr); if (retval) { @@ -3148,7 +3190,7 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, } if (retval) { - e2fsck_pass1_threads_join(infos, num_threads, global_ctx); + e2fsck_pass1_threads_join(infos, global_ctx); return retval; } *pinfo = infos; @@ -3179,13 +3221,7 @@ static void init_ext2_max_sizes() static void e2fsck_pass1_multithread(e2fsck_t global_ctx) { struct e2fsck_thread_info *infos = NULL; - int num_threads = global_ctx->fs_num_threads; errcode_t retval; - unsigned flexbg_size = 1; - int max_threads; - - if (num_threads < 1) - num_threads = 1; retval = _e2fsck_pass1_prepare(global_ctx); if (retval) @@ -3194,31 +3230,16 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) pthread_mutex_init(&global_ctx->fs_fix_mutex, NULL); pthread_mutex_init(&global_ctx->fs_block_map_mutex, NULL); pthread_mutex_init(&global_ctx->fs_ea_mutex, NULL); - if (ext2fs_has_feature_flex_bg(global_ctx->fs->super)) - flexbg_size = 1 << global_ctx->fs->super->s_log_groups_per_flex; - - max_threads = global_ctx->fs->group_desc_count / flexbg_size; - if (max_threads == 0) - num_threads = 1; - else if (max_threads % num_threads) { - int times = max_threads / num_threads; - - if (times == 0) - num_threads = max_threads; - else - num_threads = max_threads / times; - } - global_ctx->fs_num_threads = num_threads; init_ext2_max_sizes(); - retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); + retval = e2fsck_pass1_threads_start(&infos, global_ctx); if (retval) { com_err(global_ctx->program_name, retval, _("while starting pass1 threads\n")); goto out_abort; } - retval = e2fsck_pass1_threads_join(infos, num_threads, global_ctx); + retval = e2fsck_pass1_threads_join(infos, global_ctx); if (retval) { com_err(global_ctx->program_name, retval, _("while joining pass1 threads\n")); From patchwork Wed Apr 8 10:45:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267899 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=nqzTHDqT; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dg2THrz9sSb for ; Wed, 8 Apr 2020 20:46:59 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728311AbgDHKq6 (ORCPT ); Wed, 8 Apr 2020 06:46:58 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:40661 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKq6 (ORCPT ); Wed, 8 Apr 2020 06:46:58 -0400 Received: by mail-pf1-f196.google.com with SMTP id c20so2225332pfi.7 for ; Wed, 08 Apr 2020 03:46:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TkttZMbI5YmsiyEYMKmULwkeHyD5ZLQHYlN7QL62d5o=; b=nqzTHDqT/tnoE4/g+ZgR9LRSRYSrOXR4vO9IpWimvMES+7saxwKe/tk0x9mr9po8dX PawI5YG6hxe//v54dKa5ZGd/wyP4YqKsQUlJuJrSM5wA6nv8Z8swx1sLtOSynb8R7fpt 0u2N0whNHOuqcgvGNiB3gVpambUsJgaZZP9Mcf452P+lJngpUHH/HEjqq/9wmD2fmkn9 4NiP825FzTTeHGYo7cdi56k8alV8pcFsozr8lRJIbqvfuG2E/wVC04om7rTca5S7DQC1 OcthbI3pbr6Ooei/J66bIkiUDDKDWqzw8ooYLaNQ/oYInyq0NEuzeo4FNpFt/GbOobDg a3lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TkttZMbI5YmsiyEYMKmULwkeHyD5ZLQHYlN7QL62d5o=; b=C3E0WKWRSEk31DADOq8Mq/Fq+2HzsegILuT/iPTC0CWJRlC0ApfJAT8Y45BeDbxnwD ntlqMn/fAs1fldCpx8fKtnTh9hdl5umhecc9oO4xju+heOLGdlzJePrQxSuJLrQ27HJW AWe2RV0/E402O8ADO8aLGvKiEpBjQpSSS8MXqkyu9I8hb/3YQBTkpNutyovMIV9qgefL /CqmmQJpuFEULYWO3u64SRaH7W75G3/El8s1Y5FOl5c4Sxwo6igIECwzKE1xFicUV6Zd T1c9SiUycu96aL1dvc/y1ob7aQi7PAXlQIeG/xhw3CDePqfcTrIOrygG7L5fLAJOZIEK uk5Q== X-Gm-Message-State: AGi0PubdlogjU+kaReJCS3LEOA9gntzIFXVlFkb8gu6P/UqCtR5pVlAW OnhFAfkw7VoCwwSq4zt+jjVUP10uKjc= X-Google-Smtp-Source: APiQypJ/HTanFVZ0U82iQpk479gF6YIYAQOaWgZSMMcmmTYw0Dwom3sZatWQJiH8dKPACtGKg95orQ== X-Received: by 2002:a62:5e86:: with SMTP id s128mr7354533pfb.157.1586342816428; Wed, 08 Apr 2020 03:46:56 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:55 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 39/46] e2fsck: kick off ea mutex lock from pfsck Date: Wed, 8 Apr 2020 19:45:07 +0900 Message-Id: <1586342714-12536-40-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong With this patch, we no longer share ea related refcounts globally, so that mutex lock for ea could be dropped, this is important for Lustre, since Lustre backend filesystem use xattrs heavily. Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 3 +- e2fsck/pass1.c | 279 +++++++++++++++++++++++++++++++++++++----------- 2 files changed, 217 insertions(+), 65 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 26ef1d81..b25ee666 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -367,6 +367,7 @@ struct e2fsck_struct { ext2_refcount_t refcount; ext2_refcount_t refcount_extra; + ext2_refcount_t refcount_orig; /* * Quota blocks and inodes to be charged for each ea block. @@ -454,8 +455,6 @@ struct e2fsck_struct { pthread_mutex_t fs_fix_mutex; /* protect block_found_map, block_dup_map */ pthread_mutex_t fs_block_map_mutex; - /* protect ea related structure */ - pthread_mutex_t fs_ea_mutex; }; #ifdef DEBUG_THREADS diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index efd2e72d..127b390d 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -161,18 +161,6 @@ static inline void e2fsck_pass1_block_map_unlock(e2fsck_t ctx) pthread_mutex_unlock(&global_ctx->fs_block_map_mutex); } -static inline void e2fsck_pass1_ea_lock(e2fsck_t ctx) -{ - e2fsck_get_lock_context(ctx); - pthread_mutex_lock(&global_ctx->fs_ea_mutex); -} - -static inline void e2fsck_pass1_ea_unlock(e2fsck_t ctx) -{ - e2fsck_get_lock_context(ctx); - pthread_mutex_unlock(&global_ctx->fs_ea_mutex); -} - /* * Free all memory allocated by pass1 in preparation for restarting * things. @@ -451,16 +439,15 @@ static void inc_ea_inode_refs(e2fsck_t ctx, struct problem_context *pctx, struct ext2_ext_attr_entry *first, void *end) { struct ext2_ext_attr_entry *entry; - e2fsck_t global_ctx = ctx->global_ctx ? ctx->global_ctx : ctx; for (entry = first; (void *)entry < end && !EXT2_EXT_IS_LAST_ENTRY(entry); entry = EXT2_EXT_ATTR_NEXT(entry)) { if (!entry->e_value_inum) continue; - if (!global_ctx->ea_inode_refs) { + if (!ctx->ea_inode_refs) { pctx->errcode = ea_refcount_create(0, - &global_ctx->ea_inode_refs); + &ctx->ea_inode_refs); if (pctx->errcode) { pctx->num = 4; fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); @@ -468,7 +455,7 @@ static void inc_ea_inode_refs(e2fsck_t ctx, struct problem_context *pctx, return; } } - ea_refcount_increment(global_ctx->ea_inode_refs, + ea_refcount_increment(ctx->ea_inode_refs, entry->e_value_inum, 0); } } @@ -595,10 +582,8 @@ fix: * EA(s) in automatic fashion -bzzz */ if (problem == 0 || !fix_problem(ctx, problem, pctx)) { - e2fsck_pass1_ea_lock(ctx); inc_ea_inode_refs(ctx, pctx, (struct ext2_ext_attr_entry *)start, end); - e2fsck_pass1_ea_unlock(ctx); return; } @@ -1410,16 +1395,6 @@ static void _e2fsck_pass1_post(e2fsck_t ctx) ctx->refcount_extra = 0; } - if (ctx->ea_block_quota_blocks) { - ea_refcount_free(ctx->ea_block_quota_blocks); - ctx->ea_block_quota_blocks = 0; - } - - if (ctx->ea_block_quota_inodes) { - ea_refcount_free(ctx->ea_block_quota_inodes); - ctx->ea_block_quota_inodes = 0; - } - if (ctx->invalid_bitmaps) handle_fs_bad_blocks(ctx); @@ -2315,6 +2290,16 @@ void _e2fsck_pass1(e2fsck_t ctx) /* We don't need the encryption policy => ID map any more */ destroy_encryption_policy_map(ctx); + if (ctx->ea_block_quota_blocks) { + ea_refcount_free(ctx->ea_block_quota_blocks); + ctx->ea_block_quota_blocks = 0; + } + + if (ctx->ea_block_quota_inodes) { + ea_refcount_free(ctx->ea_block_quota_inodes); + ctx->ea_block_quota_inodes = 0; + } + if (ctx->flags & E2F_FLAG_RESTART) { /* * Only the master copy of the superblock and block @@ -2854,6 +2839,155 @@ static void e2fsck_pass1_merge_quota_ctx(e2fsck_t global_ctx, e2fsck_t thread_ct quota_release_context(&thread_ctx->qctx); } +static errcode_t e2fsck_pass1_merge_ea_inode_refs(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + ea_value_t count; + blk64_t blk; + errcode_t retval; + + if (!thread_ctx->ea_inode_refs) + return 0; + + if (!global_ctx->ea_inode_refs) { + global_ctx->ea_inode_refs = thread_ctx->ea_inode_refs; + thread_ctx->ea_inode_refs = NULL; + return 0; + } + + ea_refcount_intr_begin(thread_ctx->ea_inode_refs); + while (1) { + if ((blk = ea_refcount_intr_next(thread_ctx->ea_inode_refs, + &count)) == 0) + break; + if (!global_ctx->block_ea_map || + !ext2fs_fast_test_block_bitmap2(global_ctx->block_ea_map, + blk)) { + retval = ea_refcount_store(global_ctx->ea_inode_refs, + blk, count); + if (retval) + return retval; + } + } + + return retval; +} + +static ea_value_t ea_refcount_usage(e2fsck_t ctx, blk64_t blk, + ea_value_t *orig) +{ + ea_value_t count_cur; + ea_value_t count_extra = 0; + ea_value_t count_orig; + + ea_refcount_fetch(ctx->refcount_orig, blk, &count_orig); + ea_refcount_fetch(ctx->refcount, blk, &count_cur); + /* most of time this is not needed */ + if (ctx->refcount_extra && count_cur == 0) + ea_refcount_fetch(ctx->refcount_extra, blk, &count_extra); + + if (orig) + *orig = count_orig; + + return count_orig + count_extra - count_cur; +} + +static errcode_t e2fsck_pass1_merge_ea_refcount(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + ea_value_t count; + blk64_t blk; + errcode_t retval = 0; + + if (!thread_ctx->refcount) + return 0; + + if (!global_ctx->refcount) { + global_ctx->refcount = thread_ctx->refcount; + thread_ctx->refcount = NULL; + global_ctx->refcount_extra = thread_ctx->refcount; + thread_ctx->refcount_extra = NULL; + return 0; + } + + ea_refcount_intr_begin(thread_ctx->refcount); + while (1) { + if ((blk = ea_refcount_intr_next(thread_ctx->refcount, + &count)) == 0) + break; + /** + * this EA has never seen before, so just store its + * refcount and refcount_extra into global_ctx if needed. + */ + if (!global_ctx->block_ea_map || + !ext2fs_fast_test_block_bitmap2(global_ctx->block_ea_map, + blk)) { + ea_value_t extra; + + retval = ea_refcount_store(global_ctx->refcount, + blk, count); + if (retval) + return retval; + + if (count > 0 || !thread_ctx->refcount_extra) + continue; + ea_refcount_fetch(thread_ctx->refcount_extra, blk, + &extra); + if (extra == 0) + continue; + + if (!global_ctx->refcount_extra) { + retval = ea_refcount_create(0, + &global_ctx->refcount_extra); + if (retval) + return retval; + } + retval = ea_refcount_store(global_ctx->refcount_extra, + blk, extra); + if (retval) + return retval; + + } else { + ea_value_t orig; + ea_value_t thread_usage; + ea_value_t global_usage; + ea_value_t new; + + thread_usage = ea_refcount_usage(thread_ctx, + blk, &orig); + global_usage = ea_refcount_usage(global_ctx, + blk, NULL); + if (thread_usage + global_usage <= orig) { + new = orig - thread_usage - global_usage; + retval = ea_refcount_store(global_ctx->refcount, + blk, new); + if (retval) + return retval; + continue; + } + /* update it is as zero */ + retval = ea_refcount_store(global_ctx->refcount, + blk, 0); + if (retval) + return retval; + /* Ooops, this EA was referenced more than it stated */ + if (!global_ctx->refcount_extra) { + retval = ea_refcount_create(0, + &global_ctx->refcount_extra); + if (retval) + return retval; + } + new = global_usage + thread_usage - orig; + retval = ea_refcount_store(global_ctx->refcount_extra, + blk, new); + if (retval) + return retval; + } + } + + return retval; +} + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2899,6 +3033,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx int invalid_bitmaps = global_ctx->invalid_bitmaps; ext2_refcount_t refcount = global_ctx->refcount; ext2_refcount_t refcount_extra = global_ctx->refcount_extra; + ext2_refcount_t refcount_orig = global_ctx->refcount_orig; ext2_refcount_t ea_block_quota_blocks = global_ctx->ea_block_quota_blocks; ext2_refcount_t ea_block_quota_inodes = global_ctx->ea_block_quota_inodes; ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; @@ -2934,6 +3069,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->inode_link_info = inode_link_info; global_ctx->refcount = refcount; global_ctx->refcount_extra = refcount_extra; + global_ctx->refcount_orig = refcount_orig; global_ctx->ea_block_quota_blocks = ea_block_quota_blocks; global_ctx->ea_block_quota_inodes = ea_block_quota_inodes; global_ctx->block_ea_map = block_ea_map; @@ -2979,6 +3115,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx com_err(global_ctx->program_name, 0, _("while merging dirs to hash\n")); return retval; } + e2fsck_pass1_merge_ea_inode_refs(global_ctx, thread_ctx); + e2fsck_pass1_merge_ea_refcount(global_ctx, thread_ctx); global_ctx->qctx = qctx; e2fsck_pass1_merge_quota_ctx(global_ctx, thread_ctx); retval = ext2fs_merge_bitmap(thread_ctx->block_found_map, @@ -3003,6 +3141,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_imagic_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_reg_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inodes_to_rebuild); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, block_ea_map); return 0; } @@ -3026,8 +3165,25 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) PASS1_FREE_CTX_BITMAP(thread_ctx, inode_reg_map); PASS1_FREE_CTX_BITMAP(thread_ctx, inodes_to_rebuild); PASS1_FREE_CTX_BITMAP(thread_ctx, block_found_map); + PASS1_FREE_CTX_BITMAP(thread_ctx, block_ea_map); ext2fs_free_icount(thread_ctx->inode_count); ext2fs_free_icount(thread_ctx->inode_link_info); + if (thread_ctx->refcount) { + ea_refcount_free(thread_ctx->refcount); + thread_ctx->refcount = NULL; + } + if (thread_ctx->refcount_extra) { + ea_refcount_free(thread_ctx->refcount_extra); + thread_ctx->refcount_extra = NULL; + } + if (thread_ctx->ea_inode_refs) { + ea_refcount_free(thread_ctx->ea_inode_refs); + thread_ctx->ea_inode_refs = NULL; + } + if (thread_ctx->refcount_orig) { + ea_refcount_free(thread_ctx->refcount_orig); + thread_ctx->refcount_orig = NULL; + } ext2fs_free_mem(&thread_ctx); return retval; @@ -3229,7 +3385,6 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) pthread_mutex_init(&global_ctx->fs_fix_mutex, NULL); pthread_mutex_init(&global_ctx->fs_block_map_mutex, NULL); - pthread_mutex_init(&global_ctx->fs_ea_mutex, NULL); init_ext2_max_sizes(); retval = e2fsck_pass1_threads_start(&infos, global_ctx); @@ -3564,30 +3719,35 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, } /* If ea bitmap hasn't been allocated, create it */ - e2fsck_pass1_ea_lock(ctx); - if (!global_ctx->block_ea_map) { + if (!ctx->block_ea_map) { pctx->errcode = e2fsck_allocate_block_bitmap(fs, _("ext attr block map"), EXT2FS_BMAP64_RBTREE, "block_ea_map", - &global_ctx->block_ea_map); + &ctx->block_ea_map); if (pctx->errcode) { pctx->num = 2; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, pctx); ctx->flags |= E2F_FLAG_ABORT; - e2fsck_pass1_ea_unlock(ctx); return 0; } } /* Create the EA refcount structure if necessary */ - if (!global_ctx->refcount) { + if (!ctx->refcount) { + pctx->errcode = ea_refcount_create(0, + &ctx->refcount); + if (pctx->errcode) { + pctx->num = 1; + fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); + ctx->flags |= E2F_FLAG_ABORT; + return 0; + } pctx->errcode = ea_refcount_create(0, - &global_ctx->refcount); + &ctx->refcount_orig); if (pctx->errcode) { pctx->num = 1; fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); ctx->flags |= E2F_FLAG_ABORT; - e2fsck_pass1_ea_unlock(ctx); return 0; } } @@ -3598,44 +3758,39 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, #endif /* Have we seen this EA block before? */ - if (ext2fs_fast_test_block_bitmap2(global_ctx->block_ea_map, + if (ext2fs_fast_test_block_bitmap2(ctx->block_ea_map, blk)) { ea_block_quota->blocks = EXT2FS_C2B(fs, 1); ea_block_quota->inodes = 0; - if (global_ctx->ea_block_quota_blocks) { - ea_refcount_fetch(global_ctx->ea_block_quota_blocks, + if (ctx->ea_block_quota_blocks) { + ea_refcount_fetch(ctx->ea_block_quota_blocks, blk, "a_blocks); if (quota_blocks) ea_block_quota->blocks = quota_blocks; } - if (global_ctx->ea_block_quota_inodes) - ea_refcount_fetch(global_ctx->ea_block_quota_inodes, + if (ctx->ea_block_quota_inodes) + ea_refcount_fetch(ctx->ea_block_quota_inodes, blk, &ea_block_quota->inodes); - if (ea_refcount_decrement(global_ctx->refcount, - blk, 0) == 0) { - e2fsck_pass1_ea_unlock(ctx); + if (ea_refcount_decrement(ctx->refcount, + blk, 0) == 0) return 1; - } /* Ooops, this EA was referenced more than it stated */ - if (!global_ctx->refcount_extra) { + if (!ctx->refcount_extra) { pctx->errcode = ea_refcount_create(0, - &global_ctx->refcount_extra); + &ctx->refcount_extra); if (pctx->errcode) { pctx->num = 2; fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); ctx->flags |= E2F_FLAG_ABORT; - e2fsck_pass1_ea_unlock(ctx); return 0; } } - ea_refcount_increment(global_ctx->refcount_extra, blk, 0); - e2fsck_pass1_ea_unlock(ctx); + ea_refcount_increment(ctx->refcount_extra, blk, 0); return 1; } - e2fsck_pass1_ea_unlock(ctx); /* * OK, we haven't seen this EA block yet. So we need to @@ -3762,45 +3917,43 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, return 0; } - e2fsck_pass1_ea_lock(ctx); if (quota_blocks != EXT2FS_C2B(fs, 1U)) { - if (!global_ctx->ea_block_quota_blocks) { + if (!ctx->ea_block_quota_blocks) { pctx->errcode = ea_refcount_create(0, - &global_ctx->ea_block_quota_blocks); + &ctx->ea_block_quota_blocks); if (pctx->errcode) { pctx->num = 3; goto refcount_fail; } } - ea_refcount_store(global_ctx->ea_block_quota_blocks, + ea_refcount_store(ctx->ea_block_quota_blocks, blk, quota_blocks); } if (quota_inodes) { - if (!global_ctx->ea_block_quota_inodes) { + if (!ctx->ea_block_quota_inodes) { pctx->errcode = ea_refcount_create(0, - &global_ctx->ea_block_quota_inodes); + &ctx->ea_block_quota_inodes); if (pctx->errcode) { pctx->num = 4; refcount_fail: fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); ctx->flags |= E2F_FLAG_ABORT; - e2fsck_pass1_ea_unlock(ctx); return 0; } } - ea_refcount_store(global_ctx->ea_block_quota_inodes, + ea_refcount_store(ctx->ea_block_quota_inodes, blk, quota_inodes); } ea_block_quota->blocks = quota_blocks; ea_block_quota->inodes = quota_inodes; - inc_ea_inode_refs(global_ctx, pctx, first, end); - ea_refcount_store(global_ctx->refcount, blk, header->h_refcount - 1); + inc_ea_inode_refs(ctx, pctx, first, end); + ea_refcount_store(ctx->refcount, blk, header->h_refcount - 1); + ea_refcount_store(ctx->refcount_orig, blk, header->h_refcount); mark_block_used(ctx, blk); - ext2fs_fast_mark_block_bitmap2(global_ctx->block_ea_map, blk); - e2fsck_pass1_ea_unlock(ctx); + ext2fs_fast_mark_block_bitmap2(ctx->block_ea_map, blk); return 1; clear_extattr: From patchwork Wed Apr 8 10:45:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267900 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=aQiFrvh4; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dh3hgvz9sSg for ; Wed, 8 Apr 2020 20:47:00 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728314AbgDHKrA (ORCPT ); Wed, 8 Apr 2020 06:47:00 -0400 Received: from mail-pj1-f65.google.com ([209.85.216.65]:35611 "EHLO mail-pj1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKq7 (ORCPT ); Wed, 8 Apr 2020 06:46:59 -0400 Received: by mail-pj1-f65.google.com with SMTP id mn19so374061pjb.0 for ; Wed, 08 Apr 2020 03:46:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3GhVIIliF2VmpaNbOu8hpb6WcYuxNJZdzLQZU/mZZJY=; b=aQiFrvh4DBcTu3m94B9FpzF+2NslKFjpYHfLLlAg0mn57YWC4PFndYlUMkVvYLZZms uhKGqLRSC4JIT2J9PpwSKeCoF//WhI8XGGalWr3q8zA6GY6yvm//K+PLG9c+8fCfyNQl naj3hRDHQfAqlxmvfQ4pLqTFIlxQgC1mnF6laYM1t4c1xVrI8T3FfsPn7hoKj+qUlxXo hGt68QgF0PTrJqGv03MbMXz4nHc+J5Lq4/Dqpav850jHBdtlDU+wcR9ueK9sXz+fAiUP aYyNqPkS22lPldThyK7qa/yfGXjXH/8wdKXsMnlGq7T/pEVl0KPQPXElfcrpae4LdAGU PPgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3GhVIIliF2VmpaNbOu8hpb6WcYuxNJZdzLQZU/mZZJY=; b=Vg0pa0ydCEl492375tTJj96mB61AepYb0amoP4z2s0rRYsrYnke0ssox43MfsL+TBg ni4atpEVM3IawpZXsDJk4vIJrDM5ap1NZG+QzPVgCaI1Ljc8MzWNXB3KN8tRqD03RM3u ylPL6p3jM6mnrVZaduZEiJZ05OiLRF2MczUaHnzIauG6QZ6prw+fNLdQy5M7kllEFaTF 0r4hX0l9vyfcFdLb3WjgqV/pqRXJx3VcLDseZN0pKovXlTQt9tjL38IABjkADvFcd2QE kNkjQCubxg8XEbSHj2VoES2dIzSqoft3C+WKPdLtJEQLWvhlFk9qq1U51YD8/QYfb+yp TnFQ== X-Gm-Message-State: AGi0PuYfhaS1u6TabjjyIKR5gmpdodELQQKCQiyVbybFbRSXmYwFJ1pg 7YgarWKTRUv/bnqzh1HoIofEJ1bnEtQ= X-Google-Smtp-Source: APiQypJQYOz+RjFZS60MLVsZnGuG9FKi601bHo3fRykVFdwbLeZjofnVNosZesvF+99GdKJ8HvoTCA== X-Received: by 2002:a17:902:9a08:: with SMTP id v8mr5523136plp.70.1586342818714; Wed, 08 Apr 2020 03:46:58 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:46:58 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 40/46] e2fsck: merge encrypted_files after threads finish Date: Wed, 8 Apr 2020 19:45:08 +0900 Message-Id: <1586342714-12536-41-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Signed-off-by: Wang Shilong --- e2fsck/e2fsck.h | 1 + e2fsck/encrypted_files.c | 175 +++++++++++++++++++++++++++++++++------ e2fsck/pass1.c | 13 ++- 3 files changed, 161 insertions(+), 28 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index b25ee666..3267f546 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -588,6 +588,7 @@ __u32 find_encryption_policy(e2fsck_t ctx, ext2_ino_t ino); void destroy_encryption_policy_map(e2fsck_t ctx); void destroy_encrypted_file_info(e2fsck_t ctx); +int merge_two_encrypted_files(e2fsck_t src_ctx, e2fsck_t dest_ctx); /* extents.c */ errcode_t e2fsck_rebuild_extents_later(e2fsck_t ctx, ext2_ino_t ino); diff --git a/e2fsck/encrypted_files.c b/e2fsck/encrypted_files.c index 16be2d6d..40540963 100644 --- a/e2fsck/encrypted_files.c +++ b/e2fsck/encrypted_files.c @@ -280,6 +280,9 @@ out: static int handle_nomem(e2fsck_t ctx, struct problem_context *pctx, size_t size_needed) { + if (!pctx) + return -ENOMEM; + pctx->num = size_needed; fix_problem(ctx, PR_1_ALLOCATE_ENCRYPTED_INODE_LIST, pctx); /* Should never get here */ @@ -287,11 +290,155 @@ static int handle_nomem(e2fsck_t ctx, struct problem_context *pctx, return 0; } +static int increase_file_ranges_capacity(e2fsck_t ctx, + struct encrypted_file_info *info, + struct problem_context *pctx) +{ + int size = sizeof(struct encrypted_file_range); + + if (info->file_ranges_count == info->file_ranges_capacity) { + /* Double the capacity by default. */ + size_t new_capacity = info->file_ranges_capacity * 2; + + /* ... but go from 0 to 128 right away. */ + if (new_capacity < 128) + new_capacity = 128; + + /* We won't need more than the filesystem's inode count. */ + if (new_capacity > ctx->fs->super->s_inodes_count) + new_capacity = ctx->fs->super->s_inodes_count; + + /* To be safe, ensure the capacity really increases. */ + if (new_capacity < info->file_ranges_capacity + 1) + new_capacity = info->file_ranges_capacity + 1; + + if (ext2fs_resize_mem(info->file_ranges_capacity * size, + new_capacity * size, &info->file_ranges) != 0) + return handle_nomem(ctx, pctx, + new_capacity * size); + + info->file_ranges_capacity = new_capacity; + } + + return 0; +} + +int find_entry_insert(e2fsck_t dest_ctx, + struct encrypted_file_range *insert_range) +{ + size_t l, r, m; + struct encrypted_file_range *range; + int merge_left = 0, merge_right = 0; + struct encrypted_file_info *dest_info = dest_ctx->encrypted_files; + int ret; + + l = 0; + r = dest_info->file_ranges_count; + if (r < 1) + return -EINVAL; + + while (l < r) { + m = l + (r - l) / 2; + range = &dest_info->file_ranges[m]; + + if (insert_range->first_ino < range->first_ino) + r = m; + else if (insert_range->first_ino > range->last_ino) + l = m + 1; + else /* should not happen */ { + return -EINVAL; + } + } + + /* check wheather it could be merged left */ + if (l >= 1) { + range = &dest_info->file_ranges[l - 1]; + if (range->last_ino + 1 == + insert_range->first_ino && + range->policy_id == insert_range->policy_id) { + range->last_ino = insert_range->last_ino; + merge_left = 1; + } + } + + /* check wheather it could be merged right */ + if (l < dest_info->file_ranges_count - 1) { + range = &dest_info->file_ranges[l + 1]; + if (range->first_ino == + insert_range->last_ino + 1 && + range->policy_id == insert_range->policy_id) { + range->first_ino = insert_range->first_ino; + merge_right = 1; + } + } + /* check if we could shrink array */ + if (merge_left && merge_right) { + for (m = l; m < dest_info->file_ranges_count - 1; + m++) + dest_info->file_ranges[m] = + dest_info->file_ranges[m + 1]; + + dest_info->file_ranges_count--; + return 0; + } else if (merge_left || merge_right) { /* return directly */ + return 0; + } + + ret = increase_file_ranges_capacity(dest_ctx, dest_info, NULL); + if (ret) + return ret; + + /* move forward */ + for (m = dest_info->file_ranges_count; m >= l; m--) + dest_info->file_ranges[m + 1] = + dest_info->file_ranges[m]; + + dest_info->file_ranges[l] = *insert_range; + dest_info->file_ranges_count++; + return 0; +} + +int merge_two_encrypted_files(e2fsck_t src_ctx, e2fsck_t dest_ctx) +{ + struct encrypted_file_info *src_info = src_ctx->encrypted_files; + struct encrypted_file_info *dest_info = dest_ctx->encrypted_files; + struct encrypted_file_range *range; + __u32 policy_id; + errcode_t retval; + size_t i; + + /* nothing to merge */ + if (!src_info) + return 0; + + if (!dest_info) { + dest_ctx->encrypted_files = src_info; + src_ctx->encrypted_files = NULL; + return 0; + } + + for (i = 0; i < src_info->file_ranges_count; i++) { + range = &src_info->file_ranges[i]; + retval = get_encryption_policy_id(dest_ctx, range->first_ino, + &policy_id); + if (retval != 0) + return retval; + /* reset policy id */ + range->policy_id = policy_id; + retval = find_entry_insert(dest_ctx, range); + if (retval) + return retval; + } + + return 0; +} + static int append_ino_and_policy_id(e2fsck_t ctx, struct problem_context *pctx, ext2_ino_t ino, __u32 policy_id) { struct encrypted_file_info *info = ctx->encrypted_files; struct encrypted_file_range *range; + int ret; /* See if we can just extend the last range. */ if (info->file_ranges_count > 0) { @@ -310,32 +457,10 @@ static int append_ino_and_policy_id(e2fsck_t ctx, struct problem_context *pctx, } } /* Nope, a new range is needed. */ + ret = increase_file_ranges_capacity(ctx, info, pctx); + if (ret) + return ret; - if (info->file_ranges_count == info->file_ranges_capacity) { - /* Double the capacity by default. */ - size_t new_capacity = info->file_ranges_capacity * 2; - - /* ... but go from 0 to 128 right away. */ - if (new_capacity < 128) - new_capacity = 128; - - /* We won't need more than the filesystem's inode count. */ - if (new_capacity > ctx->fs->super->s_inodes_count) - new_capacity = ctx->fs->super->s_inodes_count; - - /* To be safe, ensure the capacity really increases. */ - if (new_capacity < info->file_ranges_capacity + 1) - new_capacity = info->file_ranges_capacity + 1; - - if (ext2fs_resize_mem(info->file_ranges_capacity * - sizeof(*range), - new_capacity * sizeof(*range), - &info->file_ranges) != 0) - return handle_nomem(ctx, pctx, - new_capacity * sizeof(*range)); - - info->file_ranges_capacity = new_capacity; - } range = &info->file_ranges[info->file_ranges_count++]; range->first_ino = ino; range->last_ino = ino; diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 127b390d..4173d920 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2287,9 +2287,6 @@ void _e2fsck_pass1(e2fsck_t ctx) ext2fs_close_inode_scan(scan); scan = NULL; - /* We don't need the encryption policy => ID map any more */ - destroy_encryption_policy_map(ctx); - if (ctx->ea_block_quota_blocks) { ea_refcount_free(ctx->ea_block_quota_blocks); ctx->ea_block_quota_blocks = 0; @@ -3040,6 +3037,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; + struct encrypted_file_info *dest_info = global_ctx->encrypted_files; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -3074,6 +3072,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->ea_block_quota_inodes = ea_block_quota_inodes; global_ctx->block_ea_map = block_ea_map; global_ctx->ea_inode_refs = ea_inode_refs; + global_ctx->encrypted_files = dest_info; PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_directory_count); PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_regular_count); PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_blockdev_count); @@ -3130,6 +3129,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->invalid_inode_table_flag = invalid_inode_table_flag; global_ctx->invalid_bitmaps = invalid_bitmaps; e2fsck_pass1_merge_invalid_bitmaps(global_ctx, thread_ctx); + retval = merge_two_encrypted_files(thread_ctx, global_ctx); + if (retval) { + com_err(global_ctx->program_name, 0, + _("while merging encrypted files")); + return retval; + } /* * PASS1_COPY_CTX_BITMAP might return directly from this function, @@ -3184,6 +3189,8 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) ea_refcount_free(thread_ctx->refcount_orig); thread_ctx->refcount_orig = NULL; } + destroy_encrypted_file_info(thread_ctx); + ext2fs_free_mem(&thread_ctx); return retval; From patchwork Wed Apr 8 10:45:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267901 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=E/2RP02e; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dk2gH6z9sSh for ; Wed, 8 Apr 2020 20:47:02 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728318AbgDHKrC (ORCPT ); Wed, 8 Apr 2020 06:47:02 -0400 Received: from mail-pg1-f196.google.com ([209.85.215.196]:43637 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKrB (ORCPT ); Wed, 8 Apr 2020 06:47:01 -0400 Received: by mail-pg1-f196.google.com with SMTP id s4so3136216pgk.10 for ; Wed, 08 Apr 2020 03:47:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uAq99OOvuKKwevmXOhPesNPdE1EHSYa4GsrsBMnryfo=; b=E/2RP02eXAY0BY4HFoMq1K0ND8DeN8xjcMx1Hpj4cs8WdtD42nZoxaHJhmHGgmarjM 4LbfHcBu1o+8tvo9G2KGJoxZhLe7YcQMImGwBWMGWOeWtyfuH/fv2hc9Cg9t7V4aKcja 8y8FNvQGlygvQinrrgkNhhR8EsmkVq8s6izOXdnwd+WqotGLQribsJEZxrmHr96ez5IN mNSaA4N0GEwUbsajf9NQ7TgEsOlHkr4ZHVN4unfjXJYA7tzIttgxlN6G2THwI5v39aHq 36qEzFurVqSjiT0Q+ihHaXuLdo+V8zSDvrqEnKxWR/Xnx1hU0IAFaH8EESKd0R76ERR/ wnpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uAq99OOvuKKwevmXOhPesNPdE1EHSYa4GsrsBMnryfo=; b=Dt7T/+XgetC3ZxOZqJpmLPtdxk6DbCorKfC7mYY+CSA96Qf2EBDwFI4thy3mhfeGc3 WGR2ZRf54KalWJw/+cJjlyzTFScpQ+9nDPw2C8yUga5tyS6ZHRx6sGn0btI3iRrMRBoE gnkH1kGEbCmZa/ghG603aH26++tTSvCanjDSOfjLtg2qNVyG6m6oMsEbtRumUfOPgON2 Q9quhoSMXL9WwaWvQu1+f9HvjYru+l/NunsQvREvNAi928egv31WfEGXaqW9jomK/hxe zdQhv9AqLBfebOOkHn8YvlePdCGkx0qYZUULMA1pFf7oqWiU/oLN/fsOE5AljSpVrxXk Ab5w== X-Gm-Message-State: AGi0PuZQnozOXC/LPijkdWiey4Wg7SB24hnDx4X29lHIaI/1IEOlwsBz BxUCblsMT6szVX3EME474SSx0wYjPfo= X-Google-Smtp-Source: APiQypJZtl54kstqSt/VkTh6gcTTc5jOgs44khldE5avSjuXhELRQ8m+7e4rojuIhC8WK9N3e+hu2w== X-Received: by 2002:a62:1dd3:: with SMTP id d202mr7140239pfd.47.1586342820757; Wed, 08 Apr 2020 03:47:00 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.46.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:47:00 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 41/46] e2fsck: merge inode_bad_map after threads finish Date: Wed, 8 Apr 2020 19:45:09 +0900 Message-Id: <1586342714-12536-42-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Change-Id: I19138835c8532eab8f91b711ba25300b33329902 Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 1 + 1 file changed, 1 insertion(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 4173d920..d7d37d8b 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -3141,6 +3141,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx * so please do NOT leave any garbage behind after returning. */ PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_used_map); + PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_bad_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_dir_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_bb_map); PASS1_MERGE_CTX_BITMAP(global_ctx, thread_ctx, inode_imagic_map); From patchwork Wed Apr 8 10:45:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267902 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=keW/WNWK; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dm5ZNPz9sSb for ; Wed, 8 Apr 2020 20:47:04 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728321AbgDHKrE (ORCPT ); Wed, 8 Apr 2020 06:47:04 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:41117 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKrE (ORCPT ); Wed, 8 Apr 2020 06:47:04 -0400 Received: by mail-pl1-f194.google.com with SMTP id d24so2369925pll.8 for ; Wed, 08 Apr 2020 03:47:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RAjuh+1btcAwdLSNciTi1bbEpinto8548BF42CS+vUs=; b=keW/WNWKaSxYxgkbLOGpu+2lrONGpCmNLP2LUO3v4uJEMpaXgEUjchWvPBxHw8IJ0H HW3kzyuwgaCMle8NYUai0WU1usLyGQi2XMFrX4ehgbjfEiIlmWCFtkK0D5z3U0zpE6EO 9BdKrZ+O4xUPrAeytTCmclP9LSPYskh4MECcEUbY0qRhOLAhcG7G+3ga/JHU8xNwke5l 7u8BRYVq9xNUm5YFbiJhkKmwaweAFBrqKmp0W46Trz1XVgAwIdSTbjfFOBT7neN3MowK 7kI8htvhjNIhleh8fylQJkVIZ47EjIFRcKpyDyvj733mzmy0WzAgGqWKeg3eXa+LCPkr 8nXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RAjuh+1btcAwdLSNciTi1bbEpinto8548BF42CS+vUs=; b=JDqaTHgcNNINo0SUoTp2AEaaZGktcQJT0j/iwCE1l7wBvDUtfd4o+08ELNUykT8ZM0 J7JMlgxCM1xf5wr0O4hscjZz5QT3nuwKhJSm+UV7TGuSSzsr3n2LSM05TUFQiQG7PK2l eN816wokKi5fC9HSQ15v65Ef08uOjFWpfE+E03/9y1/qm/dHcKXOtI+jcNSleZQ8WJkB 2j+suWYvNx3AUhoHdSIskO/MO3FtMBmWqBE+Mmvj4qszizqhU1lqiugr0N+o7o+CKr6N HQ3Y1AUgwVgNM6tvROy7O11qfN5XXTXpZwUXx5WBeXiSl2H9JTFa/vsV2rF1aBcAng1d nglw== X-Gm-Message-State: AGi0PuYNe5VfR56Bpy/xFByf/fTDYjGn2s33EMFsc8K9Sb3BjLeyxn0h 6SZGOcUIiXKWccTDY2CMqN79pALeCsw= X-Google-Smtp-Source: APiQypKyOS9UcZgY/VXinJgFVzpBh//1Hj4NyKs6Z3FNqzdeqNr+F4tSI/o1mY4PlPNrX4M7FBgqqQ== X-Received: by 2002:a17:90b:8e:: with SMTP id bb14mr4766011pjb.13.1586342822827; Wed, 08 Apr 2020 03:47:02 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.47.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:47:02 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 42/46] e2fsck: simplify e2fsck context merging codes Date: Wed, 8 Apr 2020 19:45:10 +0900 Message-Id: <1586342714-12536-43-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong We tried to copy thread context to global context directly and then copy back some saved variables before merging. Since we have finished almost all necessary variables in the e2fsck context, we could simplify codes, and this could help us understand what is missing rather than hide problems. Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 147 +++++++++---------------------------------------- 1 file changed, 25 insertions(+), 122 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index d7d37d8b..3c3d9251 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2387,11 +2387,6 @@ do { \ } \ } while (0) -#define PASS1_MERGE_CTX_COUNT(_dest, _src, _field) \ -do { \ - _dest->_field = _field + _src->_field; \ -} while (0) - static errcode_t pass1_open_io_channel(ext2_filsys fs, const char *io_options, io_manager manager, int flags) @@ -2508,6 +2503,7 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) ext2_badblocks_list badblocks; ext2_dblist dblist; int flags; + e2fsck_t dest_ctx = dest->priv_data; dest_io = dest->io; dest_image_io = dest->image_io; @@ -2523,6 +2519,7 @@ static int _e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) dest->block_map = block_map; dest->badblocks = badblocks; dest->dblist = dblist; + dest->priv_data = dest_ctx; dest->flags = src->flags | flags; if (!(src->flags & EXT2_FLAG_VALID) || !(flags & EXT2_FLAG_VALID)) ext2fs_unmark_valid(dest); @@ -2985,130 +2982,42 @@ static errcode_t e2fsck_pass1_merge_ea_refcount(e2fsck_t global_ctx, return retval; } -static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) +static errcode_t e2fsck_pass1_merge_context(e2fsck_t global_ctx, + e2fsck_t thread_ctx) { - errcode_t retval; - int flags = global_ctx->flags; - ext2_filsys thread_fs = thread_ctx->fs; - ext2_filsys global_fs = global_ctx->fs; - FILE *global_logf = global_ctx->logf; - FILE *global_problem_logf = global_ctx->problem_logf; - struct dir_info_db *dir_info = global_ctx->dir_info; - struct dx_dir_info *dx_dir_info = global_ctx->dx_dir_info; - ext2fs_inode_bitmap inode_used_map = global_ctx->inode_used_map; - ext2fs_inode_bitmap inode_dir_map = global_ctx->inode_dir_map; - ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; - ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; - ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; - ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; - ext2_icount_t inode_count = global_ctx->inode_count; - ext2_icount_t inode_link_info = global_ctx->inode_link_info; - __u32 fs_directory_count = global_ctx->fs_directory_count; - __u32 fs_regular_count = global_ctx->fs_regular_count; - __u32 fs_blockdev_count = global_ctx->fs_blockdev_count; - __u32 fs_chardev_count = global_ctx->fs_chardev_count; - __u32 fs_links_count = global_ctx->fs_links_count; - __u32 fs_symlinks_count = global_ctx->fs_symlinks_count; - __u32 fs_fast_symlinks_count = global_ctx->fs_fast_symlinks_count; - __u32 fs_fifo_count = global_ctx->fs_fifo_count; - __u32 fs_total_count = global_ctx->fs_total_count; - __u32 fs_badblocks_count = global_ctx->fs_badblocks_count; - __u32 fs_sockets_count = global_ctx->fs_sockets_count; - __u32 fs_ind_count = global_ctx->fs_ind_count; - __u32 fs_dind_count = global_ctx->fs_dind_count; - __u32 fs_tind_count = global_ctx->fs_tind_count; - __u32 fs_fragmented = global_ctx->fs_fragmented; - __u32 fs_fragmented_dir = global_ctx->fs_fragmented_dir; - __u32 large_files = global_ctx->large_files; - int dx_dir_info_size = global_ctx->dx_dir_info_size; - int dx_dir_info_count = global_ctx->dx_dir_info_count; - ext2_u32_list dirs_to_hash = global_ctx->dirs_to_hash; - quota_ctx_t qctx = global_ctx->qctx; - int *invalid_block_bitmap_flag = global_ctx->invalid_block_bitmap_flag; - int *invalid_inode_bitmap_flag = global_ctx->invalid_inode_bitmap_flag; - int *invalid_inode_table_flag = global_ctx->invalid_inode_table_flag; - int invalid_bitmaps = global_ctx->invalid_bitmaps; - ext2_refcount_t refcount = global_ctx->refcount; - ext2_refcount_t refcount_extra = global_ctx->refcount_extra; - ext2_refcount_t refcount_orig = global_ctx->refcount_orig; - ext2_refcount_t ea_block_quota_blocks = global_ctx->ea_block_quota_blocks; - ext2_refcount_t ea_block_quota_inodes = global_ctx->ea_block_quota_inodes; - ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; - ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; - ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; - ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; - struct encrypted_file_info *dest_info = global_ctx->encrypted_files; - -#ifdef HAVE_SETJMP_H - jmp_buf old_jmp; + errcode_t retval; - memcpy(old_jmp, global_ctx->abort_loc, sizeof(jmp_buf)); -#endif - memcpy(global_ctx, thread_ctx, sizeof(struct e2fsck_struct)); -#ifdef HAVE_SETJMP_H - memcpy(global_ctx->abort_loc, old_jmp, sizeof(jmp_buf)); -#endif + global_ctx->fs_directory_count += thread_ctx->fs_directory_count; + global_ctx->fs_regular_count += thread_ctx->fs_regular_count; + global_ctx->fs_blockdev_count += thread_ctx->fs_blockdev_count; + global_ctx->fs_chardev_count += thread_ctx->fs_chardev_count; + global_ctx->fs_links_count += thread_ctx->fs_links_count; + global_ctx->fs_symlinks_count += thread_ctx->fs_symlinks_count; + global_ctx->fs_fast_symlinks_count += thread_ctx->fs_fast_symlinks_count; + global_ctx->fs_fifo_count += thread_ctx->fs_fifo_count; + global_ctx->fs_total_count += thread_ctx->fs_total_count; + global_ctx->fs_badblocks_count += thread_ctx->fs_badblocks_count; + global_ctx->fs_sockets_count += thread_ctx->fs_sockets_count; + global_ctx->fs_ind_count += thread_ctx->fs_ind_count; + global_ctx->fs_dind_count += thread_ctx->fs_dind_count; + global_ctx->fs_tind_count += thread_ctx->fs_tind_count; + global_ctx->fs_fragmented += thread_ctx->fs_fragmented; + global_ctx->fs_fragmented_dir += thread_ctx->fs_fragmented_dir; + global_ctx->large_files += thread_ctx->large_files; + global_ctx->flags |= thread_ctx->flags; - global_ctx->inode_used_map = inode_used_map; - global_ctx->inode_dir_map = inode_dir_map; - global_ctx->inode_bb_map = inode_bb_map; - global_ctx->inode_imagic_map = inode_imagic_map; - global_ctx->inodes_to_rebuild = inodes_to_rebuild; - global_ctx->inode_reg_map = inode_reg_map; - global_ctx->block_dup_map = block_dup_map; - global_ctx->block_found_map = block_found_map; - global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); - global_ctx->dx_dir_info = dx_dir_info; - global_ctx->dx_dir_info_count = dx_dir_info_count; - global_ctx->dx_dir_info_size = dx_dir_info_size; e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); - global_ctx->inode_count = inode_count; - global_ctx->inode_link_info = inode_link_info; - global_ctx->refcount = refcount; - global_ctx->refcount_extra = refcount_extra; - global_ctx->refcount_orig = refcount_orig; - global_ctx->ea_block_quota_blocks = ea_block_quota_blocks; - global_ctx->ea_block_quota_inodes = ea_block_quota_inodes; - global_ctx->block_ea_map = block_ea_map; - global_ctx->ea_inode_refs = ea_inode_refs; - global_ctx->encrypted_files = dest_info; - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_directory_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_regular_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_blockdev_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_chardev_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_links_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_symlinks_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fast_symlinks_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fifo_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_total_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_badblocks_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_sockets_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_ind_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_dind_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_tind_count); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fragmented); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, fs_fragmented_dir); - PASS1_MERGE_CTX_COUNT(global_ctx, thread_ctx, large_files); - - global_ctx->flags |= flags; - - retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); + retval = e2fsck_pass1_merge_fs(global_ctx->fs, thread_ctx->fs); if (retval) { com_err(global_ctx->program_name, 0, _("while merging fs\n")); return retval; } - global_fs->priv_data = global_ctx; - global_ctx->fs = global_fs; - global_ctx->logf = global_logf; - global_ctx->problem_logf = global_problem_logf; - global_ctx->global_ctx = NULL; retval = e2fsck_pass1_merge_icounts(global_ctx, thread_ctx); if (retval) { com_err(global_ctx->program_name, 0, _("while merging icounts\n")); return retval; } - global_ctx->dirs_to_hash = dirs_to_hash; retval = e2fsck_pass1_merge_dirs_to_hash(global_ctx, thread_ctx); if (retval) { com_err(global_ctx->program_name, 0, _("while merging dirs to hash\n")); @@ -3116,18 +3025,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx } e2fsck_pass1_merge_ea_inode_refs(global_ctx, thread_ctx); e2fsck_pass1_merge_ea_refcount(global_ctx, thread_ctx); - global_ctx->qctx = qctx; e2fsck_pass1_merge_quota_ctx(global_ctx, thread_ctx); retval = ext2fs_merge_bitmap(thread_ctx->block_found_map, global_ctx->block_found_map, global_ctx->block_dup_map); - e2fsck_pass1_block_map_unlock(global_ctx); if (retval == EEXIST) global_ctx->flags |= E2F_FLAG_DUP_BLOCK; - global_ctx->invalid_block_bitmap_flag = invalid_block_bitmap_flag; - global_ctx->invalid_inode_bitmap_flag = invalid_inode_bitmap_flag; - global_ctx->invalid_inode_table_flag = invalid_inode_table_flag; - global_ctx->invalid_bitmaps = invalid_bitmaps; e2fsck_pass1_merge_invalid_bitmaps(global_ctx, thread_ctx); retval = merge_two_encrypted_files(thread_ctx, global_ctx); if (retval) { @@ -3156,7 +3059,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; - retval = e2fsck_pass1_thread_join_one(global_ctx, thread_ctx); + retval = e2fsck_pass1_merge_context(global_ctx, thread_ctx); ext2fs_free_mem(&thread_ctx->fs); if (thread_ctx->logf) fclose(thread_ctx->logf); From patchwork Wed Apr 8 10:45:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267903 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=qZsMFfDj; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dp3yq8z9sSb for ; Wed, 8 Apr 2020 20:47:06 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728322AbgDHKrG (ORCPT ); Wed, 8 Apr 2020 06:47:06 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:32920 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKrG (ORCPT ); Wed, 8 Apr 2020 06:47:06 -0400 Received: by mail-pf1-f194.google.com with SMTP id c138so2889072pfc.0 for ; Wed, 08 Apr 2020 03:47:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=zpCt75xNZByviQmE+AnXigExECBgdAVpN3uQUy9EVLo=; b=qZsMFfDj2a1Ldqjx8753NxhY/8S4RrXPGqxxGki/0G3zR00CPd+6knVXel4fIriJFy v07NrJtgX06kEx4Zm3hB6bezzAOhDBexDv7CpoxY4hoKGZ+Y0TNmPozar1RZcsTKJuOE +4a0IZg8l3XWjLcCYqhGj/ZN7j5fZLXUz17jLHQFfWod46c1OiIaqGjfzw8E/QKPcADf Nn25Rz8FON+88iu9YKdcYsF4uLgqqT8PIT6REeRX6ABewtn5/l4otHH/Jp/gWlnQBbLT DPx93b9+7Lt0Lsa4O4ZzMtl6CBLopwg7VvUC17HPrp3D3S1dWuGY14VGlP4zaRDRVhSy ze9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=zpCt75xNZByviQmE+AnXigExECBgdAVpN3uQUy9EVLo=; b=QJtMmOyKScSqoZA9E4aubU0d80HgA2Lp5T5S4L1JHAVznkBZPc09IGQ51Gjfiy/xP7 DTbMaIGr2Abad/M6fQBzPJN4RmGRLuswSNGv/e0kW3DzDv+iMe7zp6xSgpo38qDpW21C TW9lcDz4TgkxNOoquQsjEhcyUbNuejcCu5vmfo4VF8tjXFIn0kJMBVs4YOAaZi2VvA+x dN+z8t3CH8kI4FEIU4ykKCX1gjivFBv8IfAN4uB0n6cHHK2KZYaZ1pJE4h4vnzJDhFCA /zgWu7g9h4rerLI6LhC2GjPZCjMfNbSVq7fGRkIlOZDd2xQDzroLSr01K7c+r6Q01Per oj7Q== X-Gm-Message-State: AGi0PuaHzSGU2yiVtq2mo0/J1anCF7Hl2i27C5h6IBAoXxjPPL8gFaKQ gur4Iu64++9cbKBkHGkEJTNE+XWvEvc= X-Google-Smtp-Source: APiQypIv/zHxO0qEX3SyHzS9jJAvaPYv8+RPN40RiNZiJzGT5cjeisyVttJCmOnR8OmfrTWEl34qPA== X-Received: by 2002:a63:b214:: with SMTP id x20mr6386559pge.43.1586342824865; Wed, 08 Apr 2020 03:47:04 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.47.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:47:04 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 43/46] e2fsck: merge options after threads finish Date: Wed, 8 Apr 2020 19:45:11 +0900 Message-Id: <1586342714-12536-44-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong It will be possible that threads might append E2F_OPT_YES, so we need merge options to global, test f_yesall cover this. Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 3c3d9251..19475815 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -3005,6 +3005,8 @@ static errcode_t e2fsck_pass1_merge_context(e2fsck_t global_ctx, global_ctx->fs_fragmented_dir += thread_ctx->fs_fragmented_dir; global_ctx->large_files += thread_ctx->large_files; global_ctx->flags |= thread_ctx->flags; + /* threads might enable E2F_OPT_YES */ + global_ctx->options |= thread_ctx->options; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); From patchwork Wed Apr 8 10:45:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267904 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=pDMzW3c5; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dr4dmYz9sSG for ; Wed, 8 Apr 2020 20:47:08 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728327AbgDHKrI (ORCPT ); Wed, 8 Apr 2020 06:47:08 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:41566 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKrI (ORCPT ); Wed, 8 Apr 2020 06:47:08 -0400 Received: by mail-pf1-f193.google.com with SMTP id b8so376269pfp.8 for ; Wed, 08 Apr 2020 03:47:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qxQKpRrLqljFuiBTr4O911Th7guq7u5hMiM/VjA89uo=; b=pDMzW3c5JGbEhxNw5P4casDxZ8saMYhOPLNXHbiFIe4Q/uE/AlxuEqWmB/7222kQ3O cUeGTJOvPE70Rc7NpRdwejOkyMVtgz7gkHa/RVpNPbfrDqq0ZCgRx47BSc+v7sEbqC27 GR98X+EzCdSwf6OvgFt9qX4OpjhDoDF1XvJTeIHCcybbwRXiJD8HTFxUgcFnezSpZrGG gI4F/C3fQR/v5/MCpjGrxsf9RC/Q0B5ZstBB8Hj/NZozRI+8lSeNkZJwpih2Istt/9+d XaUYtMai1ShdxsYTMsiV9Fs2RQclK1pwb/9g3DEUbOVCextLs40RC2Ggcr1fKWejWaF9 4QhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qxQKpRrLqljFuiBTr4O911Th7guq7u5hMiM/VjA89uo=; b=p5jNHMSgy7Op6KlzCq7enMRP6cnwbFHRCsvACcumfbwCIppfPcbvw+OClOdlCGdgCx MJJ1Ngr8lprAISRD+z9Sozl8iNq+hlNJ4d8q468JHJtZm+ZFYhRKkZJdv62OV25aKHgi Kqo4ucuK5L9RR/I7wEH39MGqCqDEfrmbOENJs4POPp+Xez0ZvYXtaziyg3QMj8D5kEJ9 Dl6i9KnUYxPxuRFhrj46u6Vkxj8oWxLFCZHHQ7ZKu0chRhe/MqYDOQ/LPu6GMta6xINT 6JtTboFk7FKYBX9Mj+IfvJbDIcXC9jmDkHZJEjrvObSZHFLgWasShcepXP9pO1nq+qVe Sfng== X-Gm-Message-State: AGi0Pubj/RlNyTdEZ0PCm0C2hYLufHQtAnqIW1yAFEFSKM/oYPR7luYP 59hwWhiK+hbeCeFFeRLrBL0wQ9sdeMQ= X-Google-Smtp-Source: APiQypK9aez2Svf6X+GBeZ/yGSOftwjLHDvL7YSp0mBN+ccQixhHctOYmKzVzUS6uhdw3Jq85U6ocg== X-Received: by 2002:aa7:940f:: with SMTP id x15mr2033989pfo.312.1586342826904; Wed, 08 Apr 2020 03:47:06 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.47.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:47:06 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 44/46] e2fsck: reset lost_and_found after threads finish Date: Wed, 8 Apr 2020 19:45:12 +0900 Message-Id: <1586342714-12536-45-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong This should not be kept, the reaons is similar to what e2fsck_pass1 has done before. Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 19475815..61b667e0 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -3007,6 +3007,11 @@ static errcode_t e2fsck_pass1_merge_context(e2fsck_t global_ctx, global_ctx->flags |= thread_ctx->flags; /* threads might enable E2F_OPT_YES */ global_ctx->options |= thread_ctx->options; + /* + * The l+f inode may have been cleared, so zap it now and + * later passes will recalculate it if necessary + */ + global_ctx->lost_and_found = 0; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); From patchwork Wed Apr 8 10:45:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267905 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Ge2A+sq4; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dt3Yv9z9sSG for ; Wed, 8 Apr 2020 20:47:10 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728328AbgDHKrK (ORCPT ); Wed, 8 Apr 2020 06:47:10 -0400 Received: from mail-pj1-f65.google.com ([209.85.216.65]:40260 "EHLO mail-pj1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKrJ (ORCPT ); Wed, 8 Apr 2020 06:47:09 -0400 Received: by mail-pj1-f65.google.com with SMTP id kx8so997845pjb.5 for ; Wed, 08 Apr 2020 03:47:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RI4K8Bq4KjVualKZfgzoqkpxnsQBg30vpB9GLwz60hg=; b=Ge2A+sq4SP97doT7iy7PGpcAz3TbuMzUoi5WAvjj4ZUQSrQKDlHmaW1Hrt/cqWGLGo XUnD2WIjqZxJ17/euePeIFBuzDj7d0AfojSa9f2UDyK6MzqaRlyfXD9DZjwazcnl2+qv YCaIvdLqgtHYa24n9W4lX2X9MRMYKHBBRykTnvF5Ti7clNcq87aFfuxIzzrJsvnXzqn6 fIYE3JAvpVCx4F0GHZS1oLNOKRnPlTaxSiw8Ndgn2WZFxK2iQHzNgTh/h8T1Exrn+1k3 92DHTqm/e7NudwrS3crFPurbIMpTlG3OHB8NKzoriNi7CnK1oaIhsl4vuUZUufSSzKWA NAvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RI4K8Bq4KjVualKZfgzoqkpxnsQBg30vpB9GLwz60hg=; b=M65VgZebxZoHlAukgwaIgW2+o7JPg3eXDFhulaqoqybgR4De3J3ZA1Bs7pFOzZO+Vk XRHK9yYf3qWICEBHnFV89Nqk7u+JxCP9oWC6GEiAPLVwJ4SeDtM/P0Zns/7BK5E8d/zB eiKwRXsLvnUcao2v+8xSfS67+dQBCu7IukGflAtNWoqz709j/qk1dOLbqqXpWN8ayIWG 7xjg+THVv7hYCncyT1C+Y/uq1eguml7mityJdy2vSMBCApSwwy1573Q5IbFsSxu45LFT FaoLUFHPgmoovKQuIPnUCtYn2QmoqwTYdwjojwgPcXc5VJx7iU9n0IE+2AlMWI1VpaJz 4RRg== X-Gm-Message-State: AGi0PuZSkq7khdroG1SbIYb71rbJCZfRMx09EGlJTWzup3KA1WQ0XDKf fgJEb05k+san2m4/ZrLjCgzeOYiOpg4= X-Google-Smtp-Source: APiQypJmuoiunQChpcDDB7W2MmWCcV6+QRvLhEaYYw+OwD1q20pA5wANVUeQHXL/1TCQ/7Ri3O3sNw== X-Received: by 2002:a17:902:b281:: with SMTP id u1mr6637545plr.287.1586342829004; Wed, 08 Apr 2020 03:47:09 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.47.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:47:08 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Wang Shilong Subject: [RFC PATCH 45/46] LU-8465 e2fsck: merge extent depth count after threads finish Date: Wed, 8 Apr 2020 19:45:13 +0900 Message-Id: <1586342714-12536-46-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong tests covered by f_extent_htree. Signed-off-by: Wang Shilong --- e2fsck/pass1.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 61b667e0..cd51de4a 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2986,6 +2986,7 @@ static errcode_t e2fsck_pass1_merge_context(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; + int i; global_ctx->fs_directory_count += thread_ctx->fs_directory_count; global_ctx->fs_regular_count += thread_ctx->fs_regular_count; @@ -3012,6 +3013,10 @@ static errcode_t e2fsck_pass1_merge_context(e2fsck_t global_ctx, * later passes will recalculate it if necessary */ global_ctx->lost_and_found = 0; + /* merge extent depth count */ + for (i = 0; i < MAX_EXTENT_DEPTH_COUNT; i++) + global_ctx->extent_depth_count[i] += + thread_ctx->extent_depth_count[i]; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); From patchwork Wed Apr 8 10:45:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wang Shilong X-Patchwork-Id: 1267906 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ezMMIpr2; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 48y1Dw6svnz9sSG for ; Wed, 8 Apr 2020 20:47:12 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728330AbgDHKrM (ORCPT ); Wed, 8 Apr 2020 06:47:12 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:36964 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726980AbgDHKrM (ORCPT ); Wed, 8 Apr 2020 06:47:12 -0400 Received: by mail-pf1-f193.google.com with SMTP id u65so2232369pfb.4 for ; Wed, 08 Apr 2020 03:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/sCPK4EAfucZ4oL+FKdVtn5ycTMzZXmhQ6JEHaJ1dFc=; b=ezMMIpr2Zt054mV9TP1vYQ2G77U4W6uE+IIR1EfvJ9u7CKClGGI21DVrs4MCKpvPUM cjdeRcu9vLFbg+bGgQG4i8P1YxiOL3lue5nK6s/hxiKzBhJOo1uItYQDKx0FlU8qLd3q NoK5pt2jDrH9zNAp118JO6gQUjY/1jHd6PrP9qC+OABPv+PV+bhFE5xQRUEupkEZt7uB QusswkQWbHcb0MVLNphWBoRgdVrivF5vJiCbC4SLXxRDNHr6ZtCwUen4xkUuE3CI6plk n3c9F/+5042Yr7nQMDe3Yz3SyAHlJCHKYKFDQUjEHq9XijqVEFtxJ1QuR6Q6oGhNSn1h cvcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/sCPK4EAfucZ4oL+FKdVtn5ycTMzZXmhQ6JEHaJ1dFc=; b=W/vAKVtUpCWqy6zm4lOzQEBge+SkFfmSjqo2SVOfpQ8ExVqKICJZjIPuGvHNq7Czi5 EQaF5WYh3m0csiJZ/stKdDqbnaOfW1oBa+OegzWjYSew9S8542e29AyoDNpNJCBZktA5 UMhaNkNaTglhITN2m3FDeUVzX6BET/PzxXGT0fr199oNf2yN6sftmIrh25+fEzyhPt1C Ezt1XcWXGzo4M5D2J+MvfCIeVk2PAFyRxsecZZ74swgivc8Vys76ATtvfGkIFtf5VfNp M/xwBpsS0gX/F21zloVmc6zT68+c6AoJiKawHs/f/pqenu+VIRdR7YEQFPUMToIbBwzI kPfg== X-Gm-Message-State: AGi0PuaBgGdUtbKGr46DpRoHLc+esmAx1wInQGZtVjdWYPL9dhBgfwY4 hptReCnBWSDmo7VYMZBQvY7xhtn44Tk= X-Google-Smtp-Source: APiQypKPbDw9Z5j/dAsB8HEokI1LeqahT/Gn1FJd7BD3CXskBGQms0/nfluFvtu1gqzLwbUGOnGF6A== X-Received: by 2002:a62:2c87:: with SMTP id s129mr7279655pfs.263.1586342831111; Wed, 08 Apr 2020 03:47:11 -0700 (PDT) Received: from localhost.localdomain (ftp.datadirectnet.jp. [182.171.80.51]) by smtp.gmail.com with ESMTPSA id y17sm16177024pfl.104.2020.04.08.03.47.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Apr 2020 03:47:10 -0700 (PDT) From: Wang Shilong To: linux-ext4@vger.kernel.org Cc: lixi@ddn.com, adilger@dilger.ca, sihara@ddn.com, Li Dongyang Subject: [RFC PATCH 46/46] libext2fs: optimize ext2fs_convert_subcluster_bitmap() Date: Wed, 8 Apr 2020 19:45:14 +0900 Message-Id: <1586342714-12536-47-git-send-email-wangshilong1991@gmail.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> References: <1586342714-12536-1-git-send-email-wangshilong1991@gmail.com> Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Dongyang For a bigalloc filesystem, converting the block bitmap from blocks to chunks in ext2fs_convert_subcluster_bitmap() can take a long time when the device is huge, because we test the bitmap bit-by-bit using ext2fs_test_block_bitmap2(). Use ext2fs_find_first_set_block_bitmap2() which is more efficient for mke2fs when the fs is mostly empty. e2fsck can also benefit from this during pass1 block scanning. Time taken for "mke2fs -O bigalloc,extent -C 131072 -b 4096" on a 1PB device: without patch: real 27m49.457s user 21m36.474s sys 6m9.514s with patch: real 6m31.908s user 0m1.806s sys 6m29.697s Change-Id: I054ed3ae9e25b20de34f61358dfce3c4bd6f3092 Signed-off-by: Li Dongyang Signed-off-by: Li Xi --- lib/ext2fs/gen_bitmap64.c | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/lib/ext2fs/gen_bitmap64.c b/lib/ext2fs/gen_bitmap64.c index d354cffa..299fee6e 100644 --- a/lib/ext2fs/gen_bitmap64.c +++ b/lib/ext2fs/gen_bitmap64.c @@ -856,8 +856,7 @@ errcode_t ext2fs_convert_subcluster_bitmap(ext2_filsys fs, ext2fs_generic_bitmap_64 bmap, cmap; ext2fs_block_bitmap gen_bmap = *bitmap, gen_cmap; errcode_t retval; - blk64_t i, b_end, c_end; - int n, ratio; + blk64_t i, next, b_end, c_end; bmap = (ext2fs_generic_bitmap_64) gen_bmap; if (fs->cluster_ratio_bits == ext2fs_get_bitmap_granularity(gen_bmap)) @@ -874,18 +873,13 @@ errcode_t ext2fs_convert_subcluster_bitmap(ext2_filsys fs, bmap->end = bmap->real_end; c_end = cmap->end; cmap->end = cmap->real_end; - n = 0; - ratio = 1 << fs->cluster_ratio_bits; while (i < bmap->real_end) { - if (ext2fs_test_block_bitmap2(gen_bmap, i)) { - ext2fs_mark_block_bitmap2(gen_cmap, i); - i += ratio - n; - n = 0; - continue; - } - i++; n++; - if (n >= ratio) - n = 0; + retval = ext2fs_find_first_set_block_bitmap2(gen_bmap, + i, bmap->real_end, &next); + if (retval) + break; + ext2fs_mark_block_bitmap2(gen_cmap, next); + i = EXT2FS_C2B(fs, EXT2FS_B2C(fs, next) + 1); } bmap->end = b_end; cmap->end = c_end;