From patchwork Wed Nov 18 15:38:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402308 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=mNTjKqev; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8J6Hsgz9sTc for ; Thu, 19 Nov 2020 02:40:48 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726588AbgKRPk3 (ORCPT ); Wed, 18 Nov 2020 10:40:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726574AbgKRPk2 (ORCPT ); Wed, 18 Nov 2020 10:40:28 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93695C0613D4 for ; Wed, 18 Nov 2020 07:40:28 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id m186so2894801ybm.22 for ; Wed, 18 Nov 2020 07:40:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5slokjrbk5GSblwDfGqS+jirkAWYeNGrs+vpDmDImQ0=; b=mNTjKqevsB1KPw+w74ymz9e1fndIksc9qrZ3tlicMOFPpZLBb0p9GOntI79iga4CEm 52mVdpGXApXTwrcXJrEyUo1C/PdWW2WB0XIVH6rzJo546s+/EPJOUvpPCoeKkCnq+Fap Vvf2LPuHzLuoLjRXt+NeyLhy1MDhyLtxt1C2N6ugoqS0mKnMGRkfALRcIXlyh/pxz+ss 1uF9BJNEPrJy8bmvZhWCDlyjmrhipvvk+IK0F1LTTELPwLw3/sSK4EtekB5OzUPnFi88 w30NZjD0biqFU9mtvFJBCfVZFQeEb7vaVO0odmNXyQzpW4Hi5BzhceYeamdqu9gSwQLB 4ZLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5slokjrbk5GSblwDfGqS+jirkAWYeNGrs+vpDmDImQ0=; b=JwiVtoR3fb6ex48cBr8WJezAhKt8awaTRvdx5bVCYXh3b/Cpk2aW+DDYCfuFoM3YDD ovy592JYtAm6CJJD6wopMV6YI4maUXvPkk43Rt5eUfcKgiLBRzeZozPJFkQGEzM96R+7 CWuPk312vIGMf43pmKaEon4skjiJUAoM4mbRfVg12pbS/HaDfHeVXM+8AbkRZSJGKTES baNCJI611MUOAltqTQ2omygoebOTp7T3SKsKdyBB2/A/Y5NEbMys8bTlct3gYAME1owi vl9VlxcNp8+vg9ydjUAd8Dxod2mvoFsgi5ORivb4ZiY7mM87CXHBSEoZ4g7okTSQt6t1 lcTw== X-Gm-Message-State: AOAM531wBfvSF+tmBi+LlROyW5AdoVSuS+g+Cv7pprF2nv/NFGDRbsMM 8dYBfnZW0XhunTwfYRKCw6ysQT+q2deGEqeZQLSXcrb9LYOPjB0oZ26OZn6L8HkcZqFdFyoAebV VEcQ+rrUQoBiq+iqOWKnyOKgM2si8QfPc32NQsuHf2BSWvUG8yp1dYCKEVBFAGmfTGZE6a73j/t 4ChC2Su8g= X-Google-Smtp-Source: ABdhPJy9a9Rup3j4GAb5Jrxk74rMAoL50qDVkP7M7b0flHplUqkYlyPL39bsE+RfyjOYM3Udq/DG2Z+oskNYA51oP9A= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:abce:: with SMTP id v72mr6622354ybi.138.1605714027642; Wed, 18 Nov 2020 07:40:27 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:47 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-2-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 01/61] e2fsck: add -m option for multithread From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi -m option is added but no actual functionality is added. This patch only adds the logic that when -m is specified, one of -p/-y/-n options should be specified. And when -m is specified, -C shouldn't be specified and the completion progress report won't be triggered by sending SIGUSR1/SIGUSR2 signals. This simplifies the implementation of multi-thread fsck in the future. Completion progress support with multi-thread fsck will be added back after multi-thread fsck implementation is finished. Right now, disable it to simplify the implementation of multi-thread fsck. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam Reviewed-by: Harshad Shirwadkar --- e2fsck/e2fsck.h | 1 + e2fsck/unix.c | 31 ++++++++++++++++++++----- tests/f_multithread/expect.1 | 23 ++++++++++++++++++ tests/f_multithread/expect.2 | 7 ++++++ tests/f_multithread/image.gz | 1 + tests/f_multithread/name | 1 + tests/f_multithread/script | 4 ++++ tests/f_multithread_completion/expect.1 | 2 ++ tests/f_multithread_completion/expect.2 | 23 ++++++++++++++++++ tests/f_multithread_completion/image.gz | 1 + tests/f_multithread_completion/name | 1 + tests/f_multithread_completion/script | 4 ++++ tests/f_multithread_no/expect.1 | 24 +++++++++++++++++++ tests/f_multithread_no/expect.2 | 23 ++++++++++++++++++ tests/f_multithread_no/image.gz | 1 + tests/f_multithread_no/name | 1 + tests/f_multithread_no/script | 4 ++++ tests/f_multithread_preen/expect.1 | 11 +++++++++ tests/f_multithread_preen/expect.2 | 23 ++++++++++++++++++ tests/f_multithread_preen/image.gz | 1 + tests/f_multithread_preen/name | 1 + tests/f_multithread_preen/script | 4 ++++ tests/f_multithread_yes/expect.1 | 2 ++ tests/f_multithread_yes/expect.2 | 23 ++++++++++++++++++ tests/f_multithread_yes/image.gz | 1 + tests/f_multithread_yes/name | 1 + tests/f_multithread_yes/script | 4 ++++ 27 files changed, 217 insertions(+), 6 deletions(-) create mode 100644 tests/f_multithread/expect.1 create mode 100644 tests/f_multithread/expect.2 create mode 120000 tests/f_multithread/image.gz create mode 100644 tests/f_multithread/name create mode 100644 tests/f_multithread/script create mode 100644 tests/f_multithread_completion/expect.1 create mode 100644 tests/f_multithread_completion/expect.2 create mode 120000 tests/f_multithread_completion/image.gz create mode 100644 tests/f_multithread_completion/name create mode 100644 tests/f_multithread_completion/script create mode 100644 tests/f_multithread_no/expect.1 create mode 100644 tests/f_multithread_no/expect.2 create mode 120000 tests/f_multithread_no/image.gz create mode 100644 tests/f_multithread_no/name create mode 100644 tests/f_multithread_no/script create mode 100644 tests/f_multithread_preen/expect.1 create mode 100644 tests/f_multithread_preen/expect.2 create mode 120000 tests/f_multithread_preen/image.gz create mode 100644 tests/f_multithread_preen/name create mode 100644 tests/f_multithread_preen/script create mode 100644 tests/f_multithread_yes/expect.1 create mode 100644 tests/f_multithread_yes/expect.2 create mode 120000 tests/f_multithread_yes/image.gz create mode 100644 tests/f_multithread_yes/name create mode 100644 tests/f_multithread_yes/script diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 85f953b2..e2784248 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -177,6 +177,7 @@ struct resource_track { #define E2F_OPT_ICOUNT_FULLMAP 0x20000 /* use an array for inode counts */ #define E2F_OPT_UNSHARE_BLOCKS 0x40000 #define E2F_OPT_CLEAR_UNINIT 0x80000 /* Hack to clear the uninit bit */ +#define E2F_OPT_MULTITHREAD 0x100000 /* Use multiple threads to speedup */ /* * E2fsck flags diff --git a/e2fsck/unix.c b/e2fsck/unix.c index 1cb51672..051b31a5 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -75,13 +75,14 @@ int journal_enable_debug = -1; static void usage(e2fsck_t ctx) { fprintf(stderr, - _("Usage: %s [-panyrcdfktvDFV] [-b superblock] [-B blocksize]\n" + _("Usage: %s [-pamnyrcdfktvDFV] [-b superblock] [-B blocksize]\n" "\t\t[-l|-L bad_blocks_file] [-C fd] [-j external_journal]\n" "\t\t[-E extended-options] [-z undo_file] device\n"), ctx->program_name); fprintf(stderr, "%s", _("\nEmergency help:\n" " -p Automatic repair (no questions)\n" + " -m multiple threads to speedup fsck\n" " -n Make no changes to the filesystem\n" " -y Assume \"yes\" to all questions\n" " -c Check for bad blocks and add them to the badblock list\n" @@ -847,7 +848,8 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) phys_mem_kb = get_memory_size() / 1024; ctx->readahead_kb = ~0ULL; - while ((c = getopt(argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) + + while ((c = getopt(argc, argv, "pamnyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) switch (c) { case 'C': ctx->progress = e2fsck_update_progress; @@ -888,6 +890,9 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) } ctx->options |= E2F_OPT_PREEN; break; + case 'm': + ctx->options |= E2F_OPT_MULTITHREAD; + break; case 'n': if (ctx->options & (E2F_OPT_YES|E2F_OPT_PREEN)) goto conflict_opt; @@ -1006,6 +1011,18 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) _("The -n and -l/-L options are incompatible.")); fatal_error(ctx, 0); } + if (ctx->options & E2F_OPT_MULTITHREAD) { + if ((ctx->options & (E2F_OPT_YES|E2F_OPT_NO|E2F_OPT_PREEN)) == 0) { + com_err(ctx->program_name, 0, "%s", + _("The -m option should be used together with one of -p/-y/-n options.")); + fatal_error(ctx, 0); + } + if (ctx->progress) { + com_err(ctx->program_name, 0, "%s", + _("Only one of the options -C or -m may be specified.")); + fatal_error(ctx, 0); + } + } if (ctx->options & E2F_OPT_NO) ctx->options |= E2F_OPT_READONLY; @@ -1112,10 +1129,12 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) #ifdef SA_RESTART sa.sa_flags = SA_RESTART; #endif - sa.sa_handler = signal_progress_on; - sigaction(SIGUSR1, &sa, 0); - sa.sa_handler = signal_progress_off; - sigaction(SIGUSR2, &sa, 0); + if ((ctx->options & E2F_OPT_MULTITHREAD) == 0) { + sa.sa_handler = signal_progress_on; + sigaction(SIGUSR1, &sa, 0); + sa.sa_handler = signal_progress_off; + sigaction(SIGUSR2, &sa, 0); + } #endif /* Update our PATH to include /sbin if we need to run badblocks */ diff --git a/tests/f_multithread/expect.1 b/tests/f_multithread/expect.1 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread/expect.1 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread/expect.2 b/tests/f_multithread/expect.2 new file mode 100644 index 00000000..a833aefc --- /dev/null +++ b/tests/f_multithread/expect.2 @@ -0,0 +1,7 @@ +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 0 diff --git a/tests/f_multithread/image.gz b/tests/f_multithread/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread/name b/tests/f_multithread/name new file mode 100644 index 00000000..df838ea6 --- /dev/null +++ b/tests/f_multithread/name @@ -0,0 +1 @@ +test "e2fsck -m" option \ No newline at end of file diff --git a/tests/f_multithread/script b/tests/f_multithread/script new file mode 100644 index 00000000..0fe96cd0 --- /dev/null +++ b/tests/f_multithread/script @@ -0,0 +1,4 @@ +FSCK_OPT="-fy -m" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_completion/expect.1 b/tests/f_multithread_completion/expect.1 new file mode 100644 index 00000000..61cac9bb --- /dev/null +++ b/tests/f_multithread_completion/expect.1 @@ -0,0 +1,2 @@ +../e2fsck/e2fsck: Only one of the options -C or -m may be specified. +Exit status is 8 diff --git a/tests/f_multithread_completion/expect.2 b/tests/f_multithread_completion/expect.2 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_completion/expect.2 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_completion/image.gz b/tests/f_multithread_completion/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_completion/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_completion/name b/tests/f_multithread_completion/name new file mode 100644 index 00000000..a959045d --- /dev/null +++ b/tests/f_multithread_completion/name @@ -0,0 +1 @@ +test "e2fsck -m" option conflicts with "-C" \ No newline at end of file diff --git a/tests/f_multithread_completion/script b/tests/f_multithread_completion/script new file mode 100644 index 00000000..bf23cd61 --- /dev/null +++ b/tests/f_multithread_completion/script @@ -0,0 +1,4 @@ +FSCK_OPT="-fy -m -C 1" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_no/expect.1 b/tests/f_multithread_no/expect.1 new file mode 100644 index 00000000..d14c4083 --- /dev/null +++ b/tests/f_multithread_no/expect.1 @@ -0,0 +1,24 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? no + +Free blocks count wrong (11602, counted=11597). +Fix? no + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? no + +Free inodes count wrong (2997, counted=2992). +Fix? no + + +test_filesys: ********** WARNING: Filesystem still has errors ********** + +test_filesys: 11/3008 files (0.0% non-contiguous), 398/12000 blocks +Exit status is 4 diff --git a/tests/f_multithread_no/expect.2 b/tests/f_multithread_no/expect.2 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_no/expect.2 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_no/image.gz b/tests/f_multithread_no/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_no/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_no/name b/tests/f_multithread_no/name new file mode 100644 index 00000000..fa49692e --- /dev/null +++ b/tests/f_multithread_no/name @@ -0,0 +1 @@ +test "e2fsck -m" option works with "-n" \ No newline at end of file diff --git a/tests/f_multithread_no/script b/tests/f_multithread_no/script new file mode 100644 index 00000000..b93deb3a --- /dev/null +++ b/tests/f_multithread_no/script @@ -0,0 +1,4 @@ +FSCK_OPT="-fn -m" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_preen/expect.1 b/tests/f_multithread_preen/expect.1 new file mode 100644 index 00000000..b4b0cd9a --- /dev/null +++ b/tests/f_multithread_preen/expect.1 @@ -0,0 +1,11 @@ +../e2fsck/e2fsck: Bad magic number in super-block while trying to open test.img +test_filesys: +The superblock could not be read or does not describe a valid ext2/ext3/ext4 +filesystem. If the device is valid and it really contains an ext2/ext3/ext4 +filesystem (and not swap or ufs or something else), then the superblock +is corrupt, and you might try running e2fsck with an alternate superblock: + e2fsck -b 8193 + or + e2fsck -b 32768 + +Exit status is 8 diff --git a/tests/f_multithread_preen/expect.2 b/tests/f_multithread_preen/expect.2 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_preen/expect.2 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_preen/image.gz b/tests/f_multithread_preen/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_preen/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_preen/name b/tests/f_multithread_preen/name new file mode 100644 index 00000000..90d199df --- /dev/null +++ b/tests/f_multithread_preen/name @@ -0,0 +1 @@ +test "e2fsck -m" option works with "-p" \ No newline at end of file diff --git a/tests/f_multithread_preen/script b/tests/f_multithread_preen/script new file mode 100644 index 00000000..ecb79cd6 --- /dev/null +++ b/tests/f_multithread_preen/script @@ -0,0 +1,4 @@ +FSCK_OPT="-fp -m" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_yes/expect.1 b/tests/f_multithread_yes/expect.1 new file mode 100644 index 00000000..8b780ecf --- /dev/null +++ b/tests/f_multithread_yes/expect.1 @@ -0,0 +1,2 @@ +../e2fsck/e2fsck: The -m option should be used together with one of -p/-y/-n options. +Exit status is 8 diff --git a/tests/f_multithread_yes/expect.2 b/tests/f_multithread_yes/expect.2 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_yes/expect.2 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_yes/image.gz b/tests/f_multithread_yes/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_yes/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_yes/name b/tests/f_multithread_yes/name new file mode 100644 index 00000000..3a703195 --- /dev/null +++ b/tests/f_multithread_yes/name @@ -0,0 +1 @@ +test "e2fsck -m" option works with "-y" \ No newline at end of file diff --git a/tests/f_multithread_yes/script b/tests/f_multithread_yes/script new file mode 100644 index 00000000..38891f6a --- /dev/null +++ b/tests/f_multithread_yes/script @@ -0,0 +1,4 @@ +FSCK_OPT="-f -m" +SECOND_FSCK_OPT=-yf + +. $cmd_dir/run_e2fsck From patchwork Wed Nov 18 15:38:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402309 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=Oo8jGrQi; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8K4Mryz9sTv for ; Thu, 19 Nov 2020 02:40:49 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727362AbgKRPkc (ORCPT ); Wed, 18 Nov 2020 10:40:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726574AbgKRPkb (ORCPT ); Wed, 18 Nov 2020 10:40:31 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C3DEC0613D4 for ; Wed, 18 Nov 2020 07:40:30 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id a6so2978677ybi.0 for ; Wed, 18 Nov 2020 07:40:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=VU3wbpmS0ra35z7h/EvQ+ic9V44H3ZT9/pUw9x1+mBM=; b=Oo8jGrQioLK3FHqsas6FFxc5b7ARvAGYGzvN1aYd2YZ6wnqEEqQmfPk+9/UEN+NLnD P5Rs3il8HQ2rpNS83Tz6gToNCj5o07M6NAukRkcmnDXhKC5C4d8LrpHB3W77y/E8VhR4 SeXvD+1zwfJmYOtTqLDYTqS+3srp/I6jrke1Kzou4B8zlHxvQVr1hrH0CovEqNC5zTei bpptOpQJS26sA3fpPEkyxZTIgzMFruhdpm0EifyGxpZojpQl3XEnKo3RYwZm8tdf6Jbj 0iz4llvNQkPkfhq2Oe9l+BOXpRtm0Wue2XmvrbMmwEe6Za5lYYHKZCBTkc7weTy+Sa6O URVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=VU3wbpmS0ra35z7h/EvQ+ic9V44H3ZT9/pUw9x1+mBM=; b=SrgQvGlYBvydkMtOiARjjwSaSJcHDKWEQH56GBhhrkMMIdixtNlu606DGEr5PB3f5i 7vs5y1Nd3ADtzp/GJdD/c34FzUI/k10BhQ9TbRXYq9HjkD67bxtSJMoBBAXyvis/7Hr9 YP5WBhSqiFLY7I73fhg4Mt6Ft2fV1S+AuEv6mmXSwlCl35iDtU8YnYGOCqLg8DNZOQV1 WS3QmCKfi9s1jq993sQzTouYGlCqpKIUksMcEGu2Ji2nuteY6PNbPZYqZmhJrJbv/pKK VafB4n7+JvFw6bh4Ev7X6SR+Yq8nQ5T6wTycGULg73tbcTsyx1Hgml/xahqchy6+b7cz V+MA== X-Gm-Message-State: AOAM531ylEjtMXWpMCIqScUs9o0fruuM0nzPITftyHYTrT/j1Yp9J17Y spyp9637HyLSsOt6IV1AsHt4YmWQ/LKDapCXeTXmW9GWblJg+NU1gsRvC4Y2fJr8wOvmUiDGLjR Lr3lzhURyC/6PV4h1P2dleEDaJrEV2AWsM3V1aOMCYGQdYJoLyEw97ERFKq60xklYDZnDP1zHU9 IZKUwo1+Y= X-Google-Smtp-Source: ABdhPJxU1V/1t1Uz/jDyb8n24XNzCRDa5KdVE3FI9gkI+TMJlo+zuuZ9JYvYmTNVrtnJByxw0kff0YTstvwbU/SZFDA= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:f512:: with SMTP id a18mr10468914ybe.159.1605714029381; Wed, 18 Nov 2020 07:40:29 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:48 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-3-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 02/61] e2fsck: copy context when using multi-thread fsck From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch only copy the context to a new one when -m is enabled. It doesn't actually start any thread. When pass1 test finishes, the new context is copied back to the original context. Since the signal handler only changes the original context, so add global_ctx in "struct e2fsck_struct" and use that to check whether there is any signal of canceling. This patch handles the long jump properly so that all the existing tests can be passed even the context has been copied. Otherwise, test f_expisize_ea_del would fail when aborting. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 114 +++++++++++++++++++++++++++++++++++++++++++++---- e2fsck/unix.c | 1 + 2 files changed, 107 insertions(+), 8 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 8eecd958..64d237d3 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1144,7 +1144,22 @@ static int quota_inum_is_reserved(ext2_filsys fs, ext2_ino_t ino) return 0; } -void e2fsck_pass1(e2fsck_t ctx) +static int e2fsck_should_abort(e2fsck_t ctx) +{ + e2fsck_t global_ctx; + + if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + return 1; + + if (ctx->global_ctx) { + global_ctx = ctx->global_ctx; + if (global_ctx->flags & E2F_FLAG_SIGNAL_MASK) + return 1; + } + return 0; +} + +void e2fsck_pass1_thread(e2fsck_t ctx) { int i; __u64 max_sizes; @@ -1360,7 +1375,7 @@ void e2fsck_pass1(e2fsck_t ctx) if (ino > ino_threshold) pass1_readahead(ctx, &ra_group, &ino_threshold); ehandler_operation(old_op); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto endit; if (pctx.errcode == EXT2_ET_BAD_BLOCK_IN_INODE_TABLE) { /* @@ -1955,7 +1970,7 @@ void e2fsck_pass1(e2fsck_t ctx) if (process_inode_count >= ctx->process_inode_size) { process_inodes(ctx, block_buf); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto endit; } } @@ -2068,6 +2083,89 @@ endit: else ctx->invalid_bitmaps++; } + +static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) +{ + errcode_t retval; + e2fsck_t thread_context; + + retval = ext2fs_get_mem(sizeof(struct e2fsck_struct), &thread_context); + if (retval) { + com_err(global_ctx->program_name, retval, "while allocating memory"); + return retval; + } + memcpy(thread_context, global_ctx, sizeof(struct e2fsck_struct)); + thread_context->fs->priv_data = thread_context; + thread_context->global_ctx = global_ctx; + + *thread_ctx = thread_context; + return 0; +} + +static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + int flags = global_ctx->flags; +#ifdef HAVE_SETJMP_H + jmp_buf old_jmp; + + memcpy(old_jmp, global_ctx->abort_loc, sizeof(jmp_buf)); +#endif + memcpy(global_ctx, thread_ctx, sizeof(struct e2fsck_struct)); +#ifdef HAVE_SETJMP_H + memcpy(global_ctx->abort_loc, old_jmp, sizeof(jmp_buf)); +#endif + /* Keep the global singal flags*/ + global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | + (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); + + global_ctx->fs->priv_data = global_ctx; + ext2fs_free_mem(&thread_ctx); + return 0; +} + +void e2fsck_pass1_multithread(e2fsck_t ctx) +{ + errcode_t retval; + e2fsck_t thread_ctx; + + retval = e2fsck_pass1_thread_prepare(ctx, &thread_ctx); + if (retval) { + com_err(ctx->program_name, 0, + _("while preparing pass1 thread\n")); + ctx->flags |= E2F_FLAG_ABORT; + return; + } + +#ifdef HAVE_SETJMP_H + /* + * When fatal_error() happens, jump to here. The thread + * context's flags will be saved, but its abort_loc will + * be overwritten by original jump buffer for the later + * tests. + */ + if (setjmp(thread_ctx->abort_loc)) { + thread_ctx->flags &= ~E2F_FLAG_SETJMP_OK; + e2fsck_pass1_thread_join(ctx, thread_ctx); + return; + } + thread_ctx->flags |= E2F_FLAG_SETJMP_OK; +#endif + + e2fsck_pass1_thread(thread_ctx); + retval = e2fsck_pass1_thread_join(ctx, thread_ctx); + if (retval) { + com_err(ctx->program_name, 0, + _("while joining pass1 thread\n")); + ctx->flags |= E2F_FLAG_ABORT; + return; + } +} + +void e2fsck_pass1(e2fsck_t ctx) +{ + e2fsck_pass1_multithread(ctx); +} + #undef FINISH_INODE_LOOP /* @@ -2130,7 +2228,7 @@ static void process_inodes(e2fsck_t ctx, char *block_buf) ehandler_operation(buf); check_blocks(ctx, &pctx, block_buf, &inodes_to_process[i].ea_ibody_quota); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) break; } ctx->stashed_inode = old_stashed_inode; @@ -3300,7 +3398,7 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, inlinedata_fs = ext2fs_has_feature_inline_data(ctx->fs->super); if (check_ext_attr(ctx, pctx, block_buf, &ea_block_quota)) { - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto out; pb.num_blocks += EXT2FS_B2C(ctx->fs, ea_block_quota.blocks); } @@ -3355,7 +3453,7 @@ static void check_blocks(e2fsck_t ctx, struct problem_context *pctx, } end_problem_latch(ctx, PR_LATCH_BLOCK); end_problem_latch(ctx, PR_LATCH_TOOBIG); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) goto out; if (pctx->errcode) fix_problem(ctx, PR_1_BLOCK_ITERATE, pctx); @@ -3836,7 +3934,7 @@ static int process_bad_block(ext2_filsys fs, *block_nr = 0; return BLOCK_CHANGED; } - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) return BLOCK_ABORT; } else mark_block_used(ctx, blk); @@ -3933,7 +4031,7 @@ static int process_bad_block(ext2_filsys fs, *block_nr = 0; return BLOCK_CHANGED; } - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) return BLOCK_ABORT; return 0; } diff --git a/e2fsck/unix.c b/e2fsck/unix.c index 051b31a5..42f616e2 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -1445,6 +1445,7 @@ int main (int argc, char *argv[]) } reserve_stdio_fds(); + ctx->global_ctx = NULL; set_up_logging(ctx); if (ctx->logf) { int i; From patchwork Wed Nov 18 15:38:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402310 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=t8m018Og; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8L0jB9z9sTc for ; Thu, 19 Nov 2020 02:40:50 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727376AbgKRPke (ORCPT ); Wed, 18 Nov 2020 10:40:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726574AbgKRPkd (ORCPT ); Wed, 18 Nov 2020 10:40:33 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F71BC0613D6 for ; Wed, 18 Nov 2020 07:40:32 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id n186so2891256ybg.17 for ; Wed, 18 Nov 2020 07:40:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=BUnSJiCRb7gKUFWssFlD1/YSL9ijQFMd0Y48Ctas5M4=; b=t8m018OgOok81vplOE4kgh6U8br+MQcy+WXVd4f8X9eGwHbzah3DN/UuKfnZxp2GH2 6SJopazwnAZkjBi0q0lc6u7ZpayPEAjbKe8OeMuwvCQ3kvucRpeR3+hE4vg3vEMBa86x i/OuH08attkNNHAxRTlue5Pp79027MiQzOBwo+lQTp35zIr0Az7c9j5xtVBKHM7Lxidy gyUGRmxQ66GyGyYN23jHAupR6uXewwqFeXVrI71/luGwdp5C1eidQ2b0RvKcvQh26PMp zugKSHHcLoH9jbL7M/vCtj7FtdYFeO20doLH2VyyR1T4LKBfEZAmM8DmTDW9YEUvdbA/ TTnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BUnSJiCRb7gKUFWssFlD1/YSL9ijQFMd0Y48Ctas5M4=; b=LXPKzGGIuHEIOumoObIqj+vtHCQYyCwBU13Ebzqoa0melxVtKiGMz48ZA5whDO714t Yyr/Pdqv/E4hLp14fl8CwqnHk1t+sm/sCNMgFL0YwKVUdEOx7KNiWVSExqZZOFFCvUkW /HaX0maihZirama+Txu8HqKyIHW6v9VG/pdzKrwdFB7HNTcHoJunJtkANrg7VzZUy/BA e22GOj6n4xSGQjWxQgjU3C8qgoCkRpirSLvbgdcR6lejk4vxgcfc0e5hLFm1+rXU1NG4 GjfPeBkykppvWyMP5ipyc/SintLCL0cJ5K6erDoKVNDD4fnffTsmCXf3jkyfK/rBYtgt s64A== X-Gm-Message-State: AOAM533SXTyqJe9vqOpecxwe88pzoghlJfTxa5O+ZttliKVLz2Zp2V3V IMlhCOy1mWHyJwS3B1f23Q8zlyOeeqjJjIf+0kBmNxjLFkvvty3BccssYoFYDNoc6bOSAMvi4ES ftN4o0fnWe1M78tyrZKgxxXIgz0bwCzXvy4pwzxMKtQmWQ6vIDsABCiqmK1Q3wLS6dM82HcuIZj hqef18bco= X-Google-Smtp-Source: ABdhPJzO4xDdAu+e5ySTHKflDYD2bwh/XQJIBEh/Rnxv7RuMDvnxVNlP4ssFQv95ItAHJa5XBNI8HyhxLfJmt6jfX9A= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a5b:74f:: with SMTP id s15mr7411466ybq.11.1605714031240; Wed, 18 Nov 2020 07:40:31 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:49 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-4-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 03/61] e2fsck: copy fs when using multi-thread fsck From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch only copy the fs to a new one when -m is enabled. It doesn't actually start any thread. When pass1 test finishes, the new fs is copied back to the original context. This patch handles the fs fields in dblist, inode_map and block_map properly. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 41 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 64d237d3..5b4947b0 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -49,6 +49,8 @@ #include "e2fsck.h" #include +/* todo remove this finally */ +#include #include #include "problem.h" @@ -2084,10 +2086,23 @@ endit: ctx->invalid_bitmaps++; } +static void e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) +{ + memcpy(dest, src, sizeof(struct struct_ext2_filsys)); + if (dest->dblist) + dest->dblist->fs = dest; + if (dest->inode_map) + dest->inode_map->fs = dest; + if (dest->block_map) + dest->block_map->fs = dest; +} + static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) { errcode_t retval; e2fsck_t thread_context; + ext2_filsys thread_fs; + ext2_filsys global_fs = global_ctx->fs; retval = ext2fs_get_mem(sizeof(struct e2fsck_struct), &thread_context); if (retval) { @@ -2095,18 +2110,32 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre return retval; } memcpy(thread_context, global_ctx, sizeof(struct e2fsck_struct)); - thread_context->fs->priv_data = thread_context; thread_context->global_ctx = global_ctx; + retval = ext2fs_get_mem(sizeof(struct struct_ext2_filsys), &thread_fs); + if (retval) { + com_err(global_ctx->program_name, retval, "while allocating memory"); + goto out_context; + } + + e2fsck_pass1_copy_fs(thread_fs, global_fs); + thread_fs->priv_data = thread_context; + + thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; +out_context: + ext2fs_free_mem(&thread_context); + return retval; } static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) { - int flags = global_ctx->flags; + int flags = global_ctx->flags; + ext2_filsys thread_fs = thread_ctx->fs; + ext2_filsys global_fs = global_ctx->fs; #ifdef HAVE_SETJMP_H - jmp_buf old_jmp; + jmp_buf old_jmp; memcpy(old_jmp, global_ctx->abort_loc, sizeof(jmp_buf)); #endif @@ -2118,7 +2147,11 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); - global_ctx->fs->priv_data = global_ctx; + e2fsck_pass1_copy_fs(global_fs, thread_fs); + global_fs->priv_data = global_ctx; + global_ctx->fs = global_fs; + + ext2fs_free_mem(&thread_ctx->fs); ext2fs_free_mem(&thread_ctx); return 0; } From patchwork Wed Nov 18 15:38:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402311 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=bBMuZPIh; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8L4c49z9sTv for ; Thu, 19 Nov 2020 02:40:50 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727378AbgKRPke (ORCPT ); Wed, 18 Nov 2020 10:40:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726624AbgKRPke (ORCPT ); Wed, 18 Nov 2020 10:40:34 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA186C0613D4 for ; Wed, 18 Nov 2020 07:40:33 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id h9so2927396ybj.10 for ; Wed, 18 Nov 2020 07:40:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=/kmf5H9XnIJjHPuH6/aFSMDnGdzHvW6OhLldBrCbLGc=; b=bBMuZPIhvSZP9QHUyp4W871QKO8/iBD2+brUZldxms8ZGD3qjC0DW6unojV6HNzpfm SbEHovVv1dGDeQz8jIvONG4+d+zeAS7vvrnYrnmNuh01ue9a/ptVkXMIlAc912ko6qC/ CwSdzkv0A5OreiEYYLMCdasSW8DB1LrBHcUulzqSYLZfCM0Rq4QCxBZneUpQwpCI+5Zm lh0u3VUJBwA+mJje6PrmaTRD5gOYr6eNKo8bW+P7FTsxyy7cs4UlmeNobUL5J2ih3smS v6X40BvveDioarZhDMjr0trnj/7+09Sc/MBgx7XVT/mELxPQwudOBLZKDZUc7y6YqlAx 4mwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/kmf5H9XnIJjHPuH6/aFSMDnGdzHvW6OhLldBrCbLGc=; b=YcJUEIjEqPzikufRsvyxV402RsYVmYkEDMIHq3NudhWRCw9SM0tCooJybrGvU5tCAb eq4Pnp0egiSFQgGl+XNgWthQ0vI0R6A+MRqd0k5Tm5aJHWbqSgP48yRPjcNJJwzHdDFp Qf76YjCuVWobdaDN/VQ2HEvEuzn49eiRQKhM1dH9dW31MhpqCY8RKC5ECT2gm+INx4bf N0efSJ9ny6rXO0r74lyx11OjiPB97jjdfDOS8P+HM09FHtEr9AlLXSpFIcjub3ExQVrq ybgJ0Od3zlUSdQF+fUHFn8yJNMjf+bfQmuXf1xDk4Mi7dyvMsMM5E8wdp5JVKF7I+WoV jowA== X-Gm-Message-State: AOAM532WC6KCMCwTmf/JiUT4SwD5kKewczE+Zj5MpkSkHdCbfLQloll/ bzKAsbkiGuwxmj8yyWRDReRDdnTnRVE02wUxcbscukRf7vsEVEHXJ9XS1CIdMa8xN7gX5LxBMHS /fwm6FDb7zM9ulYsLIvo1JwvnKpUao/HR5Wqf5HydUZVxkJ09wCyVik4ZfGcvHolfS9/P8zd/lS XzHPVq/rs= X-Google-Smtp-Source: ABdhPJyvR8HEK6d2H3GMXNnz+sZfgFYt+Z1uJ4A0eWOK1Sd6rM9GUNInoGR2LZo2AKuctM11eMIE5cvhJR6OQS32z60= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:cf56:: with SMTP id f83mr6056237ybg.54.1605714033080; Wed, 18 Nov 2020 07:40:33 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:50 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-5-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 04/61] e2fsck: clear icache when using multi-thread fsck From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Li Xi , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong icache of fs will be rebuilt when needed, so after copying fs, icache can be inited to NULL. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 5b4947b0..ba513d91 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2086,8 +2086,10 @@ endit: ctx->invalid_bitmaps++; } -static void e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) +static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) { + errcode_t retval; + memcpy(dest, src, sizeof(struct struct_ext2_filsys)); if (dest->dblist) dest->dblist->fs = dest; @@ -2095,6 +2097,29 @@ static void e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) dest->inode_map->fs = dest; if (dest->block_map) dest->block_map->fs = dest; + + /* icache will be rebuilt if needed, so do not copy from @src */ + src->icache = NULL; + return 0; +} + +static void e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) +{ + struct ext2_inode_cache *icache = dest->icache; + + memcpy(dest, src, sizeof(struct struct_ext2_filsys)); + if (dest->dblist) + dest->dblist->fs = dest; + if (dest->inode_map) + dest->inode_map->fs = dest; + if (dest->block_map) + dest->block_map->fs = dest; + dest->icache = icache; + + if (src->icache) { + ext2fs_free_inode_cache(src->icache); + src->icache = NULL; + } } static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) @@ -2118,12 +2143,18 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre goto out_context; } - e2fsck_pass1_copy_fs(thread_fs, global_fs); + retval = e2fsck_pass1_copy_fs(thread_fs, global_fs); + if (retval) { + com_err(global_ctx->program_name, retval, "while copying fs"); + goto out_fs; + } thread_fs->priv_data = thread_context; thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; +out_fs: + ext2fs_free_mem(&thread_fs); out_context: ext2fs_free_mem(&thread_context); return retval; @@ -2147,7 +2178,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); - e2fsck_pass1_copy_fs(global_fs, thread_fs); + e2fsck_pass1_merge_fs(global_fs, thread_fs); global_fs->priv_data = global_ctx; global_ctx->fs = global_fs; From patchwork Wed Nov 18 15:38:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402312 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=aM6C83m8; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8N05j2z9sVC for ; Thu, 19 Nov 2020 02:40:52 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727388AbgKRPkg (ORCPT ); Wed, 18 Nov 2020 10:40:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726624AbgKRPkf (ORCPT ); Wed, 18 Nov 2020 10:40:35 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2C8FC0613D4 for ; Wed, 18 Nov 2020 07:40:35 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id r4so2036886ybs.1 for ; Wed, 18 Nov 2020 07:40:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=/Oov+Vw1l+3Sr5DeUikR0CL8hAZekIvYvpufHts8i7w=; b=aM6C83m8jGQG6S48nmVsHU8svt+eTfMCr4Cs117ZTMQVCC84LlyPTR5MYcxHdh0sf0 EU9V21RI9QjRK6Taevrlc4cpQ81Gfks/KPOQGEn6JpQavjnJfIP5DZHsYBCZLFiLzrX4 xzRnnQV0k7YMhdGXes9Y2vqjOUnG7skkPJC8H/l2/BisBSIVmLTpqcI78TGVYbUVG737 MEqz5ij67Nx7LkHadP9bkPZL3lJ6t0Mxh/yeHDS09TlJj4E/jEDpstFJpBeqwEsGeksz KWvY0Zlamb5Jv5/RllIh7PgzWM4IxxNsrZ0SS6E7sym0RvMA0GiUOz3LwzZnefYZgEPk jlWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/Oov+Vw1l+3Sr5DeUikR0CL8hAZekIvYvpufHts8i7w=; b=W/1iCXDhFXDeEL6jL9/Wyr88d/p/JzcXKOJi0UrXs7rqe23qkeBu5P9RxU/WXURVlL 1F0moVEPiXwg/svMnI0WdIAL6aI5INnBDrBzMUucK3O8q9XuPzJPCj2lYfp6zA3o8hYn xV0638Vrpza/qUMoyvn1/tKQYuGK4+qdzbYZQvFElTCFI6yyNfC36VF0wjiUbWvUeR9D cjmN9bjpU8owN/0z+5yxTILlkw+LPYiMgVN+lx81t6xZu/axzKYoXIfmwVBxNkPJ/UuO KeniFsocD4b8XvAJi0Vp3gwWCVSxDdWGtJVCu21GQn3ZwaAG7IjethybxqToq5d/kFLQ 83zA== X-Gm-Message-State: AOAM530NdaJIqFWJDe2Z40To6x2tejp3oxPEyN2xVlHgDeopjwiPiLMi Nj88dh8j8m24jmCju19DNo2CaWjSZpP3/UTwJefdfjsEcTs5F1VevhDoJJrkhByutSrZgjZO53v DnjIxnHcXG93XR6KwRHb0lUUwMtIp45obNQX/g/CooWdKLgOKvpyI8Y42RzpPhbQ5o2eoC257J+ /Uq+/r7GM= X-Google-Smtp-Source: ABdhPJwus0rqIKnu3vM1mrpH1nL5FJ8c/TiXm/fWjyQFCilXEqfwTNej2eNa9VkWPUs7Nhhrq3vdmjYLyU05ir3hYn4= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:2495:: with SMTP id k143mr1216635ybk.396.1605714034844; Wed, 18 Nov 2020 07:40:34 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:51 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-6-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 05/61] e2fsck: add assert when copying context From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Adding the assert would simplify the copying of context. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index ba513d91..d16bedd3 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -46,6 +46,7 @@ #ifdef HAVE_ERRNO_H #include #endif +#include #include "e2fsck.h" #include @@ -2129,6 +2130,19 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre ext2_filsys thread_fs; ext2_filsys global_fs = global_ctx->fs; + assert(global_ctx->inode_used_map == NULL); + assert(global_ctx->inode_dir_map == NULL); + assert(global_ctx->inode_bb_map == NULL); + assert(global_ctx->inode_imagic_map == NULL); + assert(global_ctx->inode_reg_map == NULL); + assert(global_ctx->inodes_to_rebuild == NULL); + + assert(global_ctx->block_found_map == NULL); + assert(global_ctx->block_dup_map == NULL); + assert(global_ctx->block_ea_map == NULL); + assert(global_ctx->block_metadata_map == NULL); + assert(global_ctx->fs->dblist == NULL); + retval = ext2fs_get_mem(sizeof(struct e2fsck_struct), &thread_context); if (retval) { com_err(global_ctx->program_name, retval, "while allocating memory"); From patchwork Wed Nov 18 15:38:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402313 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=PZPOu0fH; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8N3gC1z9sTv for ; Thu, 19 Nov 2020 02:40:52 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727410AbgKRPki (ORCPT ); Wed, 18 Nov 2020 10:40:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726624AbgKRPkh (ORCPT ); Wed, 18 Nov 2020 10:40:37 -0500 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98D66C0613D4 for ; Wed, 18 Nov 2020 07:40:37 -0800 (PST) Received: by mail-qv1-xf4a.google.com with SMTP id y21so1560206qve.7 for ; Wed, 18 Nov 2020 07:40:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=3EsIdul8B1iDcETbJXNdm4LPFL00xJZO0DAQ9j/K5Jw=; b=PZPOu0fHUW0bQcF+PUUr5SE5HNC/Q2cALj2eLyW7Bu2wRRz+DbGUfWOkSXxJrotytG erK7LrVZSjkA7zYKV0TK0ReXGk8bFjz2cjmc9qbvnisgvUObmEmJcoughy/9HROCuUMd t00ukZ19uWD3ZxgIFxGdR/CVXgoQmZszhzLCIkgbYwh3d+wLzf4eiX6OaExibsr/0mJm 6tf7O/+K7mWHeaHmJlpHX8gyoaCCiWgtiecJbYPnvUQqNXhFqr9u+w2KF0idf17BSDFf s1ioi7TIRnFUwBlTXomneV5ID9BvMyyWW+cXkbVupWQerNUuD8ehO9htADG6CEEkcO/J sfxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=3EsIdul8B1iDcETbJXNdm4LPFL00xJZO0DAQ9j/K5Jw=; b=RG6WjgMUroV5lrHkG7iQ3lZpahHTOIC+68oCufjIo6iDLA9NYQPsXXtQpfHpRZTN6c vHWPMnWbjyUIzUdgNpTjazv1KtmL10iCpQ0yVntISOo1coU8eeZAtXuvsDsA+D2ipzfj fjRku5lWmkC1ueO3VAW+yWYNWFdBUh8/iQ3CmOa8hNGwk83BOUrvIDdGteEFfb8tReIs 2PMFbLyBZGuC2En0PwvC+EAxxbXUvm8FdNEnpYRIvAys0GR5gMHbZAq2k7lJ3DUJ+Cno 20siCo6CbeLyd4swlookNsXYaW0rRmt8sJtnJOVMxZLZltiNdPt2/e3LGvkO2351YoXH 8HwA== X-Gm-Message-State: AOAM531urmpUu+DqVzpygYvXvjaNWabENnwyoW87enH/qYmHVK7a33pq KUBxHFSixeCwbwWR1xp/fZNZgu1JzOu1eyMa5rFffIrpzpqGZVFJwlViYWETXolPHDrB/E1wSf7 iVtejVjl7PffGGFxsdwf4GxyRHS2CO/23Qi0VDkgi4IyC5YXqXS1Pqdt+xTRMN6NyXcYcv5aYoe yCBEnduVg= X-Google-Smtp-Source: ABdhPJzeqc7MiFvXOPS3asQrl23PQQwxFkaAwqigEIvLCX/Thtg64Wy7AUxvoqfH46gOqCtUriG/b4uxO4lEiWCCGhg= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:99c6:: with SMTP id y6mr5048968qve.60.1605714036667; Wed, 18 Nov 2020 07:40:36 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:52 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-7-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 06/61] e2fsck: copy bitmaps when copying context From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch copies bitmap when the copying context. In the multi-thread fsck, each thread use different bitmap that copied from the glboal bitmap. And Bitmaps from multiple threads will be merged into a global one after the pass1 finishes. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 151 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 139 insertions(+), 12 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index d16bedd3..3a4286e1 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2087,6 +2087,22 @@ endit: ctx->invalid_bitmaps++; } +static errcode_t e2fsck_pass1_copy_bitmap(ext2_filsys fs, ext2fs_generic_bitmap *src, + ext2fs_generic_bitmap *dest) +{ + errcode_t ret; + + ret = ext2fs_copy_bitmap(*src, dest); + if (ret) + return ret; + + (*dest)->fs = fs; + ext2fs_free_generic_bmap(*src); + *src = NULL; + + return 0; +} + static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) { errcode_t retval; @@ -2094,33 +2110,52 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) memcpy(dest, src, sizeof(struct struct_ext2_filsys)); if (dest->dblist) dest->dblist->fs = dest; - if (dest->inode_map) - dest->inode_map->fs = dest; - if (dest->block_map) - dest->block_map->fs = dest; + if (src->block_map) { + retval = e2fsck_pass1_copy_bitmap(dest, &src->block_map, + &dest->block_map); + if (retval) + return retval; + } + if (src->inode_map) { + retval = e2fsck_pass1_copy_bitmap(dest, &src->inode_map, + &dest->inode_map); + if (retval) + return retval; + } /* icache will be rebuilt if needed, so do not copy from @src */ src->icache = NULL; return 0; } -static void e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) +static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) { struct ext2_inode_cache *icache = dest->icache; + errcode_t retval = 0; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); if (dest->dblist) dest->dblist->fs = dest; - if (dest->inode_map) - dest->inode_map->fs = dest; - if (dest->block_map) - dest->block_map->fs = dest; + if (src->inode_map) { + retval = e2fsck_pass1_copy_bitmap(dest, &src->inode_map, + &dest->inode_map); + if (retval) + return retval; + } + if (src->block_map) { + retval = e2fsck_pass1_copy_bitmap(dest, &src->block_map, + &dest->block_map); + if (retval) + return retval; + } dest->icache = icache; if (src->icache) { ext2fs_free_inode_cache(src->icache); src->icache = NULL; } + + return retval; } static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) @@ -2174,8 +2209,9 @@ out_context: return retval; } -static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) +static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { + errcode_t retval; int flags = global_ctx->flags; ext2_filsys thread_fs = thread_ctx->fs; ext2_filsys global_fs = global_ctx->fs; @@ -2192,13 +2228,104 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); - e2fsck_pass1_merge_fs(global_fs, thread_fs); + retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); + if (retval) { + com_err(global_ctx->program_name, 0, _("while merging fs\n")); + return retval; + } global_fs->priv_data = global_ctx; global_ctx->fs = global_fs; + if (thread_ctx->inode_used_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->inode_used_map, + &global_ctx->inode_used_map); + if (retval) + return retval; + } + if (thread_ctx->inode_bad_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->inode_bad_map, + &global_ctx->inode_bad_map); + if (retval) + return retval; + } + if (thread_ctx->inode_dir_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->inode_dir_map, + &global_ctx->inode_dir_map); + if (retval) + return retval; + } + if (thread_ctx->inode_bb_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->inode_bb_map, + &global_ctx->inode_bb_map); + if (retval) + return retval; + } + if (thread_ctx->inode_imagic_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->inode_imagic_map, + &global_ctx->inode_imagic_map); + if (retval) + return retval; + } + if (thread_ctx->inode_reg_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->inode_reg_map, + &global_ctx->inode_reg_map); + if (retval) + return retval; + } + if (thread_ctx->inodes_to_rebuild) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->inodes_to_rebuild, + &global_ctx->inodes_to_rebuild); + if (retval) + return retval; + } + if (thread_ctx->block_found_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->block_found_map, + &global_ctx->block_found_map); + if (retval) + return retval; + } + if (thread_ctx->block_dup_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->block_dup_map, + &global_ctx->block_dup_map); + if (retval) + return retval; + } + if (thread_ctx->block_ea_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->block_ea_map, + &global_ctx->block_ea_map); + if (retval) + return retval; + } + if (thread_ctx->block_metadata_map) { + retval = e2fsck_pass1_copy_bitmap(global_fs, + &thread_ctx->block_metadata_map, + &global_ctx->block_metadata_map); + if (retval) + return retval; + } + + return 0; +} + +static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + errcode_t retval; + + retval = e2fsck_pass1_thread_join_one(global_ctx, thread_ctx); ext2fs_free_mem(&thread_ctx->fs); ext2fs_free_mem(&thread_ctx); - return 0; + + return retval; } void e2fsck_pass1_multithread(e2fsck_t ctx) From patchwork Wed Nov 18 15:38:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402314 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=T8phYXG2; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8P0Ndpz9sVS for ; Thu, 19 Nov 2020 02:40:53 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727427AbgKRPkj (ORCPT ); Wed, 18 Nov 2020 10:40:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726624AbgKRPkj (ORCPT ); Wed, 18 Nov 2020 10:40:39 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43DD8C0613D4 for ; Wed, 18 Nov 2020 07:40:39 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id b25so2982126ybj.2 for ; Wed, 18 Nov 2020 07:40:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=zJ7+kIvrxgVWJHLbQfy0p/hhFwuLvV16JAIDT0Z5Cng=; b=T8phYXG2xonLTx5TMF9inTZrvbMxLAhJr0UJrKq3t5Mw2QmBj2DJySDJrSh08PMgeh rBo51bxWO5MNhauWn6BFK3irki/14uK/gCeGdV8fKPQpryHKetNMJx4Hx0NjjXxOV8r+ Babw9MtpTmaonb34Z3XJ2M/h/pGvxkiVHqMeHkRvzsWTURpEOtplR05AsQgbFtEmcDa3 LP3Fi6pUZL+l0kzBahsQ973C+iDhzg6+yekvYdyO2VNOxlm69wBlTVffGVCEaGPte5XY SDSjlAmljJKV9n/zguMlBEF55AN6FOBvMmMviABY2U1tj8SL5S9RLiLriGSWMPuCGLB+ j5tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=zJ7+kIvrxgVWJHLbQfy0p/hhFwuLvV16JAIDT0Z5Cng=; b=c7PAqmdYOXfAKyTSBaQYKRCEwBrh04hBuyqow15ZVu67JszvseNDcEKlF8vDLS+eCY 3FeM80T+CccEhHKye1oEsGdy2fhXjKhWI0YmWGb8KJLuey/uRHU2bSwqeSY7g2oYF9Og RTSnyXiGGnNnNVsA1tQfpUtJjwOVttpuik26zbROyu7YHSFWkDTJ3LQhOlz7e+fLGG59 IsvXFuoEqHSdywzIAaoMC1Cc47zMQiGhaEPC6RA7j8RI5Dzt2Xx3C35KveSzW/tc6QCT A1ODRtLRdOzXK2RQgW8tnqpr0pfB/oMNS7AF3XLfaVMp7aXCbT8QoxjDYSlsOt8Vo5Ll nDFw== X-Gm-Message-State: AOAM531OgBfNp1KSDOBuW8HnUtobrvoSGFMWlcg7qqwpMoVWgiBRTAzr p3Ho57N22aM7tAZ9RBW89P545kwuvD6NCI1CtCivO2oJ1fgp9MaScgekRIWeK8DgBMr/UHgq1h2 j06hqOG+wbrOdFCwHk9cI3aY6namIA6iQ89DGuhgy83JSmT03GQplK1FxfmY44oimbiEU6V2G7S nHJmGuh+w= X-Google-Smtp-Source: ABdhPJwpDrma0qg819Jd9HUG6ceiEBSOcZFtvpUS7eD4shWnRNpFl+Dui0W4ntVPnpXBYzam2s4WsZEqKvDrZLjY7Es= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:2482:: with SMTP id k124mr5981787ybk.343.1605714038476; Wed, 18 Nov 2020 07:40:38 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:53 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-8-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 07/61] e2fsck: copy badblocks when copying fs From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Li Xi , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong This patch copies badblocks when the copying fs. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 3a4286e1..14508dd8 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2123,6 +2123,12 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) return retval; } + if (src->badblocks) { + retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); + if (retval) + return retval; + } + /* icache will be rebuilt if needed, so do not copy from @src */ src->icache = NULL; return 0; @@ -2155,6 +2161,13 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) src->icache = NULL; } + if (src->badblocks) { + retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); + + ext2fs_badblocks_list_free(src->badblocks); + src->badblocks = NULL; + } + return retval; } From patchwork Wed Nov 18 15:38:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402315 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=qzfRpErr; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8P4V3Wz9sVC for ; Thu, 19 Nov 2020 02:40:53 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727451AbgKRPkm (ORCPT ); Wed, 18 Nov 2020 10:40:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726651AbgKRPkl (ORCPT ); Wed, 18 Nov 2020 10:40:41 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 883B8C0613D4 for ; Wed, 18 Nov 2020 07:40:41 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id n10so1412118plk.14 for ; Wed, 18 Nov 2020 07:40:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=qg5BK0rcxP0/PI+nGwJo9iQN43w6xws7xVsxbzy2MlQ=; b=qzfRpErrIEdRTyRTuM7y1Is925M7TR1zoHcNXSVdhqK5h9z7I+s8T6qo1I4QxYaH3y +3U9lNZ8tuBgR0buOLEKTr9WhdZRuX9Ysp4KlktazTTUlKv7Q9UTPQjvojenMOHDIxeJ 1S8ZT1YuFPBF2uKPSqKRy/2FqpOkTDRnjnbEP7jcPFyEEKSu5sl7J2q8WNo361pOGZ0R 3qFVuN9Y7fgaDhgwS9kUq2kobyhBKet1EGUlcpRNVShuJ/TmldV7tzlGBbeAEJlSyP60 clRAuq8YoWDgnZbJs1AYBkvz9SFLYUr1Wb+WR5kJZL+bdjXoTOSx7kkwArZPEjNi5XTw Lguw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qg5BK0rcxP0/PI+nGwJo9iQN43w6xws7xVsxbzy2MlQ=; b=FQ1bemR4LfUr+EnGUz/nKTyyTUCpmHjF1sqKMt35HKEOefSuckoW9V5B8sujZaMhQM lrjmfbToq746T4X8lMKJ2J5eEc9dmRQxuHQSte9e1ujQI5jj/rM6BA2tQrCBZCNF1TbZ vK8HeufAjSwNEe2Z1ZdI0xRjmK2F97PDpuRfDu0YgRkomBIg58mzbO6P+BxR0xab8YxG xvfgrQbZBzrhzK04xwBDlrAWkhTGxiF0zBIgpVd18w2NBM0s158bvN4c37DPyprJRsiR XgzEuSGti9ei/DgpgGXpYjTXEO4VPYOwQhAb1vvpIflwNfe9B9tt0qvTLuwTNIHTgyUI ZUuA== X-Gm-Message-State: AOAM53000aQ2P0n43yRG4S6CAElOKbHvC4SNCcU219JAvOM8YsXh81yT 7wXUxi4dMO3vwwpErEHX1WDPhxiNTv6c4XLuGOON9Z5pqKXRJg4s+Uxsf58C8cTlqYRvFBvrXAk Vvh7T0rQKej/UaI+CbORXojm4z+VuCb1BGBecy5y/hIpldiTFuxU/ooYAbIUc66PqTvOnCRCtIw Ihf229YjY= X-Google-Smtp-Source: ABdhPJwicrj/jmUEg1L2oIFuo/N1vwlTC6ISDwvnPVTGQtgdht749rOHOYazIzuWnRDcl8npoqNcdbG4sZFJ7Mj8b3A= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:90b:293:: with SMTP id az19mr46084pjb.1.1605714040309; Wed, 18 Nov 2020 07:40:40 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:54 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-9-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 08/61] e2fsck: open io-channel when copying fs From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch also add writethrough flag to the thread io-channel. When multiple threads write the same disk, we don't want the data being saved in memory cache. This will be useful in the future, but even without that flag, the tests can be passed too. This patch also cleanup the io channel cache of the global context. Otherwise, after pass1 step, the next steps would use old data saved in the cache. And the cached data might have already been overwritten in pass1. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 3 +++ e2fsck/pass1.c | 46 ++++++++++++++++++++++++++++++++++++++---- e2fsck/unix.c | 1 + lib/ext2fs/ext2_io.h | 2 ++ lib/ext2fs/ext2fs.h | 3 +++ lib/ext2fs/openfs.c | 48 +++++++++++++++++++++++++++++++------------- lib/ext2fs/undo_io.c | 19 ++++++++++++++++++ lib/ext2fs/unix_io.c | 24 +++++++++++++++++++--- 8 files changed, 125 insertions(+), 21 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index e2784248..1416f15e 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -228,6 +228,8 @@ typedef struct e2fsck_struct *e2fsck_t; #define MAX_EXTENT_DEPTH_COUNT 5 struct e2fsck_struct { + /* Global context to get the cancel flag */ + e2fsck_t global_ctx; ext2_filsys fs; const char *program_name; char *filesystem_name; @@ -247,6 +249,7 @@ struct e2fsck_struct { ext2_ino_t free_inodes; int mount_flags; int openfs_flags; + io_manager io_manager; blkid_cache blkid; /* blkid cache */ #ifdef HAVE_SETJMP_H diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 14508dd8..10efa0ed 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2103,7 +2103,9 @@ static errcode_t e2fsck_pass1_copy_bitmap(ext2_filsys fs, ext2fs_generic_bitmap return 0; } -static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) + +static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, e2fsck_t src_context, + ext2_filsys src) { errcode_t retval; @@ -2129,6 +2131,33 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) return retval; } + /* disable it for now */ + src_context->openfs_flags &= ~EXT2_FLAG_EXCLUSIVE; + retval = ext2fs_open_channel(dest, src_context->io_options, + src_context->io_manager, + src_context->openfs_flags, + src->io->block_size); + if (retval) + return retval; + + /* Block size might not be default */ + io_channel_set_blksize(dest->io, src->io->block_size); + ehandler_init(dest->io); + + assert(dest->io->magic == src->io->magic); + assert(dest->io->manager == src->io->manager); + assert(strcmp(dest->io->name, src->io->name) == 0); + assert(dest->io->block_size == src->io->block_size); + assert(dest->io->read_error == src->io->read_error); + assert(dest->io->write_error == src->io->write_error); + assert(dest->io->refcount == src->io->refcount); + assert(dest->io->flags == src->io->flags); + assert(dest->io->app_data == dest); + assert(src->io->app_data == src); + assert(dest->io->align == src->io->align); + + /* The data should be written to disk immediately */ + dest->io->flags |= CHANNEL_FLAGS_WRITETHROUGH; /* icache will be rebuilt if needed, so do not copy from @src */ src->icache = NULL; return 0; @@ -2137,9 +2166,17 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, ext2_filsys src) static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) { struct ext2_inode_cache *icache = dest->icache; - errcode_t retval = 0; + errcode_t retval = 0; + io_channel dest_io; + io_channel dest_image_io; + + dest_io = dest->io; + dest_image_io = dest->image_io; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); + dest->io = dest_io; + dest->image_io = dest_image_io; + dest->icache = icache; if (dest->dblist) dest->dblist->fs = dest; if (src->inode_map) { @@ -2154,7 +2191,6 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) if (retval) return retval; } - dest->icache = icache; if (src->icache) { ext2fs_free_inode_cache(src->icache); @@ -2168,6 +2204,7 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) src->badblocks = NULL; } + io_channel_close(src->io); return retval; } @@ -2205,7 +2242,8 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre goto out_context; } - retval = e2fsck_pass1_copy_fs(thread_fs, global_fs); + io_channel_flush_cleanup(global_fs->io); + retval = e2fsck_pass1_copy_fs(thread_fs, global_ctx, global_fs); if (retval) { com_err(global_ctx->program_name, retval, "while copying fs"); goto out_fs; diff --git a/e2fsck/unix.c b/e2fsck/unix.c index 42f616e2..f973af62 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -1525,6 +1525,7 @@ restart: } ctx->openfs_flags = flags; + ctx->io_manager = io_ptr; retval = try_open_fs(ctx, flags, io_ptr, &fs); if (!ctx->superblock && !(ctx->options & E2F_OPT_PREEN) && diff --git a/lib/ext2fs/ext2_io.h b/lib/ext2fs/ext2_io.h index 5540900a..4ad2fec8 100644 --- a/lib/ext2fs/ext2_io.h +++ b/lib/ext2fs/ext2_io.h @@ -81,6 +81,7 @@ struct struct_io_manager { errcode_t (*write_blk)(io_channel channel, unsigned long block, int count, const void *data); errcode_t (*flush)(io_channel channel); + errcode_t (*flush_cleanup)(io_channel channel); errcode_t (*write_byte)(io_channel channel, unsigned long offset, int count, const void *data); errcode_t (*set_option)(io_channel channel, const char *option, @@ -113,6 +114,7 @@ struct struct_io_manager { #define io_channel_read_blk(c,b,n,d) ((c)->manager->read_blk((c),b,n,d)) #define io_channel_write_blk(c,b,n,d) ((c)->manager->write_blk((c),b,n,d)) #define io_channel_flush(c) ((c)->manager->flush((c))) +#define io_channel_flush_cleanup(c) ((c)->manager->flush_cleanup((c))) #define io_channel_bumpcount(c) ((c)->refcount++) /* io_manager.c */ diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 69c8a3ff..5b76d02e 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -1643,6 +1643,9 @@ extern errcode_t ext2fs_open2(const char *name, const char *io_options, int flags, int superblock, unsigned int block_size, io_manager manager, ext2_filsys *ret_fs); +errcode_t ext2fs_open_channel(ext2_filsys fs, const char *io_options, + io_manager manager, int flags, + int blocksize); /* * The dgrp_t argument to these two functions is not actually a group number * but a block number offset within a group table! Convert with the formula diff --git a/lib/ext2fs/openfs.c b/lib/ext2fs/openfs.c index 3ed1e25c..7e93d145 100644 --- a/lib/ext2fs/openfs.c +++ b/lib/ext2fs/openfs.c @@ -99,6 +99,38 @@ static void block_sha_map_free_entry(void *data) return; } +errcode_t ext2fs_open_channel(ext2_filsys fs, const char *io_options, + io_manager manager, int flags, + int blocksize) +{ + errcode_t retval; + unsigned int io_flags = 0; + + if (flags & EXT2_FLAG_RW) + io_flags |= IO_FLAG_RW; + if (flags & EXT2_FLAG_EXCLUSIVE) + io_flags |= IO_FLAG_EXCLUSIVE; + if (flags & EXT2_FLAG_DIRECT_IO) + io_flags |= IO_FLAG_DIRECT_IO; + retval = manager->open(fs->device_name, io_flags, &fs->io); + if (retval) + return retval; + + if (io_options && + (retval = io_channel_set_options(fs->io, io_options))) + goto out_close; + fs->image_io = fs->io; + fs->io->app_data = fs; + + if (blocksize > 0) + io_channel_set_blksize(fs->io, blocksize); + + return 0; +out_close: + io_channel_close(fs->io); + return retval; +} + /* * Note: if superblock is non-zero, block-size must also be non-zero. * Superblock and block_size can be zero to use the default size. @@ -122,7 +154,7 @@ errcode_t ext2fs_open2(const char *name, const char *io_options, errcode_t retval; unsigned long i, first_meta_bg; __u32 features; - unsigned int blocks_per_group, io_flags; + unsigned int blocks_per_group; blk64_t group_block, blk; char *dest, *cp; int group_zero_adjust = 0; @@ -163,21 +195,9 @@ errcode_t ext2fs_open2(const char *name, const char *io_options, io_options = cp; } - io_flags = 0; - if (flags & EXT2_FLAG_RW) - io_flags |= IO_FLAG_RW; - if (flags & EXT2_FLAG_EXCLUSIVE) - io_flags |= IO_FLAG_EXCLUSIVE; - if (flags & EXT2_FLAG_DIRECT_IO) - io_flags |= IO_FLAG_DIRECT_IO; - retval = manager->open(fs->device_name, io_flags, &fs->io); + retval = ext2fs_open_channel(fs, io_options, manager, flags, 0); if (retval) goto cleanup; - if (io_options && - (retval = io_channel_set_options(fs->io, io_options))) - goto cleanup; - fs->image_io = fs->io; - fs->io->app_data = fs; retval = io_channel_alloc_buf(fs->io, -SUPERBLOCK_SIZE, &fs->super); if (retval) goto cleanup; diff --git a/lib/ext2fs/undo_io.c b/lib/ext2fs/undo_io.c index 19862414..1391a534 100644 --- a/lib/ext2fs/undo_io.c +++ b/lib/ext2fs/undo_io.c @@ -1020,6 +1020,24 @@ static errcode_t undo_flush(io_channel channel) return retval; } +/* + * Flush data buffers to disk and cleanup the cache. + */ +static errcode_t undo_flush_cleanup(io_channel channel) +{ + errcode_t retval = 0; + struct undo_private_data *data; + + EXT2_CHECK_MAGIC(channel, EXT2_ET_MAGIC_IO_CHANNEL); + data = (struct undo_private_data *) channel->private_data; + EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL); + + if (data->real) + retval = io_channel_flush_cleanup(data->real); + + return retval; +} + static errcode_t undo_set_option(io_channel channel, const char *option, const char *arg) { @@ -1091,6 +1109,7 @@ static struct struct_io_manager struct_undo_manager = { .read_blk = undo_read_blk, .write_blk = undo_write_blk, .flush = undo_flush, + .flush_cleanup = undo_flush_cleanup, .write_byte = undo_write_byte, .set_option = undo_set_option, .get_stats = undo_get_stats, diff --git a/lib/ext2fs/unix_io.c b/lib/ext2fs/unix_io.c index 628e60c3..c9defd4b 100644 --- a/lib/ext2fs/unix_io.c +++ b/lib/ext2fs/unix_io.c @@ -1030,9 +1030,9 @@ static errcode_t unix_write_byte(io_channel channel, unsigned long offset, } /* - * Flush data buffers to disk. + * Flush data buffers to disk and invalidate cache if needed */ -static errcode_t unix_flush(io_channel channel) +static errcode_t _unix_flush(io_channel channel, int invalidate) { struct unix_private_data *data; errcode_t retval = 0; @@ -1042,7 +1042,7 @@ static errcode_t unix_flush(io_channel channel) EXT2_CHECK_MAGIC(data, EXT2_ET_MAGIC_UNIX_IO_CHANNEL); #ifndef NO_IO_CACHE - retval = flush_cached_blocks(channel, data, 0); + retval = flush_cached_blocks(channel, data, invalidate); #endif #ifdef HAVE_FSYNC if (!retval && fsync(data->dev) != 0) @@ -1051,6 +1051,22 @@ static errcode_t unix_flush(io_channel channel) return retval; } +/* + * Flush data buffers to disk. + */ +static errcode_t unix_flush(io_channel channel) +{ + return _unix_flush(channel, 0); +} + +/* + * Flush data buffers to disk and invalidate cache. + */ +static errcode_t unix_flush_cleanup(io_channel channel) +{ + return _unix_flush(channel, 1); +} + static errcode_t unix_set_option(io_channel channel, const char *option, const char *arg) { @@ -1225,6 +1241,7 @@ static struct struct_io_manager struct_unix_manager = { .discard = unix_discard, .cache_readahead = unix_cache_readahead, .zeroout = unix_zeroout, + .flush_cleanup = unix_flush_cleanup, }; io_manager unix_io_manager = &struct_unix_manager; @@ -1246,6 +1263,7 @@ static struct struct_io_manager struct_unixfd_manager = { .discard = unix_discard, .cache_readahead = unix_cache_readahead, .zeroout = unix_zeroout, + .flush_cleanup = unix_flush_cleanup, }; io_manager unixfd_io_manager = &struct_unixfd_manager; From patchwork Wed Nov 18 15:38:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402316 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=jdexb142; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8Q2mg5z9sTc for ; Thu, 19 Nov 2020 02:40:54 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727472AbgKRPkn (ORCPT ); Wed, 18 Nov 2020 10:40:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50064 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726651AbgKRPkn (ORCPT ); Wed, 18 Nov 2020 10:40:43 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EC8BC0613D4 for ; Wed, 18 Nov 2020 07:40:43 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id q199so1399663pfc.21 for ; Wed, 18 Nov 2020 07:40:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=AOd0lv18DG3TAacXp8lOAI2DizjzN1cH+leZkyRtJs0=; b=jdexb142H9STPGvBoUyfpA4zYSlpvxJJEd7INsae/Pt0oSnXbLhfSR2AHlwrJEB8eo Bzn+FqGl7//tKRGwoAtqrWU49sdaNJ653PwhWHUhj20X6/hJ37MEl6oNhGUcX4AXF8MF SCu03AbYSym+iUKhji7aYehFCTWVEYI0mlzlYIYrDKxBy8AFxoQoL4RDQjqb+ImWLeVM goHqzQD6+ypoezpQGDVRGJXyl7R09qDPibSJwSOqfRcU9aS4uv5ZXHZZ0SmKjxwdmLeC QaFLMM9bSY07ec+3S6kcIthDZaFHSiR57azX3/j4uoGzAlYgDg5/3uZAGvc3222S0v+g +VDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=AOd0lv18DG3TAacXp8lOAI2DizjzN1cH+leZkyRtJs0=; b=mFP7BI7t/5if5BoHkkVEwQE14/eJEYp038B49sUXZW3O8PAmCJDslOKEeZwRNqn3f1 5rLdPELkSsoQGwNkLZan2swGQsBPETmTNDrYqnQqJ0RBzIMBfqoXdttDDg2H2eAC4fqJ KudNcNyiViuTOMDrmZYBjgRgv2hOSrjvPWKIKG9cJrblxbWWwM98KMOKhLYrZoaVuElI gA5nsjGDVz5mP36MtxulNTq+i+NGzxZgauuNrrGPKbFJLisu/n+RGstmNMtfpV1nYXuh 4ExjCJalSB8mf3tRLi60RU5HVDO4Se3fETlN5GrhwQnJbSP1nflrh2XcFgwZqPSHUK2K gO+w== X-Gm-Message-State: AOAM531dA3B/vju4DI5VynJaLngAx6tGvakEQV6Pe94IyE9rsc1pHvGT drl4zrQvTzxFUhm4Vlc4GMLIfC3Zyv6pgSyxbJcCMt3ydZDyELSqflC6dJqDgVH80l8IjaMNlLd iefUrNN0mY9LEFmY9qnqQhBONaVfw6LdgK18+zsY/214GF8ZqTaQebCGef9TVY38znwqQqGkeoF bmEkbpgQs= X-Google-Smtp-Source: ABdhPJwlR2Smca9XduHd3P6uzllSeTgjZQ9URT1Z6eLw8TLtP95ucJp+obVZIom2diVC1Uxdj0z1QzlgfQfSe7+dowI= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:902:b415:b029:d6:ec35:755b with SMTP id x21-20020a170902b415b02900d6ec35755bmr4938340plr.47.1605714042648; Wed, 18 Nov 2020 07:40:42 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:55 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-10-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 09/61] e2fsck: create logs for mult-threads From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi When multi-threads are used, different logs should be created for different threads. Each thread has log files with suffix of ".$THREAD_INDEX". And this patch adds f_multithread_logfile test case. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 6 ++++-- e2fsck/logfile.c | 10 ++++++++- e2fsck/pass1.c | 23 +++++++++++++++----- tests/f_multithread_logfile/expect.1 | 23 ++++++++++++++++++++ tests/f_multithread_logfile/image.gz | 1 + tests/f_multithread_logfile/name | 1 + tests/f_multithread_logfile/script | 32 ++++++++++++++++++++++++++++ 7 files changed, 88 insertions(+), 8 deletions(-) create mode 100644 tests/f_multithread_logfile/expect.1 create mode 120000 tests/f_multithread_logfile/image.gz create mode 100644 tests/f_multithread_logfile/name create mode 100644 tests/f_multithread_logfile/script diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 1416f15e..5ad0fe93 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -308,8 +308,10 @@ struct e2fsck_struct { /* * For pass1_check_directory and pass1_get_blocks */ - ext2_ino_t stashed_ino; - struct ext2_inode *stashed_inode; + ext2_ino_t stashed_ino; + struct ext2_inode *stashed_inode; + /* Thread index, if global_ctx is null, this field is unused */ + int thread_index; /* * Location of the lost and found directory diff --git a/e2fsck/logfile.c b/e2fsck/logfile.c index 63e9a12f..c5505d27 100644 --- a/e2fsck/logfile.c +++ b/e2fsck/logfile.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "e2fsck.h" #include @@ -291,6 +292,8 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) struct string s, s1, s2; char *s0 = 0, *log_dir = 0, *log_fn = 0; int log_dir_wait = 0; + int string_size; + char string_index[10]; s.s = s1.s = s2.s = 0; @@ -307,6 +310,12 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) goto out; expand_logfn(ctx, log_fn, &s); + if (ctx->global_ctx) { + sprintf(string_index, "%d", ctx->thread_index); + append_string(&s, ".", 1); + append_string(&s, string_index, 0); + } + if ((log_fn[0] == '/') || !log_dir || !log_dir[0]) s0 = s.s; @@ -325,7 +334,6 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) append_string(&s2, log_dir, 0); append_string(&s2, "/", 1); append_string(&s2, s.s, 0); - printf("%s\n", s2.s); } if (s0) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 10efa0ed..bae47a7f 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2250,6 +2250,9 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre } thread_fs->priv_data = thread_context; + thread_context->thread_index = 0; + set_up_logging(thread_context); + thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; @@ -2262,12 +2265,14 @@ out_context: static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { - errcode_t retval; - int flags = global_ctx->flags; - ext2_filsys thread_fs = thread_ctx->fs; - ext2_filsys global_fs = global_ctx->fs; + errcode_t retval; + int flags = global_ctx->flags; + ext2_filsys thread_fs = thread_ctx->fs; + ext2_filsys global_fs = global_ctx->fs; + FILE *global_logf = global_ctx->logf; + FILE *global_problem_logf = global_ctx->problem_logf; #ifdef HAVE_SETJMP_H - jmp_buf old_jmp; + jmp_buf old_jmp; memcpy(old_jmp, global_ctx->abort_loc, sizeof(jmp_buf)); #endif @@ -2286,6 +2291,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx } global_fs->priv_data = global_ctx; global_ctx->fs = global_fs; + global_ctx->logf = global_logf; + global_ctx->problem_logf = global_problem_logf; if (thread_ctx->inode_used_map) { retval = e2fsck_pass1_copy_bitmap(global_fs, @@ -2374,6 +2381,12 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) retval = e2fsck_pass1_thread_join_one(global_ctx, thread_ctx); ext2fs_free_mem(&thread_ctx->fs); + if (thread_ctx->logf) + fclose(thread_ctx->logf); + if (thread_ctx->problem_logf) { + fputs("\n", thread_ctx->problem_logf); + fclose(thread_ctx->problem_logf); + } ext2fs_free_mem(&thread_ctx); return retval; diff --git a/tests/f_multithread_logfile/expect.1 b/tests/f_multithread_logfile/expect.1 new file mode 100644 index 00000000..e2b954d0 --- /dev/null +++ b/tests/f_multithread_logfile/expect.1 @@ -0,0 +1,23 @@ +ext2fs_open2: Bad magic number in super-block +../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes +Pass 2: Checking directory structure +Pass 3: Checking directory connectivity +Pass 4: Checking reference counts +Pass 5: Checking group summary information +Free blocks count wrong for group #0 (7987, counted=7982). +Fix? yes + +Free blocks count wrong (11602, counted=11597). +Fix? yes + +Free inodes count wrong for group #0 (1493, counted=1488). +Fix? yes + +Free inodes count wrong (2997, counted=2992). +Fix? yes + + +test_filesys: ***** FILE SYSTEM WAS MODIFIED ***** +test_filesys: 16/3008 files (0.0% non-contiguous), 403/12000 blocks +Exit status is 1 diff --git a/tests/f_multithread_logfile/image.gz b/tests/f_multithread_logfile/image.gz new file mode 120000 index 00000000..0fd40018 --- /dev/null +++ b/tests/f_multithread_logfile/image.gz @@ -0,0 +1 @@ +../f_zero_super/image.gz \ No newline at end of file diff --git a/tests/f_multithread_logfile/name b/tests/f_multithread_logfile/name new file mode 100644 index 00000000..faaabc3b --- /dev/null +++ b/tests/f_multithread_logfile/name @@ -0,0 +1 @@ +test "e2fsck -m" option works with "-E log_filename=" diff --git a/tests/f_multithread_logfile/script b/tests/f_multithread_logfile/script new file mode 100644 index 00000000..4f9ca6f8 --- /dev/null +++ b/tests/f_multithread_logfile/script @@ -0,0 +1,32 @@ +LOG_FNAME="f_multithread_logfile_xxx" +FSCK_OPT="-fy -m -y -E log_filename=$LOG_FNAME" +SKIP_VERIFY="true" +ONE_PASS_ONLY="true" +SKIP_CLEANUP="true" + +rm -f $LOG_FNAME.* $LOG_FNAME + +. $cmd_dir/run_e2fsck + +rm -f $test_name.ok $test_name.failed +cmp -s $OUT1 $EXP1 +status1=$? + +if [ "$status1" -eq 0 ]; then + if [ ! -f $LOG_FNAME -o ! -f $LOG_FNAME.0 ]; then + echo "$LOG_FNAME or $LOG_FNAME.0 is not created" > $test_name.failed + echo "$test_name: $test_description: failed" + else + echo "$test_name: $test_description: ok" + touch $test_name.ok + fi +else + diff $DIFF_OPTS $test_dir/expect.1 \ + $test_name.1.log >> $test_name.failed + echo "$test_name: $test_description: failed" +fi + +unset IMAGE FSCK_OPT SECOND_FSCK_OPT OUT1 OUT2 EXP1 EXP2 +unset SKIP_VERIFY SKIP_CLEANUP SKIP_GUNZIP ONE_PASS_ONLY PREP_CMD +unset DESCRIPTION SKIP_UNLINK AFTER_CMD PASS_ZERO +unset LOG_FINAME From patchwork Wed Nov 18 15:38:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402317 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=F1dqzJTh; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8Q6678z9sTv for ; Thu, 19 Nov 2020 02:40:54 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727375AbgKRPkp (ORCPT ); Wed, 18 Nov 2020 10:40:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727473AbgKRPkp (ORCPT ); Wed, 18 Nov 2020 10:40:45 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40B25C0613D4 for ; Wed, 18 Nov 2020 07:40:45 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id e3so1526019pgu.1 for ; Wed, 18 Nov 2020 07:40:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=P3/DkTQEeAEAiK7t+V7JXoxL995b7e7MUdppOZEGWyo=; b=F1dqzJThGny0hiWI8nsEagdaOL9EpQNJ62M09v4GAihrwAQqx20gjHydUOOT4YVHju ZaOLrGWW0iR+P8RE1yKxhVw35513WZydJ4tEGYLrSSNbDe4+kGG+Mlmy7qW+5UlldJI8 dS3p1kJRfQXxHtxXnPPRVs2GI1zeCz/mavh1fxObuU6Be5MP8zbFjruH6lEaNXk5wuxr Z4POdnimDytoDBwBd7WaqOWUthow07HACHSR7MU9crRm90+KP5YvuH4XiEzLQ7TTRp1N DTANaMNratTPL07cFNOhMfM/exi/qGVLyB/MeEoQ1x3VO4m5Xt+8HaNc0cmbjmiHXYPf gP1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=P3/DkTQEeAEAiK7t+V7JXoxL995b7e7MUdppOZEGWyo=; b=K/FYof8HuPFSkkjm1E2XcJMtPJqOZJ723dPWyCowNnZdpvwrgZ5y3FhaU5g03GzlVp GlSyMkZJMyRAykKDgXJbUpYmZbFHIKKVvj3y3AJRPvoS/+rs+DbFSWacDrm2mvrHCQq4 ItPt8x6Ti+UfTc3Vqz7yBaepzla9lE/rHQQDVtX5+daS5VCljT7D11L55cCj/Q41KDxW Ry2fGxFd7iHtrXQQcjzLMmpTcN9PT/A0khW0G9NX0UHdwMEKwGaNTpOIA0foHIFojDej kW3YwkJAH6smoS29phJONHFz7y1D2txnHnLr2hQ+zcH1Z/blX1Izcs3kNh+dck0N/neg hJgg== X-Gm-Message-State: AOAM530imdsdDjs54K/Ilq7qgfArEvwsXlXB16n2HYxe/MRRnOOuGDjV IfogULG550H7Ln/5linJmdp9Ipj2gne68DgpTqVdW2gCvq7L3P/n5omg+YG4XNFS6iB6Vt+E+a5 jr0MztONcqTUD6eO55ZdY6Btsbx5BjaPejGsXH7P+owTPH4Sk949Lr2NVtvZZWVK93xa78le1Bd s2015UtKA= X-Google-Smtp-Source: ABdhPJzfr65oHwiIVXp2YciIyMJoEoUgWnS/zZIA99FbT0DctAa4cckAz53Aq7B6Kn/AIWNwbRNPTCkqjBiBlEMhqGA= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:aa7:8052:0:b029:196:4dbb:99fe with SMTP id y18-20020aa780520000b02901964dbb99femr5073047pfm.11.1605714044582; Wed, 18 Nov 2020 07:40:44 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:56 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-11-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 10/61] e2fsck: optionally configure one pfsck thread From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi This patch creates only one thread to do pass1 check. The same codes can be used to create multiple threads, but other functions need to be modified to get ready for that. pfsck support will be enabled with if configured with --enable-pfsck option. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- configure | 81 +++++++++++++++++++----- configure.ac | 19 ++++++ e2fsck/e2fsck.h | 13 ++++ e2fsck/logfile.c | 2 + e2fsck/pass1.c | 153 ++++++++++++++++++++++++++++++++++++++++------ e2fsck/unix.c | 10 +++ lib/config.h.in | 3 + tests/test_one.in | 8 +++ 8 files changed, 256 insertions(+), 33 deletions(-) diff --git a/configure b/configure index d90188af..1bb7a325 100755 --- a/configure +++ b/configure @@ -755,6 +755,7 @@ SET_MAKE VERSION PACKAGE GETTEXT_PACKAGE +PTHREAD_LIB TDB_MAN_COMMENT TDB_CMT UUIDD_CMT @@ -847,7 +848,6 @@ infodir docdir oldincludedir includedir -runstatedir localstatedir sharedstatedir sysconfdir @@ -912,6 +912,7 @@ enable_mmp enable_tdb enable_bmap_stats enable_bmap_stats_ops +enable_pfsck enable_nls enable_threads with_gnu_ld @@ -984,7 +985,6 @@ datadir='${datarootdir}' sysconfdir='${prefix}/etc' sharedstatedir='${prefix}/com' localstatedir='${prefix}/var' -runstatedir='${localstatedir}/run' includedir='${prefix}/include' oldincludedir='/usr/include' docdir='${datarootdir}/doc/${PACKAGE}' @@ -1237,15 +1237,6 @@ do | -silent | --silent | --silen | --sile | --sil) silent=yes ;; - -runstatedir | --runstatedir | --runstatedi | --runstated \ - | --runstate | --runstat | --runsta | --runst | --runs \ - | --run | --ru | --r) - ac_prev=runstatedir ;; - -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \ - | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \ - | --run=* | --ru=* | --r=*) - runstatedir=$ac_optarg ;; - -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb) ac_prev=sbindir ;; -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \ @@ -1383,7 +1374,7 @@ fi for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \ datadir sysconfdir sharedstatedir localstatedir includedir \ oldincludedir docdir infodir htmldir dvidir pdfdir psdir \ - libdir localedir mandir runstatedir + libdir localedir mandir do eval ac_val=\$$ac_var # Remove trailing slashes. @@ -1536,7 +1527,6 @@ Fine tuning of the installation directories: --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] - --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] @@ -1599,6 +1589,7 @@ Optional Features: --disable-tdb disable tdb support --disable-bmap-stats disable collection of bitmap stats. --enable-bmap-stats-ops enable collection of additional bitmap stats + --enable-pfsck enable parallel e2fsck --disable-nls do not use Native Language Support --enable-threads={posix|solaris|pth|windows} specify multithreading API @@ -6121,6 +6112,68 @@ $as_echo "Disabling additional bitmap statistics by default" >&6; } fi +PTHREAD_LIB='' +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_join in -lpthread" >&5 +$as_echo_n "checking for pthread_join in -lpthread... " >&6; } +if ${ac_cv_lib_pthread_pthread_join+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lpthread $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char pthread_join (); +int +main () +{ +return pthread_join (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_pthread_pthread_join=yes +else + ac_cv_lib_pthread_pthread_join=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthread_pthread_join" >&5 +$as_echo "$ac_cv_lib_pthread_pthread_join" >&6; } +if test "x$ac_cv_lib_pthread_pthread_join" = xyes; then : + PTHREAD_LIB=-pthread +fi + + +# Check whether --enable-pfsck was given. +if test "${enable_pfsck+set}" = set; then : + enableval=$enable_pfsck; if test "$enableval" = "no" || test -z "PTHREAD_LIB" +then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: Disabling parallel e2fsck" >&5 +$as_echo "Disabling parallel e2fsck" >&6; } +else + +$as_echo "#define CONFIG_PFSCK 1" >>confdefs.h + + { $as_echo "$as_me:${as_lineno-$LINENO}: result: Enabling parallel e2fsck" >&5 +$as_echo "Enabling parallel e2fsck" >&6; } +fi + +else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: Disabling parallel e2fsck" >&5 +$as_echo "Disabling parallel e2fsck" >&6; } + +fi + MAKEFILE_LIBRARY=$srcdir/lib/Makefile.library GETTEXT_PACKAGE=e2fsprogs @@ -7338,8 +7391,6 @@ main () if (*(data + i) != *(data3 + i)) return 14; close (fd); - free (data); - free (data3); return 0; } _ACEOF diff --git a/configure.ac b/configure.ac index 7d921074..e73dbf50 100644 --- a/configure.ac +++ b/configure.ac @@ -877,6 +877,25 @@ fi AC_MSG_RESULT([Disabling additional bitmap statistics by default]) ) dnl +dnl handle --enable-pfsck +dnl +PTHREAD_LIB='' +AC_CHECK_LIB(pthread,pthread_join,PTHREAD_LIB=-pthread) +AC_SUBST(PTHREAD_LIB) +AC_ARG_ENABLE([pfsck], +[ --enable-pfsck enable parallel e2fsck], +if test "$enableval" = "no" || test -z "PTHREAD_LIB" +then + AC_MSG_RESULT([Disabling parallel e2fsck]) +else + AC_DEFINE(CONFIG_PFSCK, 1, + [Define to 1 if parallel e2fsck is enabled]) + AC_MSG_RESULT([Enabling parallel e2fsck]) +fi +, +AC_MSG_RESULT([Disabling parallel e2fsck]) +) +dnl dnl dnl MAKEFILE_LIBRARY=$srcdir/lib/Makefile.library diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 5ad0fe93..ccb66ae7 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -426,6 +426,19 @@ struct e2fsck_struct { char *undo_file; }; +#ifdef CONFIG_PFSCK +struct e2fsck_thread_info { + /* ID returned by pthread_create() */ + pthread_t eti_thread_id; + /* Application-defined thread index */ + int eti_thread_index; + /* Thread has been started */ + int eti_started; + /* Context used for this thread */ + e2fsck_t eti_thread_ctx; +}; +#endif + /* Data structures to evaluate whether an extent tree needs rebuilding. */ struct extent_tree_level { unsigned int num_extents; diff --git a/e2fsck/logfile.c b/e2fsck/logfile.c index c5505d27..8bda6b81 100644 --- a/e2fsck/logfile.c +++ b/e2fsck/logfile.c @@ -310,11 +310,13 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) goto out; expand_logfn(ctx, log_fn, &s); +#ifdef CONFIG_PFSCK if (ctx->global_ctx) { sprintf(string_index, "%d", ctx->thread_index); append_string(&s, ".", 1); append_string(&s, string_index, 0); } +#endif if ((log_fn[0] == '/') || !log_dir || !log_dir[0]) s0 = s.s; diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index bae47a7f..0d0fe366 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -47,6 +47,9 @@ #include #endif #include +#ifdef CONFIG_PFSCK +#include +#endif #include "e2fsck.h" #include @@ -1162,7 +1165,7 @@ static int e2fsck_should_abort(e2fsck_t ctx) return 0; } -void e2fsck_pass1_thread(e2fsck_t ctx) +void e2fsck_pass1_run(e2fsck_t ctx) { int i; __u64 max_sizes; @@ -2087,6 +2090,7 @@ endit: ctx->invalid_bitmaps++; } +#ifdef CONFIG_PFSCK static errcode_t e2fsck_pass1_copy_bitmap(ext2_filsys fs, ext2fs_generic_bitmap *src, ext2fs_generic_bitmap *dest) { @@ -2392,18 +2396,38 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) return retval; } -void e2fsck_pass1_multithread(e2fsck_t ctx) +static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, + int num_threads, e2fsck_t global_ctx) { - errcode_t retval; - e2fsck_t thread_ctx; + errcode_t rc; + errcode_t ret = 0; + int i; + struct e2fsck_thread_info *pinfo; - retval = e2fsck_pass1_thread_prepare(ctx, &thread_ctx); - if (retval) { - com_err(ctx->program_name, 0, - _("while preparing pass1 thread\n")); - ctx->flags |= E2F_FLAG_ABORT; - return; + for (i = 0; i < num_threads; i++) { + pinfo = &infos[i]; + + if (!pinfo->eti_started) + continue; + + rc = pthread_join(pinfo->eti_thread_id, NULL); + if (rc) { + com_err(global_ctx->program_name, rc, + _("while joining thread\n")); + if (ret == 0) + ret = rc; + } + e2fsck_pass1_thread_join(global_ctx, infos[i].eti_thread_ctx); } + free(infos); + + return ret; +} + +static void *e2fsck_pass1_thread(void *arg) +{ + struct e2fsck_thread_info *info = arg; + e2fsck_t thread_ctx = info->eti_thread_ctx; #ifdef HAVE_SETJMP_H /* @@ -2414,25 +2438,118 @@ void e2fsck_pass1_multithread(e2fsck_t ctx) */ if (setjmp(thread_ctx->abort_loc)) { thread_ctx->flags &= ~E2F_FLAG_SETJMP_OK; - e2fsck_pass1_thread_join(ctx, thread_ctx); - return; + goto out; } thread_ctx->flags |= E2F_FLAG_SETJMP_OK; #endif - e2fsck_pass1_thread(thread_ctx); - retval = e2fsck_pass1_thread_join(ctx, thread_ctx); + e2fsck_pass1_run(thread_ctx); + +out: + return NULL; +} + +static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, + int num_threads, e2fsck_t global_ctx) +{ + struct e2fsck_thread_info *infos; + pthread_attr_t attr; + errcode_t retval; + errcode_t ret; + struct e2fsck_thread_info *tmp_pinfo; + int i; + e2fsck_t thread_ctx; + + retval = pthread_attr_init(&attr); if (retval) { - com_err(ctx->program_name, 0, - _("while joining pass1 thread\n")); - ctx->flags |= E2F_FLAG_ABORT; - return; + com_err(global_ctx->program_name, retval, + _("while setting pthread attribute\n")); + return retval; } + + infos = calloc(num_threads, sizeof(struct e2fsck_thread_info)); + if (infos == NULL) { + retval = -ENOMEM; + com_err(global_ctx->program_name, retval, + _("while allocating memory for threads\n")); + pthread_attr_destroy(&attr); + return retval; + } + + for (i = 0; i < num_threads; i++) { + tmp_pinfo = &infos[i]; + tmp_pinfo->eti_thread_index = i; + retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx); + if (retval) { + com_err(global_ctx->program_name, retval, + _("while preparing pass1 thread\n")); + break; + } + tmp_pinfo->eti_thread_ctx = thread_ctx; + + retval = pthread_create(&tmp_pinfo->eti_thread_id, &attr, + &e2fsck_pass1_thread, tmp_pinfo); + if (retval) { + com_err(global_ctx->program_name, retval, + _("while creating thread\n")); + e2fsck_pass1_thread_join(global_ctx, thread_ctx); + break; + } + + tmp_pinfo->eti_started = 1; + } + + /* destroy the thread attribute object, since it is no longer needed */ + ret = pthread_attr_destroy(&attr); + if (ret) { + com_err(global_ctx->program_name, ret, + _("while destroying thread attribute\n")); + if (retval == 0) + retval = ret; + } + + if (retval) { + e2fsck_pass1_threads_join(infos, num_threads, global_ctx); + return retval; + } + *pinfo = infos; + return 0; +} + +static void e2fsck_pass1_multithread(e2fsck_t global_ctx) +{ + struct e2fsck_thread_info *infos = NULL; + int num_threads = 1; + errcode_t retval; + + retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); + if (retval) { + com_err(global_ctx->program_name, retval, + _("while starting pass1 threads\n")); + goto out_abort; + } + + retval = e2fsck_pass1_threads_join(infos, num_threads, global_ctx); + if (retval) { + com_err(global_ctx->program_name, retval, + _("while joining pass1 threads\n")); + goto out_abort; + } + return; +out_abort: + global_ctx->flags |= E2F_FLAG_ABORT; + return; } +#endif void e2fsck_pass1(e2fsck_t ctx) { + +#ifdef CONFIG_PFSCK e2fsck_pass1_multithread(ctx); +#else + e2fsck_pass1_run(ctx); +#endif } #undef FINISH_INODE_LOOP diff --git a/e2fsck/unix.c b/e2fsck/unix.c index f973af62..30c2bf31 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -82,7 +82,9 @@ static void usage(e2fsck_t ctx) fprintf(stderr, "%s", _("\nEmergency help:\n" " -p Automatic repair (no questions)\n" +#ifdef CONFIG_PFSCK " -m multiple threads to speedup fsck\n" +#endif " -n Make no changes to the filesystem\n" " -y Assume \"yes\" to all questions\n" " -c Check for bad blocks and add them to the badblock list\n" @@ -849,7 +851,11 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) phys_mem_kb = get_memory_size() / 1024; ctx->readahead_kb = ~0ULL; +#ifdef CONFIG_PFSCK while ((c = getopt(argc, argv, "pamnyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) +#else + while ((c = getopt(argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) +#endif switch (c) { case 'C': ctx->progress = e2fsck_update_progress; @@ -890,9 +896,11 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) } ctx->options |= E2F_OPT_PREEN; break; +#ifdef CONFIG_PFSCK case 'm': ctx->options |= E2F_OPT_MULTITHREAD; break; +#endif case 'n': if (ctx->options & (E2F_OPT_YES|E2F_OPT_PREEN)) goto conflict_opt; @@ -1011,6 +1019,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) _("The -n and -l/-L options are incompatible.")); fatal_error(ctx, 0); } +#ifdef CONFIG_PFSCK if (ctx->options & E2F_OPT_MULTITHREAD) { if ((ctx->options & (E2F_OPT_YES|E2F_OPT_NO|E2F_OPT_PREEN)) == 0) { com_err(ctx->program_name, 0, "%s", @@ -1023,6 +1032,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) fatal_error(ctx, 0); } } +#endif if (ctx->options & E2F_OPT_NO) ctx->options |= E2F_OPT_READONLY; diff --git a/lib/config.h.in b/lib/config.h.in index b448482c..21efda62 100644 --- a/lib/config.h.in +++ b/lib/config.h.in @@ -18,6 +18,9 @@ /* Define to 1 to enable mmp support */ #undef CONFIG_MMP +/* Define to 1 if parallel e2fsck is enabled */ +#undef CONFIG_PFSCK + /* Define to 1 to enable tdb support */ #undef CONFIG_TDB diff --git a/tests/test_one.in b/tests/test_one.in index 5d7607ad..c0a0b4fd 100644 --- a/tests/test_one.in +++ b/tests/test_one.in @@ -27,6 +27,7 @@ esac test_dir=$1 cmd_dir=$SRCDIR +pfsck_enabled="no" if test "$TEST_CONFIG"x = x; then TEST_CONFIG=$SRCDIR/test_config @@ -52,6 +53,13 @@ else test_description= fi +$FSCK --help 2>&1 | grep -q -w -- -m && pfsck_enabled=yes +if [ "$pfsck_enabled" != "yes" ] ; then + echo "$test_dir" | grep -q multithread && + echo "$test_name: $test_description: skipped (pfsck disabled)" && + exit 0 +fi + if [ -n "$SKIP_SLOW_TESTS" -a -f $test_dir/is_slow_test ]; then echo "$test_name: $test_description: skipped (slow test)" exit 0 From patchwork Wed Nov 18 15:38:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402318 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=Zw4vY3dS; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8R4Qzdz9sTc for ; Thu, 19 Nov 2020 02:40:55 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727479AbgKRPks (ORCPT ); Wed, 18 Nov 2020 10:40:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727477AbgKRPkr (ORCPT ); Wed, 18 Nov 2020 10:40:47 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E30AC0613D4 for ; Wed, 18 Nov 2020 07:40:47 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id b25so2982507ybj.2 for ; Wed, 18 Nov 2020 07:40:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ndvprwttcZ5SVlppQepEdUUybSw42we0bey0lTA9TnU=; b=Zw4vY3dSfMfuiDwvS26yWBK+Ej8+Puc5hlbu8oGoU/jJMtL1sYqsQnV18lqqjfaqzy nfSX8rdwFHSRBWsqf8rzS2unmXI7ruqD9zQEvH/RxqjDcwKxld82KX9CjMHyEx70oIGv Twu3TrxInqe4qIoA+kJ+TTMXcy04uFF+nkHnAqj6Cetr91fvLpuWBXpb/zsTPJ+MI3ue ZozaVf/SvE6o5/kIBX6Daz3IhO7S3YicxVt3Nie/83BShmF65tk/lmFNuzRIzyr1Yd1p DYgbXG7trDkLNlgkLitTKlfKa6PjJkVn79JNZtldIP+WkzH/1pTHMwUSWI57MpeewSRP Uqwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ndvprwttcZ5SVlppQepEdUUybSw42we0bey0lTA9TnU=; b=YcTLWeMNO+h0dwcoU+jlCmIz+XP4JEiS3jxYr4xYG1im439yvSb152iHFg8nYndBcg UV4x6JGuTc9s+buAdXBMSvjSmPpg26dJRX2sxP46rVzuXV+9PcGF6jI4azKftcHW49Xq pm2zDrpv8hVF4Ny1M+PZ6peR60QnlwjXDi8QHBlF9xgdq4NwutO/EUJS/TYDgMpiOZuK qKDQZNasQAkfW48JIHIR1N1+HePs95sPeMXrG6gagZjip6QgGQ4PSmjWxYCFMIQdflAP y0qhVM4yhyB5Dm3zNvDmj4Ys44K4/VxwDOhjbnLRmocDtQmA/lFBuf1czYIXNI9oYoSw Fj0A== X-Gm-Message-State: AOAM5326q/PRX86JpRmPDk78SDIYpM9vCrgu/9J5cJpiDeZ5Hdk6kfaB k6W1c0sG+/QsielYLfiLk0MgZEzQotR9Y91p8pCr4W5b6tjvIeeUp3yOQN02U6K294NFFAZhBip w8eAmv4rntvKShh5eG6bBU4I5MWW+IQtz7A4r/IY6jOeCgF/vR1mR6ZtPJg7Cs0TCYzyliFcCAr 7C24sIB38= X-Google-Smtp-Source: ABdhPJzmhfTKOFuSUM3NRcKpKPF5Iuqy4Qe8surkVhih7nJ/9VSsEHs9X24I939sXAN0kSx66g5j9iZvx4lr+lZZSMM= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:7444:: with SMTP id p65mr6512909ybc.149.1605714046371; Wed, 18 Nov 2020 07:40:46 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:57 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-12-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 11/61] e2fsck: add start/end group for thread From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi When multi-threads are used for check, each thread needs to jump to different group in pass1 check. This patch adds the group jumping support. But still, only one thread is used to check. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 23 ++++++++++++++++-- e2fsck/logfile.c | 3 ++- e2fsck/pass1.c | 51 +++++++++++++++++++++++++++++++++------ e2fsck/problem.c | 5 ++++ e2fsck/problem.h | 3 +++ lib/ext2fs/ext2_err.et.in | 3 +++ 6 files changed, 78 insertions(+), 10 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index ccb66ae7..ba1af6bf 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -227,6 +227,22 @@ typedef struct e2fsck_struct *e2fsck_t; #define MAX_EXTENT_DEPTH_COUNT 5 +#ifdef CONFIG_PFSCK +/* + * Fields that used for multi-thread + */ +struct e2fsck_thread { + /* Thread index */ + int et_thread_index; + /* The start group number for this thread */ + dgrp_t et_group_start; + /* The end (not included) group number for this thread*/ + dgrp_t et_group_end; + /* The next group number to check */ + dgrp_t et_group_next; +}; +#endif + struct e2fsck_struct { /* Global context to get the cancel flag */ e2fsck_t global_ctx; @@ -310,8 +326,11 @@ struct e2fsck_struct { */ ext2_ino_t stashed_ino; struct ext2_inode *stashed_inode; - /* Thread index, if global_ctx is null, this field is unused */ - int thread_index; + + /* if @global_ctx is null, this field is unused */ +#ifdef CONFIG_PFSCK + struct e2fsck_thread thread_info; +#endif /* * Location of the lost and found directory diff --git a/e2fsck/logfile.c b/e2fsck/logfile.c index 8bda6b81..d177e4b5 100644 --- a/e2fsck/logfile.c +++ b/e2fsck/logfile.c @@ -312,7 +312,8 @@ static FILE *set_up_log_file(e2fsck_t ctx, const char *key, const char *fn) expand_logfn(ctx, log_fn, &s); #ifdef CONFIG_PFSCK if (ctx->global_ctx) { - sprintf(string_index, "%d", ctx->thread_index); + sprintf(string_index, "%d", + ctx->thread_info.et_thread_index); append_string(&s, ".", 1); append_string(&s, string_index, 0); } diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 0d0fe366..bb32511f 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1370,6 +1370,23 @@ void e2fsck_pass1_run(e2fsck_t ctx) /* Set up ctx->lost_and_found if possible */ (void) e2fsck_get_lost_and_found(ctx, 0); +#ifdef CONFIG_PFSCK + if (ctx->global_ctx) { + if (ctx->options & E2F_OPT_DEBUG && + ctx->options & E2F_OPT_MULTITHREAD) + fprintf(stderr, "thread %d jumping to group %d\n", + ctx->thread_info.et_thread_index, + ctx->thread_info.et_group_start); + pctx.errcode = ext2fs_inode_scan_goto_blockgroup(scan, + ctx->thread_info.et_group_start); + if (pctx.errcode) { + fix_problem(ctx, PR_1_PASS_HEADER, &pctx); + ctx->flags |= E2F_FLAG_ABORT; + goto endit; + } + } +#endif + while (1) { if (ino % (fs->super->s_inodes_per_group * 4) == 1) { if (e2fsck_mmp_update(fs)) @@ -1413,6 +1430,8 @@ void e2fsck_pass1_run(e2fsck_t ctx) ext2fs_mark_inode_bitmap2(ctx->inode_used_map, ino); continue; } + if (pctx.errcode == EXT2_ET_SCAN_FINISHED) + break; if (pctx.errcode && pctx.errcode != EXT2_ET_INODE_CSUM_INVALID && pctx.errcode != EXT2_ET_INODE_IS_GARBAGE) { @@ -2212,12 +2231,14 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) return retval; } -static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx) +static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx, + int thread_index) { - errcode_t retval; - e2fsck_t thread_context; - ext2_filsys thread_fs; - ext2_filsys global_fs = global_ctx->fs; + errcode_t retval; + e2fsck_t thread_context; + ext2_filsys thread_fs; + ext2_filsys global_fs = global_ctx->fs; + struct e2fsck_thread *tinfo; assert(global_ctx->inode_used_map == NULL); assert(global_ctx->inode_dir_map == NULL); @@ -2254,9 +2275,15 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre } thread_fs->priv_data = thread_context; - thread_context->thread_index = 0; + thread_context->thread_info.et_thread_index = thread_index; set_up_logging(thread_context); + assert(thread_index == 0); + tinfo = &thread_context->thread_info; + tinfo->et_group_start = 0; + tinfo->et_group_next = 0; + tinfo->et_group_end = thread_fs->group_desc_count; + thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; @@ -2479,7 +2506,7 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, for (i = 0; i < num_threads; i++) { tmp_pinfo = &infos[i]; tmp_pinfo->eti_thread_index = i; - retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx); + retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, i); if (retval) { com_err(global_ctx->program_name, retval, _("while preparing pass1 thread\n")); @@ -2564,6 +2591,7 @@ static errcode_t scan_callback(ext2_filsys fs, { struct scan_callback_struct *scan_struct; e2fsck_t ctx; + struct e2fsck_thread *tinfo; scan_struct = (struct scan_callback_struct *) priv_data; ctx = scan_struct->ctx; @@ -2575,6 +2603,15 @@ static errcode_t scan_callback(ext2_filsys fs, ctx->fs->group_desc_count)) return EXT2_ET_CANCEL_REQUESTED; +#ifdef CONFIG_PFSCK + if (ctx->global_ctx) { + tinfo = &ctx->thread_info; + tinfo->et_group_next++; + if (tinfo->et_group_next >= tinfo->et_group_end) + return EXT2_ET_SCAN_FINISHED; + } +#endif + return 0; } diff --git a/e2fsck/problem.c b/e2fsck/problem.c index e79c853b..22c2652c 100644 --- a/e2fsck/problem.c +++ b/e2fsck/problem.c @@ -1269,6 +1269,11 @@ static struct e2fsck_problem problem_table[] = { N_("Encrypted @i %i has corrupt encryption @a.\n"), PROMPT_CLEAR_INODE, 0, 0, 0, 0 }, + /* Failed to goto block group */ + { PR_1_SCAN_GOTO, + N_("failed to goto block group"), + PROMPT_NONE, PR_FATAL, 0, 0, 0 }, + /* Pass 1b errors */ /* Pass 1B: Rescan for duplicate/bad blocks */ diff --git a/e2fsck/problem.h b/e2fsck/problem.h index 4185e517..e28adebb 100644 --- a/e2fsck/problem.h +++ b/e2fsck/problem.h @@ -707,6 +707,9 @@ struct problem_context { /* Encrypted inode has corrupt encryption extended attribute */ #define PR_1_CORRUPT_ENCRYPTION_XATTR 0x01008B +/* Failed to goto block group */ +#define PR_1_SCAN_GOTO 0x0100A0 + /* * Pass 1b errors */ diff --git a/lib/ext2fs/ext2_err.et.in b/lib/ext2fs/ext2_err.et.in index 0c76fee6..cdb37423 100644 --- a/lib/ext2fs/ext2_err.et.in +++ b/lib/ext2fs/ext2_err.et.in @@ -548,4 +548,7 @@ ec EXT2_ET_EA_INODE_CORRUPTED, ec EXT2_ET_NO_GDESC, "Group descriptors not loaded" +ec EXT2_ET_SCAN_FINISHED, + "Scanning finished" + end From patchwork Wed Nov 18 15:38:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402319 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=afoHOGz7; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8S0kjpz9sPB for ; Thu, 19 Nov 2020 02:40:56 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727481AbgKRPkt (ORCPT ); Wed, 18 Nov 2020 10:40:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727477AbgKRPkt (ORCPT ); Wed, 18 Nov 2020 10:40:49 -0500 Received: from mail-qt1-x84a.google.com (mail-qt1-x84a.google.com [IPv6:2607:f8b0:4864:20::84a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E6CDC0613D4 for ; Wed, 18 Nov 2020 07:40:49 -0800 (PST) Received: by mail-qt1-x84a.google.com with SMTP id h26so1677329qtm.2 for ; Wed, 18 Nov 2020 07:40:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ZnGtja/sj84oN6FfDg+4r0m87hM6YEhtSifoAfNEgoM=; b=afoHOGz7MaVLAy+GOsejvY11HQGdmyf5+LvejNyyQ5wymuYdcCK2w8TQSu5o8/Gi+V o6/HIYdE+anipOffQpOuKgaHRCdkXzioC9RoZ2yllN3dv6wCWYUM+qqcSw1RUcXHOlyd vYuM7vixkcoJ959ECK3+DrJ4YDVlztC/CLUCD6D5WmCk2Flcyqe/kERUJHSvYakaWuHY UZ5qtJOIH1k5tkWtU7RcGslCKoCgeC737VbDGDwl8CqB4aOi/tfm4ffqkAmHVKohUOgT uR9C9HProSYdr9VMX1e5fCfDcYf7yhVz8zcvRWr0OfDGYw4W0hy5qKSdSREdKWVPcYfe J5xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ZnGtja/sj84oN6FfDg+4r0m87hM6YEhtSifoAfNEgoM=; b=fyvhxL7t3gWQ5Rc/AYQx5HqCZ42GHlwyO6VaZCQEZL5KnFstIB1zk+eGSKrLbKQb4H e2emIMzqp9hRDbiRYdcehN+sLU60uTnkR3uhm9ObWcDwg4sfz6wPWeCxcZN9K9VwGijH tlljNB8ns7X+tE8XAyUnr76ZYwhgSP8sm+wrnFH5KZ5HzU4UJY/RkFWB1r6ACYgZA0Yq xGHLVc8bVgk2Q3iflhqteRpZmsGgd46P5kJRCr3j0WN42SMrkB255lLoM+lVLJfAPAf6 zXr5nepNUYpxkjLDe4kZwxJbXsHyY883PuyHcJ84kFY0vCMQhknsI2r2bGJWMtMcgffP MfKw== X-Gm-Message-State: AOAM533qRboRlIGkBImtEnRWMMyp6jtpTVptmAIVOC1y34NuX7x+D3Ml LQ75hPJsKzqv8AjpfjtUVTBMOM0ZP3z+HeTO+qMqRMv2mk7RIbUp77tnlhPxiSLQ4YSTj/k6jlk xX32t7OM/Y/D6LaxA6FD35gYSeA794Rzo/0zKTairL++PocBtOirAznE04vrAGmLGlRPFloM6vp Auw1pIGk8= X-Google-Smtp-Source: ABdhPJyv/UBiUt2jpG+vPOw1wjZ37BDclwGekKpeALVXyXmv99GBYMb1gldHmCLJ2uMEZzbBlhifLf1EeHV2pYlFK+o= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:e0c9:: with SMTP id x9mr5478809qvk.56.1605714048142; Wed, 18 Nov 2020 07:40:48 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:58 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-13-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 12/61] e2fsck: split groups to different threads From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi The start/end groups of a thread is calculated according to the thread number. But still, only one thread is used to check. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index bb32511f..fd354529 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2232,13 +2232,14 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) } static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx, - int thread_index) + int thread_index, int num_threads) { errcode_t retval; e2fsck_t thread_context; ext2_filsys thread_fs; ext2_filsys global_fs = global_ctx->fs; struct e2fsck_thread *tinfo; + dgrp_t average_group; assert(global_ctx->inode_used_map == NULL); assert(global_ctx->inode_dir_map == NULL); @@ -2278,11 +2279,20 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre thread_context->thread_info.et_thread_index = thread_index; set_up_logging(thread_context); - assert(thread_index == 0); + /* + * Distribute work to multiple threads: + * Each thread work on fs->group_desc_count / nthread groups. + */ tinfo = &thread_context->thread_info; - tinfo->et_group_start = 0; - tinfo->et_group_next = 0; - tinfo->et_group_end = thread_fs->group_desc_count; + average_group = thread_fs->group_desc_count / num_threads; + if (average_group == 0) + average_group = 1; + tinfo->et_group_start = average_group * thread_index; + if (thread_index == num_threads - 1) + tinfo->et_group_end = thread_fs->group_desc_count; + else + tinfo->et_group_end = average_group * (thread_index + 1); + tinfo->et_group_next = tinfo->et_group_start; thread_context->fs = thread_fs; *thread_ctx = thread_context; @@ -2506,7 +2516,8 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, for (i = 0; i < num_threads; i++) { tmp_pinfo = &infos[i]; tmp_pinfo->eti_thread_index = i; - retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, i); + retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, + i, num_threads); if (retval) { com_err(global_ctx->program_name, retval, _("while preparing pass1 thread\n")); From patchwork Wed Nov 18 15:38:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402320 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=aCmx3gWv; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8S4GVcz9sTc for ; Thu, 19 Nov 2020 02:40:56 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727483AbgKRPkv (ORCPT ); Wed, 18 Nov 2020 10:40:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727477AbgKRPkv (ORCPT ); Wed, 18 Nov 2020 10:40:51 -0500 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E678DC0613D4 for ; Wed, 18 Nov 2020 07:40:50 -0800 (PST) Received: by mail-qk1-x749.google.com with SMTP id t141so1742650qke.22 for ; Wed, 18 Nov 2020 07:40:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Xw8J8ecgXUipk91IsHmPZ3f663HyE/z2PUAfN1Cueww=; b=aCmx3gWv/o0CbJK5P7FD0MtH8gJPMXuiu9/RPDVErVgi5ahTMsjc78B2ICwomlCKF0 p+H/rc+vSkH5iUv5wS9LkaVlhCeINVT6ftmUW6jEmcRJQ8BVwvwKnPXn9qvGa9TRMY4b I7mV6vrVrOZJufnpVVdDRNsz5Q+yHeOF85aQJnPbw/sZgS+Uw+Edrc66cQXh5UlNHOEU K7WCdBoJFYD1e2i4Nh9kZXAh0nKTZkkT6mkZOfqDKVn8OgHZEAOq+rz08f7pQFZRgnVZ pZaCZjT7Yc3hU0veTLOjgvFB/soUAgXNBwVFW/yNX/+EROJ8hpY5wBFemsMKV2bt4mqE ykeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Xw8J8ecgXUipk91IsHmPZ3f663HyE/z2PUAfN1Cueww=; b=HaHilc19omDbjaAO2Y/7ueavHK5m4yb5/JXXnzHCj8g7bGt+qKRhQ4QFi3zjlaVC2m Y5xnixLx5jnW/OTtLE925Kwgz6fdGER8oZ3pIE8bKX29lVfdaX6fecdhutEjtIr8DLuB 2jjk1KM3YPlYGGELpgqUMjaaODeuVXb2rMrNDdgADYuT7XfDwvNJEEgLz0Yq/rpSdJjE QjEAm3jR5Tw9/rrZorOCNI6trdJpdWXv/UEH0Pi0+nbI276r/tOHxpS1ifDUc5RF4KXE p5DIsuh/25R831HvQ1mPto6Ic61W06thNXz8icKHX2RdzQLKQu8SvD6+N/8O9toMWaO1 hklQ== X-Gm-Message-State: AOAM532HBTUnUUdhaSQVpHy6lJlN4bW0tmqHWPh8wrS/2gXbBBmRJ9fn IPAAa0Cd5mnIUVQBXCtudg3dJQGpQfSIbcuLv3VxVlwv9l0wy/p8Y+v70jkWnqB2G3rFNWkOwOE 5tRvl6e7mzWVA4MHKBU5MEaq0/yz8Kl25+GVB9IVn6mTb7O3P6nsRS3SZOz476E4DbXgWetS0Mt oalmJQLck= X-Google-Smtp-Source: ABdhPJy1fbXFka3VzcYkBCeYUgoNPiu7BDEZr4vShgXvzRqBSCfH5iBE/qC/Fv9p8BLdg+oKuRW82D2taWemGSPlM8U= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:ad4:490d:: with SMTP id bh13mr5585274qvb.14.1605714049984; Wed, 18 Nov 2020 07:40:49 -0800 (PST) Date: Wed, 18 Nov 2020 07:38:59 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-14-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 13/61] e2fsck: print thread log properly From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi When multi-thread fsck is enabled, logs printed from multiple threads could overlap with each other. The overlap sometimes makes the logs unreadable because log_out() is used multiple times for a single line. This patch adds leading [Thread XXX] to each logs if multi-thread is enabed by -m option. This patch also adds message to show the group ranges and inode numbers for each thread, which is useful for debuging multi-thread check. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 4 ++ e2fsck/pass1.c | 22 +++++++++- e2fsck/problem.c | 6 +++ e2fsck/util.c | 61 ++++++++++++++++++++++++++-- tests/f_multithread/expect.1 | 4 +- tests/f_multithread_logfile/expect.1 | 4 +- tests/f_multithread_no/expect.1 | 4 +- 7 files changed, 98 insertions(+), 7 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index ba1af6bf..06893f67 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -240,6 +240,10 @@ struct e2fsck_thread { dgrp_t et_group_end; /* The next group number to check */ dgrp_t et_group_next; + /* Scanned inode number */ + ext2_ino_t et_inode_number; + char et_log_buf[2048]; + char et_log_length; }; #endif diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index fd354529..528f0a6b 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1441,6 +1441,10 @@ void e2fsck_pass1_run(e2fsck_t ctx) } if (!ino) break; +#ifdef CONFIG_PFSCK + if (ctx->global_ctx) + ctx->thread_info.et_inode_number++; +#endif pctx.ino = ino; pctx.inode = inode; ctx->stashed_ino = ino; @@ -2293,7 +2297,12 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre else tinfo->et_group_end = average_group * (thread_index + 1); tinfo->et_group_next = tinfo->et_group_start; - + tinfo->et_inode_number = 0; + tinfo->et_log_buf[0] = '\0'; + tinfo->et_log_length = 0; + if (thread_context->options & E2F_OPT_MULTITHREAD) + log_out(thread_context, _("Scan group range [%d, %d)\n"), + tinfo->et_group_start, tinfo->et_group_end); thread_context->fs = thread_fs; *thread_ctx = thread_context; return 0; @@ -2334,6 +2343,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->fs = global_fs; global_ctx->logf = global_logf; global_ctx->problem_logf = global_problem_logf; + global_ctx->global_ctx = NULL; if (thread_ctx->inode_used_map) { retval = e2fsck_pass1_copy_bitmap(global_fs, @@ -2483,6 +2493,12 @@ static void *e2fsck_pass1_thread(void *arg) e2fsck_pass1_run(thread_ctx); out: + if (thread_ctx->options & E2F_OPT_MULTITHREAD) + log_out(thread_ctx, + _("Scanned group range [%lu, %lu), inodes %lu\n"), + thread_ctx->thread_info.et_group_start, + thread_ctx->thread_info.et_group_end, + thread_ctx->thread_info.et_inode_number); return NULL; } @@ -2618,6 +2634,10 @@ static errcode_t scan_callback(ext2_filsys fs, if (ctx->global_ctx) { tinfo = &ctx->thread_info; tinfo->et_group_next++; + if (ctx->options & E2F_OPT_DEBUG && + ctx->options & E2F_OPT_MULTITHREAD) + log_out(ctx, _("group %d finished\n"), + tinfo->et_group_next); if (tinfo->et_group_next >= tinfo->et_group_end) return EXT2_ET_SCAN_FINISHED; } diff --git a/e2fsck/problem.c b/e2fsck/problem.c index 22c2652c..1ff6b028 100644 --- a/e2fsck/problem.c +++ b/e2fsck/problem.c @@ -2481,6 +2481,12 @@ int fix_problem(e2fsck_t ctx, problem_t code, struct problem_context *pctx) if (*message) message = _(message); if (!suppress) { +#ifdef CONFIG_PFSCK + if ((ctx->options & E2F_OPT_MULTITHREAD) && ctx->global_ctx) + printf("[Thread %d] ", + ctx->thread_info.et_thread_index); +#endif + if ((ctx->options & E2F_OPT_PREEN) && !(ptr->flags & PR_PREEN_NOHDR)) { printf("%s: ", ctx->device_name ? diff --git a/e2fsck/util.c b/e2fsck/util.c index e0623e4c..a388bd70 100644 --- a/e2fsck/util.c +++ b/e2fsck/util.c @@ -11,6 +11,7 @@ #include "config.h" #include +#include #include #include #include @@ -88,13 +89,67 @@ out: exit(exit_value); } +#ifdef CONFIG_PFSCK +static void thread_log_out(struct e2fsck_thread *tinfo) +{ + printf("[Thread %d] %s", tinfo->et_thread_index, + tinfo->et_log_buf); + tinfo->et_log_length = 0; + tinfo->et_log_buf[0] = '\0'; +} +#endif + void log_out(e2fsck_t ctx, const char *fmt, ...) { va_list pvar; + struct e2fsck_thread *tinfo; + int buf_size; + int msg_size; + int left_size; + int fmt_length = strlen(fmt); + +#ifdef CONFIG_PFSCK + if ((ctx->options & E2F_OPT_MULTITHREAD) && ctx->global_ctx) { + tinfo = &ctx->thread_info; + buf_size = sizeof(tinfo->et_log_buf); + left_size = buf_size - tinfo->et_log_length; + + va_start(pvar, fmt); + msg_size = vsnprintf(tinfo->et_log_buf + tinfo->et_log_length, + left_size, fmt, pvar); + va_end(pvar); + + if (msg_size >= left_size) { + tinfo->et_log_buf[tinfo->et_log_length] = '\0'; + + assert(msg_size < buf_size); + if (msg_size < buf_size) { + thread_log_out(tinfo); + + va_start(pvar, fmt); + msg_size = vsnprintf(tinfo->et_log_buf, buf_size, + fmt, pvar); + va_end(pvar); + + tinfo->et_log_length += msg_size; + tinfo->et_log_buf[tinfo->et_log_length] = '\0'; + } + } else { + tinfo->et_log_length += msg_size; + tinfo->et_log_buf[tinfo->et_log_length] = '\0'; + } + + if (tinfo->et_log_length > 0 && + tinfo->et_log_buf[tinfo->et_log_length - 1] == '\n') + thread_log_out(tinfo); + } else +#endif + { + va_start(pvar, fmt); + vprintf(fmt, pvar); + va_end(pvar); + } - va_start(pvar, fmt); - vprintf(fmt, pvar); - va_end(pvar); if (ctx->logf) { va_start(pvar, fmt); vfprintf(ctx->logf, fmt, pvar); diff --git a/tests/f_multithread/expect.1 b/tests/f_multithread/expect.1 index e2b954d0..8d2acd2b 100644 --- a/tests/f_multithread/expect.1 +++ b/tests/f_multithread/expect.1 @@ -1,6 +1,8 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... -Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scan group range [0, 2) +[Thread 0] Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts diff --git a/tests/f_multithread_logfile/expect.1 b/tests/f_multithread_logfile/expect.1 index e2b954d0..8d2acd2b 100644 --- a/tests/f_multithread_logfile/expect.1 +++ b/tests/f_multithread_logfile/expect.1 @@ -1,6 +1,8 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... -Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scan group range [0, 2) +[Thread 0] Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts diff --git a/tests/f_multithread_no/expect.1 b/tests/f_multithread_no/expect.1 index d14c4083..f85a3382 100644 --- a/tests/f_multithread_no/expect.1 +++ b/tests/f_multithread_no/expect.1 @@ -1,6 +1,8 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... -Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scan group range [0, 2) +[Thread 0] Pass 1: Checking inodes, blocks, and sizes +[Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts From patchwork Wed Nov 18 15:39:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402321 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=CucoB8SX; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8T16F4z9sTv for ; Thu, 19 Nov 2020 02:40:57 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727485AbgKRPkx (ORCPT ); Wed, 18 Nov 2020 10:40:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727477AbgKRPkx (ORCPT ); Wed, 18 Nov 2020 10:40:53 -0500 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BEAB4C0613D4 for ; Wed, 18 Nov 2020 07:40:52 -0800 (PST) Received: by mail-qt1-x849.google.com with SMTP id l67so1659180qte.6 for ; Wed, 18 Nov 2020 07:40:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=S3c6Mh8/nw2Ggap0ePvbku8BWleCnPN1cTGG7+MtNvM=; b=CucoB8SXUYAJC+MRuNLX8MFG44XN3pDP5PsMf/s1VCEbadKkDonlHrkbaLaBw1K/uD V1BgP/R0qohZFbcPzxMneJo1EWLiNtUrlO3r8/oXxc8uqvFfyDwjlbru4E/XgcNUmzUr 8foKDa5ZSJO7b7IvYmFfVppmIH7x71MDA4dx5+09/t35zPq2DMAvESAEVMMbMpywmInN r6EO+0Hse+mGQf/cK9yopq9mTSR4ZWWvyAt7XDPz+a1D/8nGuuwoi9JASAUHWpF7exNc j5iPOfK8Pe9JwxTdAs+4jOjUsh2hJm2F08et7bYyxnpRSwAxCp+2Y2S7r9ZUuU5elFc5 PsNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=S3c6Mh8/nw2Ggap0ePvbku8BWleCnPN1cTGG7+MtNvM=; b=TvAVgibYHR0ne//1hT+Y0tRXb/tqgLZIdZhjrbfzgUKxJehNqO3Rh73Jw3bsilCLVg Drp6PlMMxN1A/higT+Ps/llQVOPe+Jnl11i+LCBVvf4FJhvMKqHnRT9i2KqWZPGYDUtH tforV58cLS1VvKTImp0+1AMrsyr1FdA7Yvnrf+tww9svcUU5n7KSUh21DQAP0C61u2KU 6tk+AjYPWCb/sGkqnJhkT+geLDPP/ZlOgPx4wVLW1V/eqSLyOD/8efRPg2fD5IxxE0J6 I+hjPpOU4g14EqLRM5/pg9hpmMrBVgm8Ry4JT/OhNOrn7kLqurWnisjNkw4Nz+NZOqKY umwA== X-Gm-Message-State: AOAM531ZC9Emo0WLsQ6cE3+Tl+m75bM4vERetRaqShR8cT2JVCoLxSae 4KKYO6XI5MP0c5TZEzdw4QgtIYjLYLfO+dKDEpx/o0twXkxQ5be85pXh8geEpzENqZHJutNP6Ln vtdRcB4gRHIqpxSYbGSkVZt9ezs25rdrN1GdzPosgYX6WzsPtaNhaRf5fSTCp4gOS9q/EotDZoh uCfBo4JDw= X-Google-Smtp-Source: ABdhPJyWp/38Ytx1aVPMcooUXBGxRNJMK9r91iLdb6Hj1O6iVODyWgkth3WDDBPD/RoDOLdUT9PCOEG1VkE2qmQ7QbI= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:99e6:: with SMTP id y38mr5506472qve.28.1605714051832; Wed, 18 Nov 2020 07:40:51 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:00 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-15-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 14/61] e2fsck: merge bitmaps after thread completes From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Li Xi , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong A new method merge_bmap has been added to bitmap operations. But only red-black bitmap has that operation now. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 239 +++++++++++++++++++++++--------------- lib/ext2fs/bitmaps.c | 10 ++ lib/ext2fs/blkmap64_rb.c | 65 +++++++++++ lib/ext2fs/bmap64.h | 4 + lib/ext2fs/ext2fs.h | 8 ++ lib/ext2fs/gen_bitmap64.c | 29 +++++ 6 files changed, 264 insertions(+), 91 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 528f0a6b..9e4abad0 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2124,12 +2124,38 @@ static errcode_t e2fsck_pass1_copy_bitmap(ext2_filsys fs, ext2fs_generic_bitmap return ret; (*dest)->fs = fs; - ext2fs_free_generic_bmap(*src); - *src = NULL; return 0; } +static void e2fsck_pass1_free_bitmap(ext2fs_generic_bitmap *bitmap) +{ + if (*bitmap) { + ext2fs_free_generic_bmap(*bitmap); + *bitmap = NULL; + } + +} + +static errcode_t e2fsck_pass1_merge_bitmap(ext2_filsys fs, ext2fs_generic_bitmap *src, + ext2fs_generic_bitmap *dest) +{ + errcode_t ret = 0; + + if (*src) { + if (*dest == NULL) { + *dest = *src; + *src = NULL; + } else { + ret = ext2fs_merge_bitmap(*src, *dest, NULL, NULL); + if (ret) + return ret; + } + (*dest)->fs = fs; + } + + return 0; +} static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, e2fsck_t src_context, ext2_filsys src) @@ -2137,6 +2163,8 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, e2fsck_t src_context, errcode_t retval; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); + dest->inode_map = NULL; + dest->block_map = NULL; if (dest->dblist) dest->dblist->fs = dest; if (src->block_map) { @@ -2196,42 +2224,50 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) errcode_t retval = 0; io_channel dest_io; io_channel dest_image_io; + ext2fs_inode_bitmap inode_map; + ext2fs_block_bitmap block_map; dest_io = dest->io; dest_image_io = dest->image_io; + inode_map = dest->inode_map; + block_map = dest->block_map; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->io = dest_io; dest->image_io = dest_image_io; dest->icache = icache; + dest->inode_map = inode_map; + dest->block_map = block_map; if (dest->dblist) dest->dblist->fs = dest; - if (src->inode_map) { - retval = e2fsck_pass1_copy_bitmap(dest, &src->inode_map, - &dest->inode_map); - if (retval) - return retval; - } - if (src->block_map) { - retval = e2fsck_pass1_copy_bitmap(dest, &src->block_map, - &dest->block_map); - if (retval) - return retval; - } if (src->icache) { ext2fs_free_inode_cache(src->icache); src->icache = NULL; } + retval = e2fsck_pass1_merge_bitmap(dest, &src->inode_map, + &dest->inode_map); + if (retval) + goto out; + + retval = e2fsck_pass1_merge_bitmap(dest, &src->block_map, + &dest->block_map); + if (retval) + goto out; + if (src->badblocks) { retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); ext2fs_badblocks_list_free(src->badblocks); src->badblocks = NULL; } - +out: io_channel_close(src->io); + if (src->inode_map) + ext2fs_free_generic_bmap(src->inode_map); + if (src->block_map) + ext2fs_free_generic_bmap(src->block_map); return retval; } @@ -2321,6 +2357,18 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_filsys global_fs = global_ctx->fs; FILE *global_logf = global_ctx->logf; FILE *global_problem_logf = global_ctx->problem_logf; + ext2fs_inode_bitmap inode_bad_map = global_ctx->inode_bad_map; + ext2fs_inode_bitmap inode_used_map = global_ctx->inode_used_map; + ext2fs_inode_bitmap inode_dir_map = global_ctx->inode_dir_map; + ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; + ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; + ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; + ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; + ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; + ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; + ext2fs_block_bitmap block_metadata_map = global_ctx->block_metadata_map; + ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; + #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2330,6 +2378,19 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx #ifdef HAVE_SETJMP_H memcpy(global_ctx->abort_loc, old_jmp, sizeof(jmp_buf)); #endif + + global_ctx->inode_used_map = inode_used_map; + global_ctx->inode_bad_map = inode_bad_map; + global_ctx->inode_dir_map = inode_dir_map; + global_ctx->inode_bb_map = inode_bb_map; + global_ctx->inode_imagic_map = inode_imagic_map; + global_ctx->inodes_to_rebuild = inodes_to_rebuild; + global_ctx->inode_reg_map = inode_reg_map; + global_ctx->block_found_map = block_found_map; + global_ctx->block_dup_map = block_dup_map; + global_ctx->block_ea_map = block_ea_map; + global_ctx->block_metadata_map = block_metadata_map; + /* Keep the global singal flags*/ global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); @@ -2345,83 +2406,62 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->problem_logf = global_problem_logf; global_ctx->global_ctx = NULL; - if (thread_ctx->inode_used_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->inode_used_map, - &global_ctx->inode_used_map); - if (retval) - return retval; - } - if (thread_ctx->inode_bad_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->inode_bad_map, - &global_ctx->inode_bad_map); - if (retval) - return retval; - } - if (thread_ctx->inode_dir_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->inode_used_map, + &global_ctx->inode_used_map); + if (retval) + return retval; + + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->inode_bad_map, + &global_ctx->inode_bad_map); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, &thread_ctx->inode_dir_map, &global_ctx->inode_dir_map); - if (retval) - return retval; - } - if (thread_ctx->inode_bb_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->inode_bb_map, - &global_ctx->inode_bb_map); - if (retval) - return retval; - } - if (thread_ctx->inode_imagic_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->inode_imagic_map, - &global_ctx->inode_imagic_map); - if (retval) - return retval; - } - if (thread_ctx->inode_reg_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->inode_reg_map, - &global_ctx->inode_reg_map); - if (retval) - return retval; - } - if (thread_ctx->inodes_to_rebuild) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->inodes_to_rebuild, - &global_ctx->inodes_to_rebuild); - if (retval) - return retval; - } - if (thread_ctx->block_found_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->block_found_map, - &global_ctx->block_found_map); - if (retval) - return retval; - } - if (thread_ctx->block_dup_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->block_dup_map, - &global_ctx->block_dup_map); - if (retval) - return retval; - } - if (thread_ctx->block_ea_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->block_ea_map, - &global_ctx->block_ea_map); - if (retval) - return retval; - } - if (thread_ctx->block_metadata_map) { - retval = e2fsck_pass1_copy_bitmap(global_fs, - &thread_ctx->block_metadata_map, - &global_ctx->block_metadata_map); - if (retval) - return retval; - } + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->inode_bb_map, + &global_ctx->inode_bb_map); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->inode_imagic_map, + &global_ctx->inode_imagic_map); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->inode_reg_map, + &global_ctx->inode_reg_map); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->inodes_to_rebuild, + &global_ctx->inodes_to_rebuild); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->block_found_map, + &global_ctx->block_found_map); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->block_dup_map, + &global_ctx->block_dup_map); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->block_ea_map, + &global_ctx->block_ea_map); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, + &thread_ctx->block_metadata_map, + &global_ctx->block_metadata_map); + if (retval) + return retval; return 0; } @@ -2438,6 +2478,17 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) fputs("\n", thread_ctx->problem_logf); fclose(thread_ctx->problem_logf); } + e2fsck_pass1_free_bitmap(&thread_ctx->inode_used_map); + e2fsck_pass1_free_bitmap(&thread_ctx->inode_bad_map); + e2fsck_pass1_free_bitmap(&thread_ctx->inode_dir_map); + e2fsck_pass1_free_bitmap(&thread_ctx->inode_bb_map); + e2fsck_pass1_free_bitmap(&thread_ctx->inode_imagic_map); + e2fsck_pass1_free_bitmap(&thread_ctx->inode_reg_map); + e2fsck_pass1_free_bitmap(&thread_ctx->inodes_to_rebuild); + e2fsck_pass1_free_bitmap(&thread_ctx->block_found_map); + e2fsck_pass1_free_bitmap(&thread_ctx->block_dup_map); + e2fsck_pass1_free_bitmap(&thread_ctx->block_ea_map); + e2fsck_pass1_free_bitmap(&thread_ctx->block_metadata_map); ext2fs_free_mem(&thread_ctx); return retval; @@ -2464,7 +2515,13 @@ static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, if (ret == 0) ret = rc; } - e2fsck_pass1_thread_join(global_ctx, infos[i].eti_thread_ctx); + rc = e2fsck_pass1_thread_join(global_ctx, infos[i].eti_thread_ctx); + if (rc) { + com_err(global_ctx->program_name, rc, + _("while joining pass1 thread\n")); + if (ret == 0) + ret = rc; + } } free(infos); diff --git a/lib/ext2fs/bitmaps.c b/lib/ext2fs/bitmaps.c index 834a3962..000df234 100644 --- a/lib/ext2fs/bitmaps.c +++ b/lib/ext2fs/bitmaps.c @@ -45,6 +45,16 @@ errcode_t ext2fs_copy_bitmap(ext2fs_generic_bitmap src, { return (ext2fs_copy_generic_bmap(src, dest)); } + +errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src, + ext2fs_generic_bitmap dest, + ext2fs_generic_bitmap dup, + ext2fs_generic_bitmap dup_allowed) +{ + return ext2fs_merge_generic_bmap(src, dest, dup, + dup_allowed); +} + void ext2fs_set_bitmap_padding(ext2fs_generic_bitmap map) { ext2fs_set_generic_bmap_padding(map); diff --git a/lib/ext2fs/blkmap64_rb.c b/lib/ext2fs/blkmap64_rb.c index 1fd55274..7150d791 100644 --- a/lib/ext2fs/blkmap64_rb.c +++ b/lib/ext2fs/blkmap64_rb.c @@ -968,11 +968,76 @@ static void rb_print_stats(ext2fs_generic_bitmap_64 bitmap EXT2FS_ATTR((unused)) } #endif +static errcode_t rb_merge_bmap(ext2fs_generic_bitmap_64 src, + ext2fs_generic_bitmap_64 dest, + ext2fs_generic_bitmap_64 dup, + ext2fs_generic_bitmap_64 dup_allowed) +{ + struct ext2fs_rb_private *src_bp, *dest_bp, *dup_bp = NULL; + struct bmap_rb_extent *src_ext; + struct rb_node *src_node; + errcode_t retval = 0; + int dup_found = 0; + __u64 i; + + src_bp = (struct ext2fs_rb_private *) src->private; + dest_bp = (struct ext2fs_rb_private *) dest->private; + if (dup) + dup_bp = (struct ext2fs_rb_private *)dup->private; + src_bp->rcursor = NULL; + dest_bp->rcursor = NULL; + + src_node = ext2fs_rb_first(&src_bp->root); + while (src_node) { + src_ext = node_to_extent(src_node); + retval = rb_test_clear_bmap_extent(dest, + src_ext->start + src->start, + src_ext->count); + if (retval) { + rb_insert_extent(src_ext->start, src_ext->count, + dest_bp); + goto next; + } + + /* unlikely case, do it one by one block */ + for (i = src_ext->start; + i < src_ext->start + src_ext->count; i++) { + retval = rb_test_clear_bmap_extent(dest, i + src->start, 1); + if (retval) { + rb_insert_extent(i, 1, dest_bp); + continue; + } + if (dup_allowed) { + retval = rb_test_clear_bmap_extent(dup_allowed, + i + src->start, 1); + /* not existed in dup_allowed */ + if (retval) { + dup_found = 1; + if (dup_bp) + rb_insert_extent(i, 1, dup_bp); + } /* else we conside it not duplicated */ + } else { + if (dup_bp) + rb_insert_extent(i, 1, dup_bp); + dup_found = 1; + } + } +next: + src_node = ext2fs_rb_next(src_node); + } + + if (dup_found && dup) + return EEXIST; + + return 0; +} + struct ext2_bitmap_ops ext2fs_blkmap64_rbtree = { .type = EXT2FS_BMAP64_RBTREE, .new_bmap = rb_new_bmap, .free_bmap = rb_free_bmap, .copy_bmap = rb_copy_bmap, + .merge_bmap = rb_merge_bmap, .resize_bmap = rb_resize_bmap, .mark_bmap = rb_mark_bmap, .unmark_bmap = rb_unmark_bmap, diff --git a/lib/ext2fs/bmap64.h b/lib/ext2fs/bmap64.h index de334548..555193ee 100644 --- a/lib/ext2fs/bmap64.h +++ b/lib/ext2fs/bmap64.h @@ -72,6 +72,10 @@ struct ext2_bitmap_ops { void (*free_bmap)(ext2fs_generic_bitmap_64 bitmap); errcode_t (*copy_bmap)(ext2fs_generic_bitmap_64 src, ext2fs_generic_bitmap_64 dest); + errcode_t (*merge_bmap)(ext2fs_generic_bitmap_64 src, + ext2fs_generic_bitmap_64 dest, + ext2fs_generic_bitmap_64 dup, + ext2fs_generic_bitmap_64 dup_allowed); errcode_t (*resize_bmap)(ext2fs_generic_bitmap_64 bitmap, __u64 new_end, __u64 new_real_end); diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 5b76d02e..0aa1d94e 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -837,6 +837,10 @@ extern void ext2fs_free_block_bitmap(ext2fs_block_bitmap bitmap); extern void ext2fs_free_inode_bitmap(ext2fs_inode_bitmap bitmap); extern errcode_t ext2fs_copy_bitmap(ext2fs_generic_bitmap src, ext2fs_generic_bitmap *dest); +errcode_t ext2fs_merge_bitmap(ext2fs_generic_bitmap src, + ext2fs_generic_bitmap dest, + ext2fs_generic_bitmap dup, + ext2fs_generic_bitmap dup_allowed); extern errcode_t ext2fs_write_inode_bitmap(ext2_filsys fs); extern errcode_t ext2fs_write_block_bitmap (ext2_filsys fs); extern errcode_t ext2fs_read_inode_bitmap (ext2_filsys fs); @@ -1433,6 +1437,10 @@ void ext2fs_set_generic_bmap_padding(ext2fs_generic_bitmap bmap); errcode_t ext2fs_resize_generic_bmap(ext2fs_generic_bitmap bmap, __u64 new_end, __u64 new_real_end); +errcode_t ext2fs_merge_generic_bmap(ext2fs_generic_bitmap gen_src, + ext2fs_generic_bitmap gen_dest, + ext2fs_generic_bitmap gen_dup, + ext2fs_generic_bitmap dup_allowed); errcode_t ext2fs_compare_generic_bmap(errcode_t neq, ext2fs_generic_bitmap bm1, ext2fs_generic_bitmap bm2); diff --git a/lib/ext2fs/gen_bitmap64.c b/lib/ext2fs/gen_bitmap64.c index b2370667..50617a34 100644 --- a/lib/ext2fs/gen_bitmap64.c +++ b/lib/ext2fs/gen_bitmap64.c @@ -344,6 +344,35 @@ errcode_t ext2fs_copy_generic_bmap(ext2fs_generic_bitmap gen_src, return 0; } +errcode_t ext2fs_merge_generic_bmap(ext2fs_generic_bitmap gen_src, + ext2fs_generic_bitmap gen_dest, + ext2fs_generic_bitmap gen_dup, + ext2fs_generic_bitmap gen_dup_allowed) +{ + ext2fs_generic_bitmap_64 src = (ext2fs_generic_bitmap_64)gen_src; + ext2fs_generic_bitmap_64 dest = (ext2fs_generic_bitmap_64)gen_dest; + ext2fs_generic_bitmap_64 dup = (ext2fs_generic_bitmap_64)gen_dup; + ext2fs_generic_bitmap_64 dup_allowed = (ext2fs_generic_bitmap_64)gen_dup_allowed; + + if (!src || !dest) + return EINVAL; + + if (!EXT2FS_IS_64_BITMAP(src) || !EXT2FS_IS_64_BITMAP(dest) || + (dup && !EXT2FS_IS_64_BITMAP(dup)) || + (dup_allowed && !EXT2FS_IS_64_BITMAP(dup_allowed))) + return EINVAL; + + if (src->bitmap_ops != dest->bitmap_ops || + (dup && src->bitmap_ops != dup->bitmap_ops) || + (dup_allowed && src->bitmap_ops != dup_allowed->bitmap_ops)) + return EINVAL; + + if (src->bitmap_ops->merge_bmap == NULL) + return EOPNOTSUPP; + + return src->bitmap_ops->merge_bmap(src, dest, dup, dup_allowed); +} + errcode_t ext2fs_resize_generic_bmap(ext2fs_generic_bitmap gen_bmap, __u64 new_end, __u64 new_real_end) From patchwork Wed Nov 18 15:39:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402323 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=Z3erXeuf; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8V2p1lz9sTc for ; Thu, 19 Nov 2020 02:40:58 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727495AbgKRPk5 (ORCPT ); Wed, 18 Nov 2020 10:40:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727477AbgKRPk4 (ORCPT ); Wed, 18 Nov 2020 10:40:56 -0500 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC40DC0613D6 for ; Wed, 18 Nov 2020 07:40:54 -0800 (PST) Received: by mail-qv1-xf4a.google.com with SMTP id q6so1538848qvr.21 for ; Wed, 18 Nov 2020 07:40:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=s6vvkqtHHiOY9yhMkp1shpLW5N5pECGkPdJxNegvhgM=; b=Z3erXeufjPGDspy5380mXSFMrs7zWT8BofvOdCuuhU0jOObB1S+5XJCPrADth9S5w6 0I3qFaqpxxMhonEeANh58gi/rdWCUqzBsc4OBqiUySEyIpyU759hklouko7FXb0m3CCL rGLM83qxlFXhfOfrHkNAvbYpndyp2CX0MDtTDTSL1I1lKhwUmtdjxHkRpRdLQmpqzoLv 5nLGuPrrWJQSM4M5sSoKevYjXz/D8zN46yiYAB7hU9EyF4xMmbjmkZf11KwN2biKOfMe SyaT3P1AIHs+jl0h9W4QqJlhZwINIpPMn2IJyKmsUmpsLGXaiPcoQZWF0AK+Kljwyl7I lI5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=s6vvkqtHHiOY9yhMkp1shpLW5N5pECGkPdJxNegvhgM=; b=Vhfs51HWgCB+G2aWPIdmPjg/rs97qqxv1UC0DzJFk030oML1V+b35qF5b/d1JPMG0F nzFu/hkfXtaLUDdpvxpIy+rrx+hQYp1t0ExHborXV40mFPuvohI8WnjclyIR+PyTRikc ZV08GaZ70iuwep+TjAKAPUiEqCqEi9nbDZBwcDuKFgO+9q0O7UIQ5TjhqgNMAKEWCc2W 1P5EP0snyWKh6MaGeu5e2y6nPwv0zrELml2/vyOoTgTCUAN/eH34UGWkI4rsBi+6zGVc IHZnTllq557H2YGaHlspws3lGWZD8xQHRFwYHwgYpyL9Od4D5VtYwku8+40tTRmYcaDL lvVQ== X-Gm-Message-State: AOAM533Nu+WzqotF3ytQhNGPQP0gbe7wGTWqYDLVqOrxX3CJXi1g1EnA i6eaR2DpGGS3EoiPNZBrfwJ41jCQBgslZMo+poDmHipM3WNilp+RYm1LhPnwRFdZ9NAZg/8cQV1 T3vSe1QNHtX3009t8zVyC8Mmu95KCbx1sp4HoHIh5I2JtGQJpA4rVCflXNYENkW+YEeLIUxGAig 8Zd10o1ws= X-Google-Smtp-Source: ABdhPJyu/Q0SdGo3aK0P/GJK/oFh0dPRGvfyXJX9dlFzrrh1a2iHBPfS3Vo00ml92qzwcnYbVI90mdf9uNWchhJunXs= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:f28d:: with SMTP id k13mr5591389qvl.31.1605714053843; Wed, 18 Nov 2020 07:40:53 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:01 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-16-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 15/61] e2fsck: do not change global variables From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Global variables used in pass1 check are changed to local variables in this patch. This will avoid conflict between threads. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 75 +++++++++++++++++++++++++++++++------------------- 1 file changed, 47 insertions(+), 28 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 9e4abad0..d4a2e707 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -86,7 +86,6 @@ static void alloc_bb_map(e2fsck_t ctx); static void alloc_imagic_map(e2fsck_t ctx); static void mark_inode_bad(e2fsck_t ctx, ino_t ino); static void handle_fs_bad_blocks(e2fsck_t ctx); -static void process_inodes(e2fsck_t ctx, char *block_buf); static EXT2_QSORT_TYPE process_inode_cmp(const void *a, const void *b); static errcode_t scan_callback(ext2_filsys fs, ext2_inode_scan scan, dgrp_t group, void * priv_data); @@ -121,15 +120,15 @@ struct process_inode_block { }; struct scan_callback_struct { - e2fsck_t ctx; - char *block_buf; + e2fsck_t ctx; + char *block_buf; + struct process_inode_block *inodes_to_process; + int *process_inode_count; }; -/* - * For the inodes to process list. - */ -static struct process_inode_block *inodes_to_process; -static int process_inode_count; +static void process_inodes(e2fsck_t ctx, char *block_buf, + struct process_inode_block *inodes_to_process, + int *process_inode_count); static __u64 ext2_max_sizes[EXT2_MAX_BLOCK_LOG_SIZE - EXT2_MIN_BLOCK_LOG_SIZE + 1]; @@ -1168,7 +1167,6 @@ static int e2fsck_should_abort(e2fsck_t ctx) void e2fsck_pass1_run(e2fsck_t ctx) { int i; - __u64 max_sizes; ext2_filsys fs = ctx->fs; ext2_ino_t ino = 0; struct ext2_inode *inode = NULL; @@ -1191,6 +1189,8 @@ void e2fsck_pass1_run(e2fsck_t ctx) ext2_ino_t ino_threshold = 0; dgrp_t ra_group = 0; struct ea_quota ea_ibody_quota; + struct process_inode_block *inodes_to_process; + int process_inode_count; init_resource_track(&rtrack, ctx->fs->io); clear_problem_context(&pctx); @@ -1215,17 +1215,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) mtrace_print("Pass 1"); #endif -#define EXT2_BPP(bits) (1ULL << ((bits) - 2)) - - for (i = EXT2_MIN_BLOCK_LOG_SIZE; i <= EXT2_MAX_BLOCK_LOG_SIZE; i++) { - max_sizes = EXT2_NDIR_BLOCKS + EXT2_BPP(i); - max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i); - max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i) * EXT2_BPP(i); - max_sizes = (max_sizes * (1UL << i)); - ext2_max_sizes[i - EXT2_MIN_BLOCK_LOG_SIZE] = max_sizes; - } -#undef EXT2_BPP - imagic_fs = ext2fs_has_feature_imagic_inodes(sb); extent_fs = ext2fs_has_feature_extents(sb); inlinedata_fs = ext2fs_has_feature_inline_data(sb); @@ -1349,6 +1338,8 @@ void e2fsck_pass1_run(e2fsck_t ctx) ctx->stashed_inode = inode; scan_struct.ctx = ctx; scan_struct.block_buf = block_buf; + scan_struct.inodes_to_process = inodes_to_process; + scan_struct.process_inode_count = &process_inode_count; ext2fs_set_inode_callback(scan, scan_callback, &scan_struct); if (ctx->progress && ((ctx->progress)(ctx, 1, 0, ctx->fs->group_desc_count))) @@ -1997,13 +1988,15 @@ void e2fsck_pass1_run(e2fsck_t ctx) goto endit; if (process_inode_count >= ctx->process_inode_size) { - process_inodes(ctx, block_buf); + process_inodes(ctx, block_buf, inodes_to_process, + &process_inode_count); if (e2fsck_should_abort(ctx)) goto endit; } } - process_inodes(ctx, block_buf); + process_inodes(ctx, block_buf, inodes_to_process, + &process_inode_count); ext2fs_close_inode_scan(scan); scan = NULL; @@ -2113,6 +2106,27 @@ endit: ctx->invalid_bitmaps++; } +static void init_ext2_max_sizes() +{ + int i; + __u64 max_sizes; + + /* + * Init ext2_max_sizes which will be immutable and shared between + * threads + */ +#define EXT2_BPP(bits) (1ULL << ((bits) - 2)) + + for (i = EXT2_MIN_BLOCK_LOG_SIZE; i <= EXT2_MAX_BLOCK_LOG_SIZE; i++) { + max_sizes = EXT2_NDIR_BLOCKS + EXT2_BPP(i); + max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i); + max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i) * EXT2_BPP(i); + max_sizes = (max_sizes * (1UL << i)); + ext2_max_sizes[i - EXT2_MIN_BLOCK_LOG_SIZE] = max_sizes; + } +#undef EXT2_BPP +} + #ifdef CONFIG_PFSCK static errcode_t e2fsck_pass1_copy_bitmap(ext2_filsys fs, ext2fs_generic_bitmap *src, ext2fs_generic_bitmap *dest) @@ -2656,6 +2670,7 @@ out_abort: void e2fsck_pass1(e2fsck_t ctx) { + init_ext2_max_sizes(); #ifdef CONFIG_PFSCK e2fsck_pass1_multithread(ctx); #else @@ -2680,7 +2695,9 @@ static errcode_t scan_callback(ext2_filsys fs, scan_struct = (struct scan_callback_struct *) priv_data; ctx = scan_struct->ctx; - process_inodes((e2fsck_t) fs->priv_data, scan_struct->block_buf); + process_inodes((e2fsck_t) fs->priv_data, scan_struct->block_buf, + scan_struct->inodes_to_process, + scan_struct->process_inode_count); if (ctx->progress) if ((ctx->progress)(ctx, 1, group+1, @@ -2706,7 +2723,9 @@ static errcode_t scan_callback(ext2_filsys fs, /* * Process the inodes in the "inodes to process" list. */ -static void process_inodes(e2fsck_t ctx, char *block_buf) +static void process_inodes(e2fsck_t ctx, char *block_buf, + struct process_inode_block *inodes_to_process, + int *process_inode_count) { int i; struct ext2_inode *old_stashed_inode; @@ -2718,15 +2737,15 @@ static void process_inodes(e2fsck_t ctx, char *block_buf) #if 0 printf("begin process_inodes: "); #endif - if (process_inode_count == 0) + if (*process_inode_count == 0) return; old_operation = ehandler_operation(0); old_stashed_inode = ctx->stashed_inode; old_stashed_ino = ctx->stashed_ino; - qsort(inodes_to_process, process_inode_count, + qsort(inodes_to_process, *process_inode_count, sizeof(struct process_inode_block), process_inode_cmp); clear_problem_context(&pctx); - for (i=0; i < process_inode_count; i++) { + for (i=0; i < *process_inode_count; i++) { pctx.inode = ctx->stashed_inode = (struct ext2_inode *) &inodes_to_process[i].inode; pctx.ino = ctx->stashed_ino = inodes_to_process[i].ino; @@ -2744,7 +2763,7 @@ static void process_inodes(e2fsck_t ctx, char *block_buf) } ctx->stashed_inode = old_stashed_inode; ctx->stashed_ino = old_stashed_ino; - process_inode_count = 0; + *process_inode_count = 0; #if 0 printf("end process inodes\n"); #endif From patchwork Wed Nov 18 15:39:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402322 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=P71L2gEd; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8T63TLz9sPB for ; Thu, 19 Nov 2020 02:40:57 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727494AbgKRPk4 (ORCPT ); Wed, 18 Nov 2020 10:40:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50110 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727487AbgKRPk4 (ORCPT ); Wed, 18 Nov 2020 10:40:56 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BC91C0613D4 for ; Wed, 18 Nov 2020 07:40:56 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id j12so1404239plj.20 for ; Wed, 18 Nov 2020 07:40:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=9jrGCNRsBXFJQqCYwRQBYmUDmSfN/AuYXhamlTnyfhs=; b=P71L2gEdD+BeKAK0HYwSPsRe5/peghmLkSaOHKrVGdMobqvnSTbpqhug68acuM98St cTbgMtJYucpQ4OpOE5XeGXpM5ipeuJNRrLOBoYPoYcBM5AB1Byi2zETv2SzDo2VHpi5+ UpiZI9m+/BGiD3bzhXPARF2ArRMnxB74ZwUuYilL+DCPVPYEe06Jt5RMqj/EPu15hyKZ mJouiMzOXecv2DW9CuKC2RQgOp/E8bOm+fo2mAttZuqIk4Kw0idsJgyG7RtepHhiByjb /59p/kEROoTn/IaZ3o3I04L29vhdcTQQVE7vlvxcBn9ynO7w/XPXbEEb8SywY6vmurnr w9BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9jrGCNRsBXFJQqCYwRQBYmUDmSfN/AuYXhamlTnyfhs=; b=IayYsFOYnqcB0WNy4qSiD4Jq1GjzFWHtPu+wBh8kFLiFvOAztT7GHOI0CImhcRTw11 P2DsGyJ1Y3CyaLxJiRRppX1F2+CzKTHriBDbuSEUbQrN0hvpY/4RAm8JzX9ATr/67igc pzE5jjpHkHKrXbizIZV9ccF+fDXtiBSjD7kQfdUfc3L/yNS08ZpMO9/A5hSvYtTxpNME UjBqRy5UQKKZupIruIOfUdCpnEP6KsN4Ji5JSLWxaPzKyZi76QP8UYnvTXBuieXLhhGD 6L/MZCSMMWBmq3qFqHQ/0cRu8H6YE+tDlC6610V5ofbHCYgSAvcwqAUJst77KLY6J4sm UWHQ== X-Gm-Message-State: AOAM533o7a6Q2OmV+2NQqtd0d5FVbQUMoBEzzQXTyxPAld+3RaMGlKsf TiMnE5D6l5mG6ZXauqd4jC00YlnMi46EE9BnUOvzzNPbPTwLTTlfl7Io1MiMiW1v1B0F2T1Szwl gtLbIIJd5kzoHbL1N4gc6yaYK8NMDyfz5CDxVVUhyJ/hso2GVuprCsbIVTAEODfPRxirZUytp+k Od2+uLAF0= X-Google-Smtp-Source: ABdhPJxCTR3h6qOXxtdM2gC4J0ZCIC0sJzilBgzL9JE3tEwapyXcKoAhSIaDWVLBNqSSvLZyKUk6F7vWV6LKkDarfBI= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a62:5c87:0:b029:197:5f13:b66c with SMTP id q129-20020a625c870000b02901975f13b66cmr4857020pfb.73.1605714055709; Wed, 18 Nov 2020 07:40:55 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:02 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-17-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 16/61] e2fsck: optimize the inserting of dir_info_db From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Binary search is now used when inserting an dir info to the array. Memmove is now used when moving array. Both of them improves the performance of inserting. This patch is also a prepartion for the merging of two dir db arrays. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/dirinfo.c | 172 ++++++++++++++++++++++++++++++----------------- 1 file changed, 112 insertions(+), 60 deletions(-) diff --git a/e2fsck/dirinfo.c b/e2fsck/dirinfo.c index 49d624c5..28baaca2 100644 --- a/e2fsck/dirinfo.c +++ b/e2fsck/dirinfo.c @@ -7,6 +7,7 @@ #undef DIRINFO_DEBUG +#include #include "config.h" #include "e2fsck.h" #include @@ -122,6 +123,104 @@ static void setup_db(e2fsck_t ctx) "directory map"); } +/* + * Return the min index that has ino larger or equal to @ino + * If not found, return -ENOENT + */ +static int +e2fsck_dir_info_min_larger_equal(struct dir_info_db *dir_info, + ext2_ino_t ino, ext2_ino_t *index) +{ + ext2_ino_t low = 0; + ext2_ino_t mid, high; + ext2_ino_t tmp_ino; + int found = 0; + + if (dir_info->count == 0) + return -ENOENT; + + high = dir_info->count - 1; + while (low <= high) { + /* sum may overflow, but result will fit into mid again */ + mid = (unsigned long long)(low + high) / 2; + tmp_ino = dir_info->array[mid].ino; + if (ino == tmp_ino) { + *index = mid; + found = 1; + return 0; + } else if (ino < tmp_ino) { + /* + * The mid ino is larger than @ino, remember the index + * here so we won't miss this ino + */ + *index = mid; + found = 1; + if (mid == 0) + break; + high = mid - 1; + } else { + low = mid + 1; + } + } + + if (found) + return 0; + + return -ENOENT; +} + +/* + * + * Insert an inode into the sorted array. The array should have at least one + * free slot. + * + * Normally, add_dir_info is called with each inode in + * sequential order; but once in a while (like when pass 3 + * needs to recreate the root directory or lost+found + * directory) it is called out of order. In those cases, we + * need to move the dir_info entries down to make room, since + * the dir_info array needs to be sorted by inode number for + * get_dir_info()'s sake. + */ +static void e2fsck_insert_dir_info(struct dir_info_db *dir_info, ext2_ino_t ino, ext2_ino_t parent) +{ + ext2_ino_t index; + struct dir_info *dir; + size_t dir_size = sizeof(*dir); + struct dir_info *array = dir_info->array; + ext2_ino_t array_count = dir_info->count; + int err; + + /* + * Removing this check won't break anything. But since seqential ino + * inserting happens a lot, this check avoids binary search. + */ + if (array_count == 0 || array[array_count - 1].ino < ino) { + dir = &array[array_count]; + dir_info->count++; + goto out; + } + + err = e2fsck_dir_info_min_larger_equal(dir_info, ino, &index); + if (err >= 0 && array[index].ino == ino) { + dir = &array[index]; + goto out; + } + if (err < 0) { + dir = &array[array_count]; + dir_info->count++; + goto out; + } + + dir = &array[index]; + memmove((char *)dir + dir_size, dir, dir_size * (array_count - index)); + dir_info->count++; +out: + dir->ino = ino; + dir->dotdot = parent; + dir->parent = parent; +} + /* * This subroutine is called during pass1 to create a directory info * entry. During pass1, the passed-in parent is 0; it will get filled @@ -171,30 +270,7 @@ void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent) } #endif - /* - * Normally, add_dir_info is called with each inode in - * sequential order; but once in a while (like when pass 3 - * needs to recreate the root directory or lost+found - * directory) it is called out of order. In those cases, we - * need to move the dir_info entries down to make room, since - * the dir_info array needs to be sorted by inode number for - * get_dir_info()'s sake. - */ - if (ctx->dir_info->count && - ctx->dir_info->array[ctx->dir_info->count-1].ino >= ino) { - for (i = ctx->dir_info->count-1; i > 0; i--) - if (ctx->dir_info->array[i-1].ino < ino) - break; - dir = &ctx->dir_info->array[i]; - if (dir->ino != ino) - for (j = ctx->dir_info->count++; j > i; j--) - ctx->dir_info->array[j] = ctx->dir_info->array[j-1]; - } else - dir = &ctx->dir_info->array[ctx->dir_info->count++]; - - dir->ino = ino; - dir->dotdot = parent; - dir->parent = parent; + e2fsck_insert_dir_info(ctx->dir_info, ino, parent); } /* @@ -204,7 +280,8 @@ void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent) static struct dir_info *e2fsck_get_dir_info(e2fsck_t ctx, ext2_ino_t ino) { struct dir_info_db *db = ctx->dir_info; - ext2_ino_t low, high, mid; + ext2_ino_t index; + int err; if (!db) return 0; @@ -245,44 +322,19 @@ static struct dir_info *e2fsck_get_dir_info(e2fsck_t ctx, ext2_ino_t ino) if (db->last_lookup && db->last_lookup->ino == ino) return db->last_lookup; - low = 0; - high = ctx->dir_info->count - 1; - if (ino == ctx->dir_info->array[low].ino) { + err = e2fsck_dir_info_min_larger_equal(ctx->dir_info, ino, &index); + if (err < 0) + return NULL; + assert(ino <= ctx->dir_info->array[index].ino); + if (ino == ctx->dir_info->array[index].ino) { #ifdef DIRINFO_DEBUG - printf("(%u,%u,%u)\n", ino, - ctx->dir_info->array[low].dotdot, - ctx->dir_info->array[low].parent); + printf("(%d,%d,%d)\n", ino, + ctx->dir_info->array[index].dotdot, + ctx->dir_info->array[index].parent); #endif - return &ctx->dir_info->array[low]; + return &ctx->dir_info->array[index]; } - if (ino == ctx->dir_info->array[high].ino) { -#ifdef DIRINFO_DEBUG - printf("(%u,%u,%u)\n", ino, - ctx->dir_info->array[high].dotdot, - ctx->dir_info->array[high].parent); -#endif - return &ctx->dir_info->array[high]; - } - - while (low < high) { - /* sum may overflow, but result will fit into mid again */ - mid = (unsigned long long)(low + high) / 2; - if (mid == low || mid == high) - break; - if (ino == ctx->dir_info->array[mid].ino) { -#ifdef DIRINFO_DEBUG - printf("(%u,%u,%u)\n", ino, - ctx->dir_info->array[mid].dotdot, - ctx->dir_info->array[mid].parent); -#endif - return &ctx->dir_info->array[mid]; - } - if (ino < ctx->dir_info->array[mid].ino) - high = mid; - else - low = mid; - } - return 0; + return NULL; } static void e2fsck_put_dir_info(e2fsck_t ctx EXT2FS_NO_TDB_UNUSED, From patchwork Wed Nov 18 15:39:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402324 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=B6Pden76; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8X4Nswz9sPB for ; Thu, 19 Nov 2020 02:41:00 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726624AbgKRPlA (ORCPT ); Wed, 18 Nov 2020 10:41:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725814AbgKRPk7 (ORCPT ); Wed, 18 Nov 2020 10:40:59 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4ED31C0613D4 for ; Wed, 18 Nov 2020 07:40:58 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id w8so2921195ybj.14 for ; Wed, 18 Nov 2020 07:40:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=yNDkWH0BXwLvDjF9+mJLlcGy9OjMar14FW9yoJaQjtI=; b=B6Pden76bt2NDqo96OwH2gvB62m6ELLNXRiMHhv77tGDBx/1/pTjBvVugYTSB3o/WX OHFjoKPkCwwZBuB02xDsJJp8BSG36Mc9y5ejaagEiPVthnr5uWXC8wRn0bDRZn6UmIXL J+rk4xWQa5KSfuwuKJQeZjmpCjb/Fz//aryJoG5FjmN7xFlhcOj076TlRdxZlTGPu7Ne FLVuC95oJQV3fuFkehRiQvQPXnYF1yd818iwBMmfgU7pcRnQgyjJ30t95FNxeTIbYl5I EZ8PmYdYhqTTEvpAQ13XPpDYj27Pyw9OnPBVeyP2cWOMQS1gdrQexVtKT5LGiDT6Xc1U 74SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yNDkWH0BXwLvDjF9+mJLlcGy9OjMar14FW9yoJaQjtI=; b=n0UDxts7Y25Yg2wyv3Vrmc6qdKKFpQr5aNQj+OsTzmJVrr458Y3IN/F52+7bmvA3Ty PPW4o8m54NRZ3j7zpo05AFX0S5WaQdVONJj2xWaR7yYj0/QKOiDQySXmCB683t0av15Y SoniIUhP1GuzRPtXkmesT5CHW2TTftw2UDCFhg7/Nf8TYd0I3oOMgu4tdIKdz6Boiw5E GZAIOm2M6J/ku+1shOlFxOQE2OVzZagE8GU7q4eU0IcqMQ7HR2bXVxyxdsFIFZdgyTSd nzw9+gn10TKaOvn7z4XIJC4efnqsHIX70ILbQxoFD/s80f7KhStUdnXwhTeLCXU6JO/4 sKOg== X-Gm-Message-State: AOAM531OPUYfgofJdBrusnyScCx/aiJQRzXW82e59hUt7HbnhpirJDTm kuebcIol1AxZ0Tlt7FJMgALazPDO+A7+GWF1l1zKVPGSShmb81adbCIHawDnSdfdpTAdutRwWX+ uxk7DCp0VOVj583wm1Nzdhjssz/BDdPzYxjntBaovcLZzUPHPlpRWfyog2N12/vCQrWAtFUMML5 ns0ZWB08Q= X-Google-Smtp-Source: ABdhPJyipPkd5wVNfZXUDE4hkO9509ktPqqaHuo7GZeAN18Vtnc60rEp1pTLaEGTsAgw2E7ElZkJsOF0mws98vjKvLw= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:b281:: with SMTP id k1mr7780646ybj.303.1605714057409; Wed, 18 Nov 2020 07:40:57 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:03 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-18-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 17/61] e2fsck: merge dir_info after thread finishes From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi dir_info need be merged after thread finish. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/dirinfo.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++ e2fsck/e2fsck.h | 2 ++ e2fsck/pass1.c | 58 +++++++++++++++++++++++++++++++++++++++--- 3 files changed, 123 insertions(+), 3 deletions(-) diff --git a/e2fsck/dirinfo.c b/e2fsck/dirinfo.c index 28baaca2..b4adaa6e 100644 --- a/e2fsck/dirinfo.c +++ b/e2fsck/dirinfo.c @@ -169,6 +169,72 @@ e2fsck_dir_info_min_larger_equal(struct dir_info_db *dir_info, return -ENOENT; } +/* + * Merge two sorted dir info to @dest + */ +void e2fsck_merge_dir_info(e2fsck_t ctx, struct dir_info_db *src, + struct dir_info_db *dest) +{ + size_t size_dir_info = sizeof(struct dir_info); + ext2_ino_t size = dest->size; + struct dir_info *src_array = src->array; + struct dir_info *dest_array = dest->array; + ext2_ino_t src_count = src->count; + ext2_ino_t dest_count = dest->count; + ext2_ino_t total_count = src_count + dest_count; + struct dir_info *tmp_array; + struct dir_info *array_ptr; + ext2_ino_t src_index = 0; + ext2_ino_t dest_index = 0; + + if (src->count == 0) + return; + + if (size < total_count) + size = total_count; + + if (size < src->size) + size = src->size; + + tmp_array = e2fsck_allocate_memory(ctx, size * size_dir_info, + "directory map"); + array_ptr = tmp_array; + /* + * This can be improved by binary search and memcpy, but codes + * would be more complex. And if the groups distributed to each + * thread are strided, this implementation won't be too bad + * comparing to the optimiztion. + */ + while (src_index < src_count || dest_index < dest_count) { + if (src_index >= src_count) { + memcpy(array_ptr, &dest_array[dest_index], + (dest_count - dest_index) * size_dir_info); + break; + } + if (dest_index >= dest_count) { + memcpy(array_ptr, &src_array[src_index], + (src_count - src_index) * size_dir_info); + break; + } + if (src_array[src_index].ino < dest_array[dest_index].ino) { + *array_ptr = src_array[src_index]; + src_index++; + } else { + assert(src_array[src_index].ino > + dest_array[dest_index].ino); + *array_ptr = dest_array[dest_index]; + dest_index++; + } + array_ptr++; + } + + if (dest->array) + ext2fs_free_mem(&dest->array); + dest->array = tmp_array; + dest->size = size; + dest->count = total_count; +} + /* * * Insert an inode into the sorted array. The array should have at least one diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 06893f67..6783ed05 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -507,6 +507,8 @@ extern void read_bad_blocks_file(e2fsck_t ctx, const char *bad_blocks_file, /* dirinfo.c */ extern void e2fsck_add_dir_info(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t parent); +void e2fsck_merge_dir_info(e2fsck_t ctx, struct dir_info_db *src, + struct dir_info_db *dest); extern void e2fsck_free_dir_info(e2fsck_t ctx); extern int e2fsck_get_num_dirinfo(e2fsck_t ctx); extern struct dir_info_iter *e2fsck_dir_info_iter_begin(e2fsck_t ctx); diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index d4a2e707..ef6b2d13 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2363,6 +2363,21 @@ out_context: return retval; } +static void e2fsck_pass1_merge_dir_info(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + if (thread_ctx->dir_info == NULL) + return; + + if (global_ctx->dir_info == NULL) { + global_ctx->dir_info = thread_ctx->dir_info; + thread_ctx->dir_info = NULL; + return; + } + + e2fsck_merge_dir_info(global_ctx, thread_ctx->dir_info, + global_ctx->dir_info); +} + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2372,6 +2387,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx FILE *global_logf = global_ctx->logf; FILE *global_problem_logf = global_ctx->problem_logf; ext2fs_inode_bitmap inode_bad_map = global_ctx->inode_bad_map; + struct dir_info_db *dir_info = global_ctx->dir_info; ext2fs_inode_bitmap inode_used_map = global_ctx->inode_used_map; ext2fs_inode_bitmap inode_dir_map = global_ctx->inode_dir_map; ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; @@ -2404,6 +2420,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->block_dup_map = block_dup_map; global_ctx->block_ea_map = block_ea_map; global_ctx->block_metadata_map = block_metadata_map; + global_ctx->dir_info = dir_info; + e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); /* Keep the global singal flags*/ global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | @@ -2503,6 +2521,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) e2fsck_pass1_free_bitmap(&thread_ctx->block_dup_map); e2fsck_pass1_free_bitmap(&thread_ctx->block_ea_map); e2fsck_pass1_free_bitmap(&thread_ctx->block_metadata_map); + e2fsck_free_dir_info(thread_ctx); ext2fs_free_mem(&thread_ctx); return retval; @@ -2667,15 +2686,48 @@ out_abort: } #endif +/* TODO: tdb needs to be handled properly for multiple threads*/ +static int multiple_threads_supported(e2fsck_t ctx) +{ +#ifdef CONFIG_TDB + unsigned int threshold; + ext2_ino_t num_dirs; + errcode_t retval; + char *tdb_dir; + int enable; + + profile_get_string(ctx->profile, "scratch_files", "directory", 0, 0, + &tdb_dir); + profile_get_uint(ctx->profile, "scratch_files", + "numdirs_threshold", 0, 0, &threshold); + profile_get_boolean(ctx->profile, "scratch_files", + "icount", 0, 1, &enable); + + retval = ext2fs_get_num_dirs(ctx->fs, &num_dirs); + if (retval) + num_dirs = 1024; /* Guess */ + + /* tdb is unsupported now */ + if (enable && tdb_dir && !access(tdb_dir, W_OK) && + (!threshold || num_dirs > threshold)) + return 0; + #endif + return 1; +} + void e2fsck_pass1(e2fsck_t ctx) { init_ext2_max_sizes(); #ifdef CONFIG_PFSCK - e2fsck_pass1_multithread(ctx); -#else - e2fsck_pass1_run(ctx); + if (multiple_threads_supported(ctx)) { + e2fsck_pass1_multithread(ctx); + return; + } + fprintf(stderr, "Fall through single thread for pass1 " + "because tdb could not handle properly\n"); #endif + e2fsck_pass1_run(ctx); } #undef FINISH_INODE_LOOP From patchwork Wed Nov 18 15:39:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402325 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=AX0qQQAr; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8Y1Wvcz9sTc for ; Thu, 19 Nov 2020 02:41:01 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726651AbgKRPlA (ORCPT ); Wed, 18 Nov 2020 10:41:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725772AbgKRPlA (ORCPT ); Wed, 18 Nov 2020 10:41:00 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12D8AC0613D4 for ; Wed, 18 Nov 2020 07:41:00 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id cm17so1327486pjb.2 for ; Wed, 18 Nov 2020 07:41:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=1EIZ1iclevf4dCNvp11Qm9wMpxo/rUpjJpT9+13i1Kc=; b=AX0qQQArai5teoJe/vPif3gogB/7nYFxg3ImYAGEgpVO2G52NhP4QHUEdFpnk5ZgEG vS3P98sPG0XArNyPoLM9YbreudWh2ho//D5cujWwJVb2ABWybVSpzTLLDfm+EcDqrHAe BH/z6QygaPu1fwPZKr+8v5++3Ga9VhBir1f3HQh276+UmNrwn8pzDGbxgrnkWY25/VXv kiZcHnOiJEJkYdAjCK92yB7wt4vb7Www9BlxlpueDmVEyPZY5y9A6iTO2VuNyz9FB55I ZpDL7BvIIviHN2Q5hh1NMXbej+VArKB6jLWqkyyBcQNQ4xXvmqs1wKAt4nj1ILPm2wGJ lv1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1EIZ1iclevf4dCNvp11Qm9wMpxo/rUpjJpT9+13i1Kc=; b=ccmn7B4+mGuIMNq0U9qETwpGRE2pPYxHogipe33IAbCxzW5vscvB7tf3Qw89fya2hP 9cbEFN4SEHNnXWXz92UTz6QceJLD89NSMmY09ghVzNIy7jP5bzHbCv5w+FsJl7TUYY3k e/q897EBQ/sH6yth3ufHCcYPYavKtiT9jJEJt3ed5JSARekNlWmx6+LbmB2CNQ/9gLjN l6rBQ0J1Z+WXaJJrZjlOq+TrX6iLGyPx4b6Xoetu7SCcrGL2mJ0kpBscRLh2d0T+PuiU UCWuffKrOfykkN4XIwHAwqjXnmDm0uSTwWyCDLTTQ9OwOHZLHrfMBwREGQSDmdte5R3H U2mA== X-Gm-Message-State: AOAM532XkTnG/upSBTKJjjJP3Dt9qe7BG6iZvIO2n/b/mUIluXcTpUaw Vvdc7NDCmvmxlXEDIMpJJP/whqbl+SFpppV5l/TJeaInIaVnLIVo0ILDQhk4Nnqjbi7ibcuw17b pjQ1NqC1xnKHXH2nKy2CfebKNB6xbDuHvDcRHV99R0AspaADG7N779CWQE7TY/CEvwMhOegd/zK A4E272Q0Y= X-Google-Smtp-Source: ABdhPJzXixA9EHZtkoCxFcDRIHZMWL7u2wtfcJdzRolHpPHKiKcnCt2w2DYrS6Vuy61JnEkadmDOy/XJQN0WV9q2plY= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:902:c20a:b029:d6:b2d6:8006 with SMTP id 10-20020a170902c20ab02900d6b2d68006mr5054069pll.31.1605714059448; Wed, 18 Nov 2020 07:40:59 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:04 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-19-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 18/61] e2fsck: rbtree bitmap for dir From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Only rbtree support merge operation now, use it for bitmaps. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 1 + 1 file changed, 1 insertion(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index ef6b2d13..2147f64b 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1235,6 +1235,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) } pctx.errcode = e2fsck_allocate_inode_bitmap(fs, _("directory inode map"), + ctx->global_ctx ? EXT2FS_BMAP64_RBTREE : EXT2FS_BMAP64_AUTODIR, "inode_dir_map", &ctx->inode_dir_map); if (pctx.errcode) { From patchwork Wed Nov 18 15:39:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402326 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=IYGLMcdr; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8Z6Hytz9sPB for ; Thu, 19 Nov 2020 02:41:02 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727387AbgKRPlC (ORCPT ); Wed, 18 Nov 2020 10:41:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726663AbgKRPlC (ORCPT ); Wed, 18 Nov 2020 10:41:02 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 150CAC0613D4 for ; Wed, 18 Nov 2020 07:41:02 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id w8so2988243ybq.4 for ; Wed, 18 Nov 2020 07:41:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=10bpX5rRWogqd3Hep7qofcUq6bra1TcXdYCBBOqTGFA=; b=IYGLMcdr82B909aHQUgP1DikCdLzITSkL1Ta6hMij8+TFM5g0Dbbd1ugPKIZiAaccI zeHws99Y5lxxxwIDECcvlNkqPHotRrvduA2RUoBV5jaoKTbfJ/mQFb7jQrzudZSIBnwY gjQBatP66OTVZhjMSqpYiaaxdmnTeM6odGljZkmC/zXFn+Ivacr0dkSW99HYO0AWzEMa 1ukDvmrtCAXpMkw+Az9MFU9TtmXdLN3hhrPsHhok5ugVEinjW9xk/VELPInGpvw9gdkb HKCSJGG4T1JeppZXUXJNepHZcirwF2/5qfvBG8YAwgB9Vk7D2vGV93IvNgNDHajCzoB/ DHxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=10bpX5rRWogqd3Hep7qofcUq6bra1TcXdYCBBOqTGFA=; b=JwhLPZped+rhPRGAJG9CsWLR96IZPd+MMd2Ws9rUwLKdW0KShj0MVrli/4mU84F992 areAosXy9lKhMJXuYtHmG9A58EtjqcrYPtmfSDgApOKS0LoCd6Sxrv58P9aHtXL7oHMA AI3t1s8jx1RREb8MBK0U2bntGWeJbmuAvm52KCDauGx1OsQVQZYVX7FTmVdMMHnJjAF+ qQOP2TO72F2nKWyvJkH2QPAU0lhP42TNFBvsp/U6ux+RfPMKnWjfRSfhoHfJqR9efxea NYTz6gigS540vknBcUxqgUrkkmCdSNVUYdstTZGK7fnvE4WJ8ODXDv+eE2ZZX858Dmn5 ttlw== X-Gm-Message-State: AOAM532QradKBv1LHdLJjccynqDy4BB+qdnVZgUqKeGzNjI2tO65DxEe +Bfjnh+Mb10gC9i6KgcQmD078cW6jQNh34V5v8B1uLTxR62EU5g/pSQMpmNmRI15ZYRdFkSdJbz Y96e0hNfuQYpScwFQkSB1IOFqy7Dfpy8uR/wBSFXY+iOrR7xH3QdVc+NzrYZCbgPwDfQD/bgAx6 SvAdU1ZSg= X-Google-Smtp-Source: ABdhPJzR0WcpK43bjdNWjQ/wBO0d14w931pa5w+K+/T0T5pdvPhPLRIp9m6nr/7Wo1VrFCbY9PpeY0+HCdNGAiqFa4c= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:c782:: with SMTP id w124mr8654009ybe.267.1605714061246; Wed, 18 Nov 2020 07:41:01 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:05 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-20-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 19/61] e2fsck: merge badblocks after thread finishes From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Li Xi , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Badblocks should be merged properly after threads finish. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 19 +++++++--- lib/ext2fs/badblocks.c | 85 ++++++++++++++++++++++++++++++++++++++---- lib/ext2fs/ext2fs.h | 2 + lib/ext2fs/ext2fsP.h | 1 - 4 files changed, 94 insertions(+), 13 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 2147f64b..da42323d 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2180,6 +2180,7 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, e2fsck_t src_context, memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->inode_map = NULL; dest->block_map = NULL; + dest->badblocks = NULL; if (dest->dblist) dest->dblist->fs = dest; if (src->block_map) { @@ -2196,7 +2197,8 @@ static errcode_t e2fsck_pass1_copy_fs(ext2_filsys dest, e2fsck_t src_context, } if (src->badblocks) { - retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); + retval = ext2fs_badblocks_copy(src->badblocks, + &dest->badblocks); if (retval) return retval; } @@ -2241,11 +2243,13 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) io_channel dest_image_io; ext2fs_inode_bitmap inode_map; ext2fs_block_bitmap block_map; + ext2_badblocks_list badblocks; dest_io = dest->io; dest_image_io = dest->image_io; inode_map = dest->inode_map; block_map = dest->block_map; + badblocks = dest->badblocks; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->io = dest_io; @@ -2253,6 +2257,7 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) dest->icache = icache; dest->inode_map = inode_map; dest->block_map = block_map; + dest->badblocks = badblocks; if (dest->dblist) dest->dblist->fs = dest; @@ -2272,10 +2277,12 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) goto out; if (src->badblocks) { - retval = ext2fs_badblocks_copy(src->badblocks, &dest->badblocks); - - ext2fs_badblocks_list_free(src->badblocks); - src->badblocks = NULL; + if (dest->badblocks == NULL) + retval = ext2fs_badblocks_copy(src->badblocks, + &dest->badblocks); + else + retval = ext2fs_badblocks_merge(src->badblocks, + dest->badblocks); } out: io_channel_close(src->io); @@ -2283,6 +2290,8 @@ out: ext2fs_free_generic_bmap(src->inode_map); if (src->block_map) ext2fs_free_generic_bmap(src->block_map); + if (src->badblocks) + ext2fs_badblocks_list_free(src->badblocks); return retval; } diff --git a/lib/ext2fs/badblocks.c b/lib/ext2fs/badblocks.c index 0f23983b..3c9a608b 100644 --- a/lib/ext2fs/badblocks.c +++ b/lib/ext2fs/badblocks.c @@ -11,6 +11,7 @@ #include "config.h" #include +#include #include #if HAVE_UNISTD_H #include @@ -56,6 +57,75 @@ static errcode_t make_u32_list(int size, int num, __u32 *list, return 0; } +static inline int insert_ok(blk_t *array, int cnt, blk_t new) +{ + return (cnt == 0 || array[cnt - 1] != new); +} + +/* + * Merge list from src to dest + */ +static errcode_t merge_u32_list(ext2_u32_list src, ext2_u32_list dest) +{ + errcode_t retval; + int src_count = src->num; + int dest_count = dest->num; + int size = src_count + dest_count; + int size_entry = sizeof(blk_t); + blk_t *array; + blk_t *src_array = src->list; + blk_t *dest_array = dest->list; + int src_index = 0; + int dest_index = 0; + int uniq_cnt = 0; + + if (src->num == 0) + return 0; + + retval = ext2fs_get_array(size, size_entry, &array); + if (retval) + return retval; + + /* + * It is possible that src list and dest list could be + * duplicated when merging badblocks. + */ + while (src_index < src_count || dest_index < dest_count) { + if (src_index >= src_count) { + for (; dest_index < dest_count; dest_index++) + if (insert_ok(array, uniq_cnt, dest_array[dest_index])) + array[uniq_cnt++] = dest_array[dest_index]; + break; + } + if (dest_index >= dest_count) { + for (; src_index < src_count; src_index++) + if (insert_ok(array, uniq_cnt, src_array[src_index])) + array[uniq_cnt++] = src_array[src_index]; + break; + } + if (src_array[src_index] < dest_array[dest_index]) { + if (insert_ok(array, uniq_cnt, src_array[src_index])) + array[uniq_cnt++] = src_array[src_index]; + src_index++; + } else if (src_array[src_index] > dest_array[dest_index]) { + if (insert_ok(array, uniq_cnt, dest_array[dest_index])) + array[uniq_cnt++] = dest_array[dest_index]; + dest_index++; + } else { + if (insert_ok(array, uniq_cnt, dest_array[dest_index])) + array[uniq_cnt++] = dest_array[dest_index]; + src_index++; + dest_index++; + } + } + + ext2fs_free_mem(&dest->list); + dest->list = array; + dest->num = uniq_cnt; + dest->size = size; + return 0; +} + /* * This procedure creates an empty u32 list. @@ -79,13 +149,7 @@ errcode_t ext2fs_badblocks_list_create(ext2_badblocks_list *ret, int size) */ errcode_t ext2fs_u32_copy(ext2_u32_list src, ext2_u32_list *dest) { - errcode_t retval; - - retval = make_u32_list(src->size, src->num, src->list, dest); - if (retval) - return retval; - (*dest)->badblocks_flags = src->badblocks_flags; - return 0; + return make_u32_list(src->size, src->num, src->list, dest); } errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src, @@ -95,6 +159,13 @@ errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src, (ext2_u32_list *) dest); } +errcode_t ext2fs_badblocks_merge(ext2_badblocks_list src, + ext2_badblocks_list dest) +{ + return merge_u32_list((ext2_u32_list) src, + (ext2_u32_list) dest); +} + /* * This procedure frees a badblocks list. * diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 0aa1d94e..ac548311 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -814,6 +814,8 @@ extern int ext2fs_badblocks_list_iterate(ext2_badblocks_iterate iter, extern void ext2fs_badblocks_list_iterate_end(ext2_badblocks_iterate iter); extern errcode_t ext2fs_badblocks_copy(ext2_badblocks_list src, ext2_badblocks_list *dest); +extern errcode_t ext2fs_badblocks_merge(ext2_badblocks_list src, + ext2_badblocks_list dest); extern int ext2fs_badblocks_equal(ext2_badblocks_list bb1, ext2_badblocks_list bb2); extern int ext2fs_u32_list_count(ext2_u32_list bb); diff --git a/lib/ext2fs/ext2fsP.h b/lib/ext2fs/ext2fsP.h index ad8b7d52..02df759a 100644 --- a/lib/ext2fs/ext2fsP.h +++ b/lib/ext2fs/ext2fsP.h @@ -34,7 +34,6 @@ struct ext2_struct_u32_list { int num; int size; __u32 *list; - int badblocks_flags; }; struct ext2_struct_u32_iterate { From patchwork Wed Nov 18 15:39:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402327 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=TWWNVsMu; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn8c4gkLz9sPB for ; Thu, 19 Nov 2020 02:41:04 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727496AbgKRPlE (ORCPT ); Wed, 18 Nov 2020 10:41:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726663AbgKRPlD (ORCPT ); Wed, 18 Nov 2020 10:41:03 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4BF2C0613D4 for ; Wed, 18 Nov 2020 07:41:03 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id v10so1398401pjg.5 for ; Wed, 18 Nov 2020 07:41:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=bMbtktrsgt2BKEt9MFrYvx/sw2jOEdjl21VAcL40JPE=; b=TWWNVsMu73wQwMedlAkK59gJe4RWuHBF95OJrD4Xw9HKEwxyjAiEVqXFJv63vh6Ogh 2OvZGoGBS9tfh66PP+3HOMvAF7unL6DQhz3h2Th1LCC5Br4XPHq20RJved4oJ+dfA6c0 PXr2TeCXns34pTzMRHEoT9MAgxuUu6ch+a/g2N/eKA6NZayOD5B+/g4iqYWLIshh+YTk AbSsqtTIRRhmHhsWBWO8A136a40Ngeb1WaHpNJ719gv/slAYRS/8LavvvER0UuCMO17O sgLr78a/HM8G6SM+PgwWSo0/fnlKKPLaK96x1VL4KUFrcPvVBe4mcTQvvgkMW+cCQrEv MTAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=bMbtktrsgt2BKEt9MFrYvx/sw2jOEdjl21VAcL40JPE=; b=gR4t89V20q4ImL/ExSpaExZ1rHfk4ndII3AfStEOUMP8/k//Bx89MT6NQyb+2X8ZHW LmrrRehhbhaz/viEUbmmOTZEUtBC/XDp4XC9o9zbHJUUhB6b7WrfDtEuPu8g5PszcMrS fSv6uR1FdDo7O6WILNYuiUD7C/4H4S8IHmtqSimtHCE01u3hv2dmXVTRtezaHzkOcEld 5dzsqm+XhxWS4INPRdud58T607WFmeOCUBNrB8F5oCiEU3jvUgBUcRMEUKNt5DWNLoaW gzjcbZfnu3PXzyeBikkwzc+32h1jic6gmn2NhlEVrGlqHV8tNYhpNUyiTrZxVcl+xAmE 7QpQ== X-Gm-Message-State: AOAM531ectjig5Gp6naWY9om0Bexs6ksHOBNjX03pC+RdFjIQhEnGiXE 2aM/1UDEeTcvAV46mI0K02jRh5SydH/R3JXcyHpGlpE9X/ti2foVEwBRC5EftSfZFzDiKHeWFWN HX4BXSmGJf1OTBb6g4/sd3603pNvldbjRPmGGuQDHH+oVO8HzAs6iEiAwriHSOoT4mhRo5ySikx mxnPls2Kg= X-Google-Smtp-Source: ABdhPJycJwOOV6dxxuR/pspcZBKIBnhPb/PtjLTMP94fhVyrgguq3Q1butzKj/l5NLsiIAn6j8Masv2B4Ve+2UYMWG8= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a62:2cc1:0:b029:18c:85f5:864b with SMTP id s184-20020a622cc10000b029018c85f5864bmr4818724pfs.29.1605714063126; Wed, 18 Nov 2020 07:41:03 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:06 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-21-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 20/61] e2fsck: merge icounts after thread finishes From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Merge inode_count and inode_link_info properly after threads finish. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 47 ++++++++++++++++++++ lib/ext2fs/ext2fs.h | 1 + lib/ext2fs/icount.c | 103 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 151 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index da42323d..cdecd7c2 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2388,6 +2388,41 @@ static void e2fsck_pass1_merge_dir_info(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->dir_info); } +static inline errcode_t +e2fsck_pass1_merge_icount(ext2_icount_t *dest_icount, + ext2_icount_t *src_icount) +{ + if (*src_icount) { + if (*dest_icount == NULL) { + *dest_icount = *src_icount; + *src_icount = NULL; + } else { + errcode_t ret; + + ret = ext2fs_icount_merge(*src_icount, + *dest_icount); + if (ret) + return ret; + } + } + + return 0; +} + +static errcode_t e2fsck_pass1_merge_icounts(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + errcode_t ret; + + ret = e2fsck_pass1_merge_icount(&global_ctx->inode_count, + &thread_ctx->inode_count); + if (ret) + return ret; + ret = e2fsck_pass1_merge_icount(&global_ctx->inode_link_info, + &thread_ctx->inode_link_info); + + return ret; +} + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2408,6 +2443,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; ext2fs_block_bitmap block_metadata_map = global_ctx->block_metadata_map; ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; + ext2_icount_t inode_count = global_ctx->inode_count; + ext2_icount_t inode_link_info = global_ctx->inode_link_info; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2432,6 +2469,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->block_metadata_map = block_metadata_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); + global_ctx->inode_count = inode_count; + global_ctx->inode_link_info = inode_link_info; /* Keep the global singal flags*/ global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | @@ -2447,6 +2486,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->logf = global_logf; global_ctx->problem_logf = global_problem_logf; global_ctx->global_ctx = NULL; + retval = e2fsck_pass1_merge_icounts(global_ctx, thread_ctx); + if (retval) { + com_err(global_ctx->program_name, 0, + _("while merging icounts\n")); + return retval; + } retval = e2fsck_pass1_merge_bitmap(global_fs, &thread_ctx->inode_used_map, @@ -2532,6 +2577,8 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) e2fsck_pass1_free_bitmap(&thread_ctx->block_ea_map); e2fsck_pass1_free_bitmap(&thread_ctx->block_metadata_map); e2fsck_free_dir_info(thread_ctx); + ext2fs_free_icount(thread_ctx->inode_count); + ext2fs_free_icount(thread_ctx->inode_link_info); ext2fs_free_mem(&thread_ctx); return retval; diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index ac548311..28662d44 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -1513,6 +1513,7 @@ extern errcode_t ext2fs_icount_decrement(ext2_icount_t icount, ext2_ino_t ino, __u16 *ret); extern errcode_t ext2fs_icount_store(ext2_icount_t icount, ext2_ino_t ino, __u16 count); +extern errcode_t ext2fs_icount_merge(ext2_icount_t src, ext2_icount_t dest); extern ext2_ino_t ext2fs_get_icount_size(ext2_icount_t icount); errcode_t ext2fs_icount_validate(ext2_icount_t icount, FILE *); diff --git a/lib/ext2fs/icount.c b/lib/ext2fs/icount.c index 888a90b2..766eccca 100644 --- a/lib/ext2fs/icount.c +++ b/lib/ext2fs/icount.c @@ -13,6 +13,7 @@ #if HAVE_UNISTD_H #include #endif +#include #include #include #include @@ -701,6 +702,108 @@ errcode_t ext2fs_icount_store(ext2_icount_t icount, ext2_ino_t ino, return 0; } +errcode_t ext2fs_icount_merge_full_map(ext2_icount_t src, ext2_icount_t dest) +{ + /* TODO: add the support for full map */ + return EOPNOTSUPP; +} + +errcode_t ext2fs_icount_merge_el(ext2_icount_t src, ext2_icount_t dest) +{ + int src_count = src->count; + int dest_count = dest->count; + int size = src_count + dest_count; + int size_entry = sizeof(struct ext2_icount_el); + struct ext2_icount_el *array; + struct ext2_icount_el *array_ptr; + struct ext2_icount_el *src_array = src->list; + struct ext2_icount_el *dest_array = dest->list; + int src_index = 0; + int dest_index = 0; + errcode_t retval; + + if (src_count == 0) + return 0; + + retval = ext2fs_get_array(size, size_entry, &array); + if (retval) + return retval; + + array_ptr = array; + /* + * This can be improved by binary search and memcpy, but codes + * would be more complex. And if number of bad blocks is small, + * the optimization won't improve performance a lot. + */ + while (src_index < src_count || dest_index < dest_count) { + if (src_index >= src_count) { + memcpy(array_ptr, &dest_array[dest_index], + (dest_count - dest_index) * size_entry); + break; + } + if (dest_index >= dest_count) { + memcpy(array_ptr, &src_array[src_index], + (src_count - src_index) * size_entry); + break; + } + if (src_array[src_index].ino < dest_array[dest_index].ino) { + *array_ptr = src_array[src_index]; + src_index++; + } else { + assert(src_array[src_index].ino > + dest_array[dest_index].ino); + *array_ptr = dest_array[dest_index]; + dest_index++; + } + array_ptr++; + } + + ext2fs_free_mem(&dest->list); + dest->list = array; + dest->count = src_count + dest_count; + dest->size = size; + dest->last_lookup = NULL; + return 0; +} + +errcode_t ext2fs_icount_merge(ext2_icount_t src, ext2_icount_t dest) +{ + errcode_t retval; + + if (src->fullmap && !dest->fullmap) + return EINVAL; + + if (!src->fullmap && dest->fullmap) + return EINVAL; + + if (src->multiple && !dest->multiple) + return EINVAL; + + if (!src->multiple && dest->multiple) + return EINVAL; + + if (src->fullmap) + return ext2fs_icount_merge_full_map(src, dest); + + retval = ext2fs_merge_bitmap(src->single, dest->single, NULL, + NULL); + if (retval) + return retval; + + if (src->multiple) { + retval = ext2fs_merge_bitmap(src->multiple, dest->multiple, + NULL, NULL); + if (retval) + return retval; + } + + retval = ext2fs_icount_merge_el(src, dest); + if (retval) + return retval; + + return 0; +} + ext2_ino_t ext2fs_get_icount_size(ext2_icount_t icount) { if (!icount || icount->magic != EXT2_ET_MAGIC_ICOUNT) From patchwork Wed Nov 18 15:39:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402328 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=pXUfNIDW; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn926FNYz9sPB for ; Thu, 19 Nov 2020 02:41:26 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727498AbgKRPlG (ORCPT ); Wed, 18 Nov 2020 10:41:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727437AbgKRPlG (ORCPT ); Wed, 18 Nov 2020 10:41:06 -0500 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2E9AC0613D4 for ; Wed, 18 Nov 2020 07:41:05 -0800 (PST) Received: by mail-qv1-xf49.google.com with SMTP id u19so1573318qvx.4 for ; Wed, 18 Nov 2020 07:41:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ADzWWIbV4bZJ8NdKcONNs35fekjgKDCN5MShyNoNJbQ=; b=pXUfNIDWimaahb1A/Khf6m26p1q6KND3vqTohWbSxwFyHQrMbeQjIV8+0j4AN7j++a sMoDs2twKXt/71Px6OH1ISPVjVZ+qaY+c2laIPzDymXrSWYi+68HL0+jVSAFFUXe2cDU ThXR3H/epxoK6MMj2uOB0IlEkYTKH2sPMT1NkJLwye/K8jCk3x0FWQhMeaqyxfXsxxYl qNWN57qUTBn5K65obREMd8Dx6rmmh29TdblKU3n/pWnAd8GADkUvAz+qXdF7hlrCvz9p i7MFPyYSsIPQD96ItsF/0mQepgji0I4Wx/QUC9qnqHEIAD96ILDGZsVUmDnRgjr68ntj GyhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ADzWWIbV4bZJ8NdKcONNs35fekjgKDCN5MShyNoNJbQ=; b=gxSVwja/8txwr4oG9jg25tsjo50E3GBN/2G9FiWmKaPcrxbBGqyzXR7B4uePQOo3al qBl44rfnJS22LCiWu1/dARgXhLxy9OCEcHQ4sn0mX0TQyjcrGKi8Kan2/yWUJ0tMgofd BL5CYP6DRKdK8YxxkX9yV/Kh7A9hfYiOW8OXZwGlzD24JvlFNpMfXbgYiq72iB06UEab XTUEO8NEicQhbA50crhQVAg0Rw1VdvrKOBRhrAqkfg/35NGwXN4QgdD2F0dpn/nrNxmg ayOORSAnaHL7ueSCqN2mlnH3ROOCynQQWEexZntCe3mzpiEgtY5k4n7ocys6LAuiNvLH n49Q== X-Gm-Message-State: AOAM530jwWhWX0Mj5Dr6MzGj55k378UUFJSUJjGuAjnSnNLlciG4OHDY tZHD3ad0fAswMji16hEv6csLLRdovGwlyzQBOJxQ57LCinoIQbEPeFfbkVHPAQFhYNd8Lc/iqsA UispgOsvnsduubctGjiyp7LMFoah+p7EcLTDPrxeIFYQ2ShW73wOrmXNDTLcPLQzXHVeBQ0G1vq N2rE//tuU= X-Google-Smtp-Source: ABdhPJxjffYdN+iwhyvG8Rri3sB8dLnwgkMpGfcO9NHK2pbyUOw7tvCX8o1cNrvMH7pJYG6eNw2K7V6xjVOZTAv9TTs= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:ad4:45eb:: with SMTP id q11mr5533170qvu.20.1605714064894; Wed, 18 Nov 2020 07:41:04 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:07 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-22-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 21/61] e2fsck: merge dblist after thread finishes From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi Merge dblist properly. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 19 +++++++++++++++++++ lib/ext2fs/dblist.c | 36 ++++++++++++++++++++++++++++++++++++ lib/ext2fs/ext2fs.h | 1 + 3 files changed, 56 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index cdecd7c2..75298d9d 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2244,12 +2244,14 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) ext2fs_inode_bitmap inode_map; ext2fs_block_bitmap block_map; ext2_badblocks_list badblocks; + ext2_dblist dblist; dest_io = dest->io; dest_image_io = dest->image_io; inode_map = dest->inode_map; block_map = dest->block_map; badblocks = dest->badblocks; + dblist = dest->dblist; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->io = dest_io; @@ -2258,6 +2260,7 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) dest->inode_map = inode_map; dest->block_map = block_map; dest->badblocks = badblocks; + dest->dblist = dblist; if (dest->dblist) dest->dblist->fs = dest; @@ -2276,6 +2279,19 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) if (retval) goto out; + if (src->dblist) { + if (dest->dblist) { + retval = ext2fs_merge_dblist(src->dblist, + dest->dblist); + if (retval) + goto out; + } else { + dest->dblist = src->dblist; + dest->dblist->fs = dest; + src->dblist = NULL; + } + } + if (src->badblocks) { if (dest->badblocks == NULL) retval = ext2fs_badblocks_copy(src->badblocks, @@ -2292,6 +2308,9 @@ out: ext2fs_free_generic_bmap(src->block_map); if (src->badblocks) ext2fs_badblocks_list_free(src->badblocks); + if (src->dblist) + ext2fs_free_dblist(src->dblist); + return retval; } diff --git a/lib/ext2fs/dblist.c b/lib/ext2fs/dblist.c index bbdb221d..1fdd8f43 100644 --- a/lib/ext2fs/dblist.c +++ b/lib/ext2fs/dblist.c @@ -119,6 +119,42 @@ errcode_t ext2fs_copy_dblist(ext2_dblist src, ext2_dblist *dest) return 0; } +/* + * Merge a directory block list @src to @dest + */ +errcode_t ext2fs_merge_dblist(ext2_dblist src, ext2_dblist dest) +{ + unsigned long long src_count = src->count; + unsigned long long dest_count = dest->count; + unsigned long long size = src_count + dest_count; + size_t size_entry = sizeof(struct ext2_db_entry2); + struct ext2_db_entry2 *array, *array2; + errcode_t retval; + + if (src_count == 0) + return 0; + + if (src->sorted || (dest->sorted && dest_count != 0)) + return EINVAL; + + retval = ext2fs_get_array(size, size_entry, &array); + if (retval) + return retval; + + array2 = array; + memcpy(array, src->list, src_count * size_entry); + array += src_count; + memcpy(array, dest->list, dest_count * size_entry); + ext2fs_free_mem(&dest->list); + + dest->list = array2; + dest->count = src_count + dest_count; + dest->size = size; + dest->sorted = 0; + + return 0; +} + /* * Close a directory block list * diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 28662d44..0fa0e22f 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -1122,6 +1122,7 @@ extern errcode_t ext2fs_add_dir_block(ext2_dblist dblist, ext2_ino_t ino, blk_t blk, int blockcnt); extern errcode_t ext2fs_add_dir_block2(ext2_dblist dblist, ext2_ino_t ino, blk64_t blk, e2_blkcnt_t blockcnt); +extern errcode_t ext2fs_merge_dblist(ext2_dblist src, ext2_dblist dest); extern void ext2fs_dblist_sort(ext2_dblist dblist, EXT2_QSORT_TYPE (*sortfunc)(const void *, const void *)); From patchwork Wed Nov 18 15:39:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402329 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=N2p0KN/k; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn935GmZz9sTc for ; Thu, 19 Nov 2020 02:41:27 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727502AbgKRPlI (ORCPT ); Wed, 18 Nov 2020 10:41:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727437AbgKRPlH (ORCPT ); Wed, 18 Nov 2020 10:41:07 -0500 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97DB9C0613D4 for ; Wed, 18 Nov 2020 07:41:07 -0800 (PST) Received: by mail-qv1-xf49.google.com with SMTP id 60so1550322qvb.15 for ; Wed, 18 Nov 2020 07:41:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=KtL1a+UcdzM6y5a8zXSch74a153jD7I4Dvtdi1s66D0=; b=N2p0KN/ke8gX9lVzxAKG3KkNQ2V0Lgm6QktGzolBZl95RLHYNnioRcNKzz9rktqclx bUXHRfIxEhcVQBOu8y+dZVAszlTslpVaNAcdxrpgOijYZ+UJP1YZ4AORHGestYTmtBWL aPtGUuJdGEX5DPjwc4fcSytF5DT3+537Lm1HsnTr9aWdgZ5rDfbmBOYXvn4YSiRhCOkg o/ccCUdSyzfmbAxzPAVy6LDVl2Ew/wZActpj1cTwyku9VjoCU+o+K2SYq8DR93ULqgFM OS9F50lIMxQ3xNZFtBeg83u5lZlCxPdW94T5T7Hec4AQL7BaQZnpssXCwO4IIaT7LFwK Ajsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=KtL1a+UcdzM6y5a8zXSch74a153jD7I4Dvtdi1s66D0=; b=sMTPX5tthNuVv6hL98FNLrN0XJSn4/c7fHF4P+9jsCVvmC9Q5ZBmG+VZnnuUKrRoQt Wxgwe5cZLukpjrCFkgDsuig8XW8pL3kUu/hc4SNvA0j6qDfGwjOL9BRbMXXKJKiI17w4 5k5tPOf+tmSuCkwsNwGaQHArqr2goelGFsNvuZnEgpABcNVBKNNd8+teOLlljK6JXnhe c2UUadpLuUAwfRVOil1d2uCKLoF4F6PvbGoJLWsCi1IRDMjw526oGuM8Nnt6csTe93OG O8mhHGF8ZnYmOMyAmayDclPEaP+cOBbODSt/ww3lbhSIhTvimeEdn5ALxt7l2MeZUJjH RWOw== X-Gm-Message-State: AOAM530V4fbIgPdlu17OWktF4un8LOHmnGogxfXrUCQX8ckfS/RktSEk 44+IBkWmA5s72rz5nqkAIYQ4pno/XpyvM0aDI5rGiwD91o5C/RbKizPD58lI2Efqn5O6+DbFYmi w6zYK6qpJpwVswLQ8I5XDy/+Mt+G8bOWryaj9jUX/MixSFqZCfIsqQLDV3Fv6eCjrxpaeX24fk0 TaD6iuwc8= X-Google-Smtp-Source: ABdhPJzk0Ih+xfqKif+8wEe0x+883imIsfQZ5rtI3hTr3CNnTCy4CrPRbz6VCqT5X/7ZbZBV/y43ZV1ZroI+0OX8juc= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:ad4:4633:: with SMTP id x19mr5590005qvv.11.1605714066752; Wed, 18 Nov 2020 07:41:06 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:08 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-23-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 22/61] e2fsck: add debug codes for multiple threads From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi These debug codes are added to run the multiple pass1 check thread one by one in order. If all the codes are correct, fsck of multiple threads should have exactly the same outcome with single thread. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 16 ++++++++++++++++ e2fsck/pass1.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 6783ed05..972c8410 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -450,6 +450,18 @@ struct e2fsck_struct { }; #ifdef CONFIG_PFSCK +#ifdef DEBUG_THREADS +/* + * Enabling DEBUG_THREADS would cause the parallel + * fsck threads run sequentially. + */ +struct e2fsck_thread_debug { + pthread_mutex_t etd_mutex; + pthread_cond_t etd_cond; + int etd_finished_threads; +}; +#endif + struct e2fsck_thread_info { /* ID returned by pthread_create() */ pthread_t eti_thread_id; @@ -459,7 +471,11 @@ struct e2fsck_thread_info { int eti_started; /* Context used for this thread */ e2fsck_t eti_thread_ctx; +#ifdef DEBUG_THREADS + struct e2fsck_thread_debug *eti_debug; +#endif }; + #endif /* Data structures to evaluate whether an extent tree needs rebuilding. */ diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 75298d9d..f36b3e70 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2641,6 +2641,18 @@ static void *e2fsck_pass1_thread(void *arg) { struct e2fsck_thread_info *info = arg; e2fsck_t thread_ctx = info->eti_thread_ctx; +#ifdef DEBUG_THREADS + struct e2fsck_thread_debug *thread_debug = info->eti_debug; +#endif + +#ifdef DEBUG_THREADS + pthread_mutex_lock(&thread_debug->etd_mutex); + while (info->eti_thread_index > thread_debug->etd_finished_threads) { + pthread_cond_wait(&thread_debug->etd_cond, + &thread_debug->etd_mutex); + } + pthread_mutex_unlock(&thread_debug->etd_mutex); +#endif #ifdef HAVE_SETJMP_H /* @@ -2665,6 +2677,14 @@ out: thread_ctx->thread_info.et_group_start, thread_ctx->thread_info.et_group_end, thread_ctx->thread_info.et_inode_number); + +#ifdef DEBUG_THREADS + pthread_mutex_lock(&thread_debug->etd_mutex); + thread_debug->etd_finished_threads++; + pthread_cond_broadcast(&thread_debug->etd_cond); + pthread_mutex_unlock(&thread_debug->etd_mutex); +#endif + return NULL; } @@ -2678,6 +2698,12 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, struct e2fsck_thread_info *tmp_pinfo; int i; e2fsck_t thread_ctx; +#ifdef DEBUG_THREADS + struct e2fsck_thread_debug thread_debug = + {PTHREAD_MUTEX_INITIALIZER, PTHREAD_COND_INITIALIZER, 0}; + + thread_debug.etd_finished_threads = 0; +#endif retval = pthread_attr_init(&attr); if (retval) { @@ -2698,6 +2724,9 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, for (i = 0; i < num_threads; i++) { tmp_pinfo = &infos[i]; tmp_pinfo->eti_thread_index = i; +#ifdef DEBUG_THREADS + tmp_pinfo->eti_debug = &thread_debug; +#endif retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, i, num_threads); if (retval) { From patchwork Wed Nov 18 15:39:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402330 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=e6LwVKmq; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn94119Yz9sPB for ; Thu, 19 Nov 2020 02:41:28 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727512AbgKRPlK (ORCPT ); Wed, 18 Nov 2020 10:41:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727437AbgKRPlJ (ORCPT ); Wed, 18 Nov 2020 10:41:09 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7ED3AC0613D4 for ; Wed, 18 Nov 2020 07:41:09 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id a6so2980610ybi.0 for ; Wed, 18 Nov 2020 07:41:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=BHD4ufBZiGRPhS9uyriqVz/N2Lvd/jndSOp/RLCo5Iw=; b=e6LwVKmqX025tcC83tKwsqfdtxt3RDgL4T7ZijVhv4FxXXilAhouESkd98qxu6FEY4 iaIJcTmCUVkCZzB2T3S9pKh0Lfg1ZzjF4C6VpPWIbqkFZj9zOYkHEh9G3MZ/7yOLErJh cXlW3vYHXAMJxRpVSnxuI23gX8FyD8vQg6Keqhb0OulK2EIhqHFH7vAxZqG6/b+8uq6V hGAZ9SAbqq9x5W6I2LohgzvIRKwujGpY1m6n3nNcrnO/VzAasvsLKD/Wur82B9g+kXFU pWYnfLtIunREn9WhSuLpIxlM9jqXzTsDPK5XwZMtJ70ppqsDmYyuVjFJpW8ILeIewtFL ZBjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BHD4ufBZiGRPhS9uyriqVz/N2Lvd/jndSOp/RLCo5Iw=; b=fKy+8QRCdSVVdztarUvRHc9padB0IIAnQ1xQQgpvQlbClhBaJAPEiZARp5M0eK2TIH GJ+yxVuq7SRZr2g03McCQcIXNUTFeQdzspEWatb5F/RAw4cH/gw1Pn9LpP9xoP8jGHk7 +2K4y6cph4Kuoof/88T6BQvaFTe7CjCLxRT5mBYSmaRuzE/a27usIyBsDIYmOyvILSjF lxXGD6PGCsw0UdiP5A0N+Ov6GDFiHZorRvPyY/sAx7LKllCA7NErjsNp9NBtJMU4kvSs 0y21yOWaE1LHuAwkBkq0BV8apDm820OdVefemNgYgCPJypXqkwUodUCJBuo97SvQLKgj RzYQ== X-Gm-Message-State: AOAM531q5wf2+5ViH69Kte3bP/9hG3YTTIV8gPU03ZjMe/EEVm+dLN18 hGd1t32dWwVeOgh7Z0mfMUCJQGRIGaoe0HkZZW8NTW7oYjx3qV61wVIjh6/3bQwLOmwLbF+Ptbv M3j6AIZU8IbMNASelj7x8BEMRm92+GEBawr1K5kAhW/sQxAGz8tF1ZUrNW/NdEPiw8vFWJbNejl 4XYV0bo4Q= X-Google-Smtp-Source: ABdhPJwrJThAwej1Tg0YqVnC577TNtiAaAuMR9GGA50r8dSE6f6d/YSyHVvvpuOidK/p4j2DdKtDZldCFMDjNmrs+Uc= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:c7c6:: with SMTP id w189mr6109606ybe.403.1605714068605; Wed, 18 Nov 2020 07:41:08 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:09 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-24-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 23/61] e2fsck: merge counts after threads finish From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Li Xi , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Merge counts properly. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index f36b3e70..af0ff724 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2464,6 +2464,23 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; ext2_icount_t inode_count = global_ctx->inode_count; ext2_icount_t inode_link_info = global_ctx->inode_link_info; + __u32 fs_directory_count = global_ctx->fs_directory_count; + __u32 fs_regular_count = global_ctx->fs_regular_count; + __u32 fs_blockdev_count = global_ctx->fs_blockdev_count; + __u32 fs_chardev_count = global_ctx->fs_chardev_count; + __u32 fs_links_count = global_ctx->fs_links_count; + __u32 fs_symlinks_count = global_ctx->fs_symlinks_count; + __u32 fs_fast_symlinks_count = global_ctx->fs_fast_symlinks_count; + __u32 fs_fifo_count = global_ctx->fs_fifo_count; + __u32 fs_total_count = global_ctx->fs_total_count; + __u32 fs_badblocks_count = global_ctx->fs_badblocks_count; + __u32 fs_sockets_count = global_ctx->fs_sockets_count; + __u32 fs_ind_count = global_ctx->fs_ind_count; + __u32 fs_dind_count = global_ctx->fs_dind_count; + __u32 fs_tind_count = global_ctx->fs_tind_count; + __u32 fs_fragmented = global_ctx->fs_fragmented; + __u32 fs_fragmented_dir = global_ctx->fs_fragmented_dir; + __u32 large_files = global_ctx->large_files; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2490,6 +2507,23 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); global_ctx->inode_count = inode_count; global_ctx->inode_link_info = inode_link_info; + global_ctx->fs_directory_count += fs_directory_count; + global_ctx->fs_regular_count += fs_regular_count; + global_ctx->fs_blockdev_count += fs_blockdev_count; + global_ctx->fs_chardev_count += fs_chardev_count; + global_ctx->fs_links_count += fs_links_count; + global_ctx->fs_symlinks_count += fs_symlinks_count; + global_ctx->fs_fast_symlinks_count += fs_fast_symlinks_count; + global_ctx->fs_fifo_count += fs_fifo_count; + global_ctx->fs_total_count += fs_total_count; + global_ctx->fs_badblocks_count += fs_badblocks_count; + global_ctx->fs_sockets_count += fs_sockets_count; + global_ctx->fs_ind_count += fs_ind_count; + global_ctx->fs_dind_count += fs_dind_count; + global_ctx->fs_tind_count += fs_tind_count; + global_ctx->fs_fragmented += fs_fragmented; + global_ctx->fs_fragmented_dir += fs_fragmented_dir; + global_ctx->large_files += large_files; /* Keep the global singal flags*/ global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | From patchwork Wed Nov 18 15:39:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402331 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=n+ZwreDc; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn943dtlz9sVC for ; Thu, 19 Nov 2020 02:41:28 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727513AbgKRPlL (ORCPT ); Wed, 18 Nov 2020 10:41:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727437AbgKRPlL (ORCPT ); Wed, 18 Nov 2020 10:41:11 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27EBFC0613D4 for ; Wed, 18 Nov 2020 07:41:11 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id z130so1501275pgz.19 for ; Wed, 18 Nov 2020 07:41:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5raDp+nOWma+/5kc4saNDNxM5weU8Iq0IzEpPV0rCbU=; b=n+ZwreDcDf1fEeDc7RNipvoTbH22hJ6ILsCqG75zc01N6R0LVo+2kbr+wshEgw+rTm 2LZKwE5lgoYKGgyIGw3JxyRCwqyZs42b/xew4e9odl4Bn8LZhH8tmacrMREMz9hYhbDi 9PrsKd2WznCb/0sCB+s9fgGz5uw7b+VOtJCTrOgQ95EYE4CudbrgzstvmA2z1G4vRdUN 4rSiD1VSgid0wMcK1FGr2lgdCKTuj+odoevd7reFGu5bCUHoeQ94raVV1Y5Ej7IHJt5Y OKd/NcLqtwhCXJdjvUNqVfBqOktqircZuzr/v6yuRrSflfg/H8fRRjKZgdTWAGq3zjox cH8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5raDp+nOWma+/5kc4saNDNxM5weU8Iq0IzEpPV0rCbU=; b=nPzXCjoWo3sAOtei+yGre0I5QBsR6jvWi9+1mQ1QSggi+X/YUt8SrRLbXxyd7YcBVy UZ56NquxdwYEyn/nPjJKtAn6U/riMMnJ2NQ5XFMWkMzMMy012/HB4lfnEtUTRwOINIM8 spiVOsrf3DHZYf9wFjF5kW6Tf/94X9k96HxSmQxWdvwahpQyQ1AhR9A5Nl4okDW6T1ul Q8xSi5dph/YBWzM7WrQ42DPS62xh25W2EytPmYBpozbse1926CkbMF5FLN9FaspeGKXT sr6HN91IiVK490tK8ryM1pInjhNC9ESnl4vtX+wsyQYCxlX0WkMCwR2hkAa3kCrdtGM+ +Siw== X-Gm-Message-State: AOAM5331Cz3lTSfNVpmiFBv+RVTTnhJrKsCgEfTyys/6AYzTBHijFUjA u4B56HN4oe2loE3GF8EVxZ2n2o+/l/65ECf6Ef5qwzH3Zu75UJNiK0+Rr766dOngZrXOfay+jeZ AVPlOXJt+2p4U1JDORLPSh5GeH9mLEFgRvFkZDkD951R8wbpnWzaorgfTjUSmFTOMZNYDLvGeXo 8M6xQPEWg= X-Google-Smtp-Source: ABdhPJxKXwfwxzbSPiUNHZyQN8+KGkmkA8oTVTFnmWJ8Ahw7mwTfjg+WU2OXb11H9dcz0NZan1eFdQ9S7wiHNnscn4k= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a63:e24:: with SMTP id d36mr8570498pgl.373.1605714070479; Wed, 18 Nov 2020 07:41:10 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:10 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-25-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 24/61] e2fsck: merge fs flags when threads finish From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Li Xi , Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Li Xi merge fs flags properly. Signed-off-by: Li Xi Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index af0ff724..783e14f0 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2245,6 +2245,7 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) ext2fs_block_bitmap block_map; ext2_badblocks_list badblocks; ext2_dblist dblist; + int flags; dest_io = dest->io; dest_image_io = dest->image_io; @@ -2252,6 +2253,7 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) block_map = dest->block_map; badblocks = dest->badblocks; dblist = dest->dblist; + flags = dest->flags; memcpy(dest, src, sizeof(struct struct_ext2_filsys)); dest->io = dest_io; @@ -2263,6 +2265,9 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) dest->dblist = dblist; if (dest->dblist) dest->dblist->fs = dest; + dest->flags = src->flags | flags; + if (!(src->flags & EXT2_FLAG_VALID) || !(flags & EXT2_FLAG_VALID)) + ext2fs_unmark_valid(dest); if (src->icache) { ext2fs_free_inode_cache(src->icache); From patchwork Wed Nov 18 15:39:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402332 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=CCbQAK18; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn946FV0z9sTc for ; Thu, 19 Nov 2020 02:41:28 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727519AbgKRPlP (ORCPT ); Wed, 18 Nov 2020 10:41:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727517AbgKRPlO (ORCPT ); Wed, 18 Nov 2020 10:41:14 -0500 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 389E5C0613D4 for ; Wed, 18 Nov 2020 07:41:13 -0800 (PST) Received: by mail-qk1-x749.google.com with SMTP id q25so1748481qkm.17 for ; Wed, 18 Nov 2020 07:41:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=LVEJhiemBDfRK4/0xdGtq4GMglkRsz7xq1OFu8Mg2Gg=; b=CCbQAK18qIB+sQgSHh7h2sIhSicHFU2l2lPavQFnWKvWRNM6GnW1zR4bxSGApdr4JI SOrZb7FOORcQZpz5RWdybc3vgyHdZ3sXi94kO8c7p5MAu5uUJOaHtHRSvPwzF6aXHzqg DK1UBMg0LXIypAYipyoZdgL8E8dbOXlJMVtTC0XyFE66qwe7R384bXSBlekTJ8azEpJX H3QIIeehdpIP0bT51P3fJneOy7vqWIYPPvfcZdYCNWndAsgTCzw1guydz9APegUfqAtU MRREQQEALLfak/rWUkF5D/LV39sxF2aS4Rw110yCneOlI1q7opRTMc9e6AhjLxd4PQkp DPRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LVEJhiemBDfRK4/0xdGtq4GMglkRsz7xq1OFu8Mg2Gg=; b=N7i3opmGtP93VXUH3ndVl9DfkfFoCaj5OVnvsk5hkPGpL1IUwGFYGT7kWEZhpS62oy VgE7NDSP5IpwCMTPpRaPqvw6Sg02bmrYbdNnwZhoE/fOAP+jFUBUCW/tINAC8NXW94Ct fZYqihMDxPyPh75As/rL1OeGZhgLFTrHSZIH5ipC21mGPvyS3+ceXttg0lvXSziR1ZgF SltJ9TlEuTfkPvb5YoOiMYCpkUQMuULcLWA8cpn+XNR1pH5Dbra6zRXtMEEaC6WZDX5u ybSi/p/K13QZBVcDVEnE4QbPMcQsxZPbrqtwpzsUY4Ydws2orUt4uCH5GQDvZYe+5Kmh mfoA== X-Gm-Message-State: AOAM531WZ7D8QHUw9qthXzlkMCzrYT2GG62upKppS6STE4/hon8cDpAf VqM5uUU2osfRkj7Zv/cyTieDaaIDUlu6WabhO6Kp3jWvWQF9HdATPxzwdol7oZDfvBA0b/e0hpU 4nAb0lzDmwDoLopC2eD4G9VcUB0FYFNVFUsEZeCMGdwbMYh0Gpj11ZpvBd8lIdBUkB0ICIX6J3m dDjlxsAw4= X-Google-Smtp-Source: ABdhPJwPcLz+gbYDMTTaYzvjBvF6DpuZns+LROSsZLP0NGCPRyeeSjJ56EIklA8TQr9UUQU96YZSC3dRyQVMR9SYTBQ= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:b181:: with SMTP id v1mr5593598qvd.36.1605714072316; Wed, 18 Nov 2020 07:41:12 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:11 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-26-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 25/61] e2fsck: merge dx_dir_info after threads finish From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Merge properly. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/dx_dirinfo.c | 64 +++++++++++++++++++++++++++++++++++++++++++++ e2fsck/e2fsck.h | 1 + e2fsck/pass1.c | 23 ++++++++++++++++ 3 files changed, 88 insertions(+) diff --git a/e2fsck/dx_dirinfo.c b/e2fsck/dx_dirinfo.c index caca3e30..91954572 100644 --- a/e2fsck/dx_dirinfo.c +++ b/e2fsck/dx_dirinfo.c @@ -5,6 +5,7 @@ * under the terms of the GNU Public License. */ +#include #include "config.h" #include "e2fsck.h" @@ -79,6 +80,69 @@ void e2fsck_add_dx_dir(e2fsck_t ctx, ext2_ino_t ino, struct ext2_inode *inode, "dx_block info array"); } +/* + * Merge two sorted dir info to @dest + */ +void e2fsck_merge_dx_dir(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + struct dx_dir_info *src_array = thread_ctx->dx_dir_info; + struct dx_dir_info *dest_array = global_ctx->dx_dir_info; + size_t size_dx_info = sizeof(struct dx_dir_info); + ext2_ino_t size = global_ctx->dx_dir_info_size; + ext2_ino_t src_count = thread_ctx->dx_dir_info_count; + ext2_ino_t dest_count = global_ctx->dx_dir_info_count; + ext2_ino_t total_count = src_count + dest_count; + struct dx_dir_info *array; + struct dx_dir_info *array_ptr; + ext2_ino_t src_index = 0, dest_index = 0; + + if (thread_ctx->dx_dir_info_count == 0) + return; + + if (size < total_count) + size = total_count; + + array = e2fsck_allocate_memory(global_ctx, size * size_dx_info, + "directory map"); + array_ptr = array; + /* + * This can be improved by binary search and memcpy, but codes + * would be more complex. And if the groups distributed to each + * thread are strided, this implementation won't be too bad + * comparing to the optimiztion. + */ + while (src_index < src_count || dest_index < dest_count) { + if (src_index >= src_count) { + memcpy(array_ptr, &dest_array[dest_index], + (dest_count - dest_index) * size_dx_info); + break; + } + if (dest_index >= dest_count) { + memcpy(array_ptr, &src_array[src_index], + (src_count - src_index) * size_dx_info); + break; + } + if (src_array[src_index].ino < dest_array[dest_index].ino) { + *array_ptr = src_array[src_index]; + src_index++; + } else { + assert(src_array[src_index].ino > + dest_array[dest_index].ino); + *array_ptr = dest_array[dest_index]; + dest_index++; + } + array_ptr++; + } + + if (global_ctx->dx_dir_info) + ext2fs_free_mem(&global_ctx->dx_dir_info); + if (thread_ctx->dx_dir_info) + ext2fs_free_mem(&thread_ctx->dx_dir_info); + global_ctx->dx_dir_info = array; + global_ctx->dx_dir_info_size = size; + global_ctx->dx_dir_info_count = total_count; +} + /* * get_dx_dir_info() --- given an inode number, try to find the directory * information entry for it. diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 972c8410..c3b0af34 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -539,6 +539,7 @@ extern int e2fsck_dir_info_get_parent(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t *parent); extern int e2fsck_dir_info_get_dotdot(e2fsck_t ctx, ext2_ino_t ino, ext2_ino_t *dotdot); +extern void e2fsck_merge_dx_dir(e2fsck_t global_ctx, e2fsck_t thread_ctx); /* dx_dirinfo.c */ extern void e2fsck_add_dx_dir(e2fsck_t ctx, ext2_ino_t ino, diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 783e14f0..7095e8b4 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2412,6 +2412,22 @@ static void e2fsck_pass1_merge_dir_info(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->dir_info); } +static void e2fsck_pass1_merge_dx_dir(e2fsck_t global_ctx, e2fsck_t thread_ctx) +{ + if (thread_ctx->dx_dir_info == NULL) + return; + + if (global_ctx->dx_dir_info == NULL) { + global_ctx->dx_dir_info = thread_ctx->dx_dir_info; + global_ctx->dx_dir_info_size = thread_ctx->dx_dir_info_size; + global_ctx->dx_dir_info_count = thread_ctx->dx_dir_info_count; + thread_ctx->dx_dir_info = NULL; + return; + } + + e2fsck_merge_dx_dir(global_ctx, thread_ctx); +} + static inline errcode_t e2fsck_pass1_merge_icount(ext2_icount_t *dest_icount, ext2_icount_t *src_icount) @@ -2457,6 +2473,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx FILE *global_problem_logf = global_ctx->problem_logf; ext2fs_inode_bitmap inode_bad_map = global_ctx->inode_bad_map; struct dir_info_db *dir_info = global_ctx->dir_info; + struct dx_dir_info *dx_dir_info = global_ctx->dx_dir_info; ext2fs_inode_bitmap inode_used_map = global_ctx->inode_used_map; ext2fs_inode_bitmap inode_dir_map = global_ctx->inode_dir_map; ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; @@ -2486,6 +2503,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx __u32 fs_fragmented = global_ctx->fs_fragmented; __u32 fs_fragmented_dir = global_ctx->fs_fragmented_dir; __u32 large_files = global_ctx->large_files; + ext2_ino_t dx_dir_info_size = global_ctx->dx_dir_info_size; + ext2_ino_t dx_dir_info_count = global_ctx->dx_dir_info_count; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2510,6 +2529,10 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->block_metadata_map = block_metadata_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); + global_ctx->dx_dir_info = dx_dir_info; + global_ctx->dx_dir_info_count = dx_dir_info_count; + global_ctx->dx_dir_info_size = dx_dir_info_size; + e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); global_ctx->inode_count = inode_count; global_ctx->inode_link_info = inode_link_info; global_ctx->fs_directory_count += fs_directory_count; From patchwork Wed Nov 18 15:39:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402334 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=bxZhwYLI; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn961GkHz9sTc for ; Thu, 19 Nov 2020 02:41:30 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727437AbgKRPlR (ORCPT ); Wed, 18 Nov 2020 10:41:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50182 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727517AbgKRPlQ (ORCPT ); Wed, 18 Nov 2020 10:41:16 -0500 Received: from mail-qt1-x84a.google.com (mail-qt1-x84a.google.com [IPv6:2607:f8b0:4864:20::84a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDA8AC0613D6 for ; Wed, 18 Nov 2020 07:41:14 -0800 (PST) Received: by mail-qt1-x84a.google.com with SMTP id y10so1673576qtw.5 for ; Wed, 18 Nov 2020 07:41:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=lmBUqkA6D+Dfo/wYKxTxIBoZlgxKgigsG3WwP4pFDxI=; b=bxZhwYLIA2DyjmSWjv9GFo3f9Yxsg2KmbYBndNOZsjit5R4Rnx3Zrntl/qhHJZLMlD wewABSwr31uLxlUv9zC3UHYCMKQOINA8fdl1kQnNrMg8FTxf6LbozwDAyB91/xsL/EGK s2QA57UjLuTc6PzX7KYiOKo7eVbVj14difRP+UGrUKXOLapqzZzg5rHlPddaOSL6ykBL +bEkQ3EOdXnugpcDKjuGPi+k1QWBL4b5p9vzv0frQyckERgNGeVZq/XcLjDKToQbM1xP shI0bYHUv4MAYl9hR32GIIBGBRNur0sgl35GoQZtViyGNWg5mvHVdqZZ27VG0BpJQIDU USAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lmBUqkA6D+Dfo/wYKxTxIBoZlgxKgigsG3WwP4pFDxI=; b=FjW9Z5OgfGOxv3AnM9vg2D7/qfvjW1erw3KWN90pk7XRqJcYOW7g/PiU/KPoYiQjPS 01UUy6gAVz4rYGslf2FdM2kJMr9exWdKHZcvSGByAfQLihDv70R3Ikik3n+NzRNuXzCv ZrrZiwLX+akWZRtwMfY7znY5B+oKGhpqo6MEorV+xrdRCVanB2AutzmFHReH+7u3WHGE ZBQP+YmPVZAXlvJgnwU1iW8UH26JBGY+5rLNF0KVx0yzWkKlwO2YKuEitU9UneULHFIc F9ItNdi4mOiAmxVBEgx6UdULlIPDLe+YOa4xGxF4/oSS4Z74VnAJrKAImrg5hvUlzwSo BvSA== X-Gm-Message-State: AOAM53383U1F0GHT44YI5WNZxarxH4TF5MLlqPYq6UuF5YuobvGjiDRl KF4UnUQs8MehxwUE/p3ok3J3uUPdV/0olWFQ+5KSLnXOxyM/Cv1PpPt+Px0WKMteYmSKLa8LgUO 4J1yyDolFYA6Jds355wFZQTN5b5IaVs7ijRaFPemU+Y2OFUIV121VFU4k8CoFVihP3wWg0uUX7P qloeNMB70= X-Google-Smtp-Source: ABdhPJy0DKe84588bI5AiHoLKb6Sd0f2W3e9Taiqk1AHDCQABK/GCvp2f5gGib4diXeLrMcUppF/MVsPmpnPdwzOuw4= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:9c05:: with SMTP id v5mr5078182qve.7.1605714073995; Wed, 18 Nov 2020 07:41:13 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:12 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-27-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 26/61] e2fsck: merge dirs_to_hash when threads finish From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong @dirs_to_hash list need be merged after threads finish, test covered by t_dangerous. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 7095e8b4..5378d7da 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2463,6 +2463,25 @@ static errcode_t e2fsck_pass1_merge_icounts(e2fsck_t global_ctx, e2fsck_t thread return ret; } +static errcode_t e2fsck_pass1_merge_dirs_to_hash(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + errcode_t retval = 0; + + if (!thread_ctx->dirs_to_hash) + return 0; + + if (!global_ctx->dirs_to_hash) + retval = ext2fs_badblocks_copy(thread_ctx->dirs_to_hash, + &global_ctx->dirs_to_hash); + else + retval = ext2fs_badblocks_merge(thread_ctx->dirs_to_hash, + global_ctx->dirs_to_hash); + + return retval; +} + + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2505,6 +2524,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx __u32 large_files = global_ctx->large_files; ext2_ino_t dx_dir_info_size = global_ctx->dx_dir_info_size; ext2_ino_t dx_dir_info_count = global_ctx->dx_dir_info_count; + ext2_u32_list dirs_to_hash = global_ctx->dirs_to_hash; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2574,6 +2594,14 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx return retval; } + global_ctx->dirs_to_hash = dirs_to_hash; + retval = e2fsck_pass1_merge_dirs_to_hash(global_ctx, thread_ctx); + if (retval) { + com_err(global_ctx->program_name, 0, + _("while merging dirs to hash\n")); + return retval; + } + retval = e2fsck_pass1_merge_bitmap(global_fs, &thread_ctx->inode_used_map, &global_ctx->inode_used_map); @@ -2660,6 +2688,8 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) e2fsck_free_dir_info(thread_ctx); ext2fs_free_icount(thread_ctx->inode_count); ext2fs_free_icount(thread_ctx->inode_link_info); + if (thread_ctx->dirs_to_hash) + ext2fs_badblocks_list_free(thread_ctx->dirs_to_hash); ext2fs_free_mem(&thread_ctx); return retval; From patchwork Wed Nov 18 15:39:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402333 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=tdKEayAE; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn955DwZz9sVC for ; Thu, 19 Nov 2020 02:41:29 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727521AbgKRPlR (ORCPT ); Wed, 18 Nov 2020 10:41:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727437AbgKRPlQ (ORCPT ); Wed, 18 Nov 2020 10:41:16 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F186C0613D4 for ; Wed, 18 Nov 2020 07:41:16 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id v12so2963061ybi.6 for ; Wed, 18 Nov 2020 07:41:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=AZGCHeBKH5vmz22llsXqlzZoKg5JQiUV/C+mJX7pT5A=; b=tdKEayAEYP7W+n1STmhODIwNqHkwgEIUfMDDICLJ72jfzKbbkRQaKmrSNFEphFX9Ry kO/qMM0uw057WzaOnl1MOS2/Vrt00KLRua48TwLW5fWFWjri0TLQqvcWGzvwXewbnL7S EAMhBu26Fi0im/XvOws+rEqstpXt5Av1OaRMlCchvAIxi5JUUDSFLRVVXCdcun6bD6bb l84F+QLO3ZPSzl5NEi+9+NkC8CG3YiVik9lTQvvO3zuYfl5HTUap0Yu/WxGG1FzzMZEz 7K227He9p7a+akLdxrTrfi4+Giy1z1O0M4HW0ndP/5/zeUWLrf2hRB6hPgOKF6hed3Zi OzHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=AZGCHeBKH5vmz22llsXqlzZoKg5JQiUV/C+mJX7pT5A=; b=m8xz2kRrQspMHrHyesgIvLCqrqBUeT45TI5dm9XLd+TGnOVtU1zPNzH6Nnvt6KGEd6 E8opqPKoVyNKuklGTOetx3dOFF+2xoSnl2NqwVH4J3ZzHtFW8zaT4oBIRTs+Rv4diz/y vxHOAb5T6mp253O+nveX4Za4ke1kq4P159tkeetMDXTxCF4TaFJOI1ar8k90wHZu54KZ fkZCjaIR4LTDlDPLAofi22la7eeMTMilQrgg70bYtv4nDzCUAzUSJP0WXyOr4H0Ryy37 PA/I95gKGNJRlNF6qXYh1wqBPAy3voWMqvK1sib24+Icc/bSnDKSnWkUV1cHhPR3agW8 uRzw== X-Gm-Message-State: AOAM530ZogXqP1wx6UOuRW7d8LjP59rkCHRqzEW1htIGE2Ed2nreL1Fs BcRCR8xI7uL1D7fWRZpDFt5fU9WzzMpFoPSTdtc0lL9nzMy5puF3YZzQxcYXM/iV2+T3TIOt/Em /D0lA1s+mWsjRPng73I1KObu2qn4iuxab7csn3nwY2hxWIutZ7HX/iMSoCGp78vNd3SUMMf6qfR 8mjfABu24= X-Google-Smtp-Source: ABdhPJzzMv9t5/qqTUioxf2gs7QpmYDExxWKfLs1avcwTcepDCkT0VCMOyhGYurCIAanb9Pg3CeOlTEEP2Xtvrg3rAE= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:2504:: with SMTP id l4mr8369407ybl.72.1605714075690; Wed, 18 Nov 2020 07:41:15 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:13 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-28-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 27/61] e2fsck: merge context flags properly From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong e2fsck might restart after pass1, so we should keep flags if possible, this patch try to fix f_illitable_flexbg failure Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 5378d7da..35ab9cae 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2573,9 +2573,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->fs_fragmented_dir += fs_fragmented_dir; global_ctx->large_files += large_files; - /* Keep the global singal flags*/ - global_ctx->flags |= (flags & E2F_FLAG_SIGNAL_MASK) | - (global_ctx->flags & E2F_FLAG_SIGNAL_MASK); + global_ctx->flags |= flags; retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); if (retval) { From patchwork Wed Nov 18 15:39:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402335 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=FQ5hyLP0; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9645SGz9sVp for ; Thu, 19 Nov 2020 02:41:30 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727527AbgKRPlU (ORCPT ); Wed, 18 Nov 2020 10:41:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727524AbgKRPlU (ORCPT ); Wed, 18 Nov 2020 10:41:20 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A463C0613D4 for ; Wed, 18 Nov 2020 07:41:18 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id e142so2927975ybf.16 for ; Wed, 18 Nov 2020 07:41:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=E9RqpHkCRIeJGjpoYdPThmM33eUrY9Rpw1GJZsh0IZI=; b=FQ5hyLP059Xu8NotXgLwrVVBbGazgOVOdnFeap+pf8Ct29G8dveLA658BcKtydTOzB An2m1hXBkixN/l9ERxayCaElInRjNi5XwUUHpeLNudI1m75dslta4Zki2HPHDSyIpoAT k2WyUJFrr4a/Iz1c5AKgASit1Iw5fZCTt8y8cdEiaHj0fhOz0zqIvNwBkVV5XKwCDDdw A1gNW6I3HHxbeYCChsg3l7JTfGc20t1EKeWb5uK2uXjWycq88Lnaw09a1l2jHktAZ9Lu jU7k+nGIbQ8Pqi+bc2J1rIHtiU2sfJbDrrUBLvraAWGdc1fXsListyXXICte6JAr8oOL xdzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=E9RqpHkCRIeJGjpoYdPThmM33eUrY9Rpw1GJZsh0IZI=; b=HZjjJ4R53wMvnmZLQ0n4VAb5sZ2v7kXzMDAnUB1NG30xj8Z+wUDY/QhkXF0T1yA+8y 4NMB3pMp3aOXSHyN5+V7wtMx1zjh/D0nFN/0GLfQ7WNWR21iY7QVe3uA1cIO1778rUaq lFKkSxHK604yQ1vR+URlsX0CqeR5rL19Z37Yf937GehafK6lSncQT4pvT4rcj0JEH73C JXKRcAnze+rvkA/GZa1DFBy0CTVNubAbvY3X7KuoSDjd5Bcx+5UMuyQ2Z+k5x5TB/JvD Hfr/Kn2m/Ec1984kIChKT3PTZveIeEFPnznxuWnC647KuKJlTuymNNoDPUZGR+4UMxKp amqg== X-Gm-Message-State: AOAM530XpHtdExcOCumnYuW9ecn5qUZJln0bdcwQ897+8IzU/VKTxaSR VE1dKn0WlUJ79r2YQhFFw7mBhHdEvVJ/2PRd6GZxke4gzH3vsvcUnM6HqwIJAt9pE9n19GhqGHI Y/YBZfUfNQLL1iFnLc5/B1nACbNy/5Q1/cJS180xGq0du0r4oY+GxBgcxfUe4Q1soEFgB5iESSt OLj1vO4FE= X-Google-Smtp-Source: ABdhPJyEbWED0YBw0tmdxnunX52QAk0k1WlsqEdBCU/EyDDIGbuvNNkgbIi02umgvN4F/K3eLPar9aZATPw3XoAAj4Y= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:3792:: with SMTP id e140mr9586871yba.277.1605714077606; Wed, 18 Nov 2020 07:41:17 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:14 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-29-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 28/61] e2fsck: merge quota context after threads finish From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Every threads calculate its own quota accounting, merge them after threads finish. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 15 ++++++++++++++- lib/support/mkquota.c | 39 +++++++++++++++++++++++++++++++++++++++ lib/support/quotaio.h | 3 +++ 3 files changed, 56 insertions(+), 1 deletion(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 35ab9cae..f2476261 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2388,6 +2388,12 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre log_out(thread_context, _("Scan group range [%d, %d)\n"), tinfo->et_group_start, tinfo->et_group_end); thread_context->fs = thread_fs; + retval = quota_init_context(&thread_context->qctx, thread_fs, 0); + if (retval) { + com_err(global_ctx->program_name, retval, + "while init quota context"); + goto out_fs; + } *thread_ctx = thread_context; return 0; out_fs: @@ -2481,7 +2487,6 @@ static errcode_t e2fsck_pass1_merge_dirs_to_hash(e2fsck_t global_ctx, return retval; } - static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2525,6 +2530,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_ino_t dx_dir_info_size = global_ctx->dx_dir_info_size; ext2_ino_t dx_dir_info_count = global_ctx->dx_dir_info_count; ext2_u32_list dirs_to_hash = global_ctx->dirs_to_hash; + quota_ctx_t qctx = global_ctx->qctx; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2600,6 +2606,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx return retval; } + global_ctx->qctx = qctx; + retval = quota_merge_and_update_usage(global_ctx->qctx, + thread_ctx->qctx); + if (retval) + return retval; + retval = e2fsck_pass1_merge_bitmap(global_fs, &thread_ctx->inode_used_map, &global_ctx->inode_used_map); @@ -2688,6 +2700,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) ext2fs_free_icount(thread_ctx->inode_link_info); if (thread_ctx->dirs_to_hash) ext2fs_badblocks_list_free(thread_ctx->dirs_to_hash); + quota_release_context(&thread_ctx->qctx); ext2fs_free_mem(&thread_ctx); return retval; diff --git a/lib/support/mkquota.c b/lib/support/mkquota.c index 6f7ae6d6..c16a3d0a 100644 --- a/lib/support/mkquota.c +++ b/lib/support/mkquota.c @@ -639,6 +639,45 @@ out: return err; } +static errcode_t merge_usage(dict_t *dest, dict_t *src) +{ + dnode_t *n; + struct dquot *src_dq, *dest_dq; + + for (n = dict_first(src); n; n = dict_next(src, n)) { + src_dq = dnode_get(n); + if (!src_dq) + continue; + dest_dq = get_dq(dest, src_dq->dq_id); + if (dest_dq == NULL) + return -ENOMEM; + dest_dq->dq_dqb.dqb_curspace += src_dq->dq_dqb.dqb_curspace; + dest_dq->dq_dqb.dqb_curinodes += src_dq->dq_dqb.dqb_curinodes; + } + + return 0; +} + + +errcode_t quota_merge_and_update_usage(quota_ctx_t dest_qctx, + quota_ctx_t src_qctx) +{ + dict_t *dict; + enum quota_type qtype; + errcode_t retval = 0; + + for (qtype = 0; qtype < MAXQUOTAS; qtype++) { + dict = src_qctx->quota_dict[qtype]; + if (!dict) + continue; + retval = merge_usage(dest_qctx->quota_dict[qtype], dict); + if (retval) + break; + } + + return retval; +} + /* * Compares the measured quota in qctx->quota_dict with that in the quota inode * on disk and updates the limits in qctx->quota_dict. 'usage_inconsistent' is diff --git a/lib/support/quotaio.h b/lib/support/quotaio.h index 60689700..bca295a1 100644 --- a/lib/support/quotaio.h +++ b/lib/support/quotaio.h @@ -40,6 +40,7 @@ #include "ext2fs/ext2_fs.h" #include "ext2fs/ext2fs.h" #include "dqblk_v2.h" +#include "support/dict.h" typedef int64_t qsize_t; /* Type in which we store size limitations */ @@ -233,6 +234,8 @@ int quota_file_exists(ext2_filsys fs, enum quota_type qtype); void quota_set_sb_inum(ext2_filsys fs, ext2_ino_t ino, enum quota_type qtype); errcode_t quota_compare_and_update(quota_ctx_t qctx, enum quota_type qtype, int *usage_inconsistent); +errcode_t quota_merge_and_update_usage(quota_ctx_t dest_qctx, + quota_ctx_t src_qctx); int parse_quota_opts(const char *opts, int (*func)(char *)); /* parse_qtype.c */ From patchwork Wed Nov 18 15:39:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402336 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=ieCSvWxw; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9707csz9sVq for ; Thu, 19 Nov 2020 02:41:31 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727524AbgKRPlW (ORCPT ); Wed, 18 Nov 2020 10:41:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726524AbgKRPlW (ORCPT ); Wed, 18 Nov 2020 10:41:22 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FD8EC0613D4 for ; Wed, 18 Nov 2020 07:41:20 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id fy23so1312017pjb.9 for ; Wed, 18 Nov 2020 07:41:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=TVe1odX/JVSYE4SRhbM5hD7LUAtvYJZZ2Y4+3CnwaDs=; b=ieCSvWxwDPesusq+/yuP0Et8s7LyCjd4diPJxNlpfGJF9F6VjaCW0upDgAbKVEs1gI pU9t3I1VCL9PyGwh6NtGAmEt6OtgAqrR68BeuJi5cI42nl+RJ6i3YbPw7teLwO6Ah5wG 6FA/oK5x2AdefRp6Ph1O4BGaERZdAhxEFAe3KLbXyG/rzsrGgVY7ODOX5g7A9u4l1Fkx 60dogVKARJtpg40UpOMEsgwvslG31o8Pg/W7RJ9NcX4o5lyLPlwAE7rmgaATMUHTw51a Tq6k7RfSbKkTjVGSHesd2lrUHaRtYIYVn2LGGbsnn6Yr7ODLssDqceemNk2W7XUgjXJ4 8Rbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=TVe1odX/JVSYE4SRhbM5hD7LUAtvYJZZ2Y4+3CnwaDs=; b=D416cLQcdum6QvFooUeaTUW8Cvg10J0NcUYhbVxwY0f4ECrVAYoPXN6Y5VCPx20P4T gaiBEMrwnbo870yNBCBOdEe3S/JdcBLk0uXH4N8r7cmbJxkjtW/BJAaUtb+JlmMOAnEO ywkkhf6TrfuuS5mtI9sw73XlFBOMHCaqBV9dfn0g75s4jxds9a6+diPfDWeMYLlri4Gh C8VoLp2vtSJyPQqN1Vg6R16lW6q7XdgkvkPzSyruBHhZ+Ckb2V8fRM3HCDnxvGfanvEo bQjK07KtRLrYMxbPfP0l17XNF3SQOtNsbhYEgQ6rWmO11Q223y/M/zVYv8OlnZHAiXj6 unPg== X-Gm-Message-State: AOAM533TBS/Ebuuo7cidf4BPoq3HkDoBvYT8l2OTEMGrYKkOqPqLsFjo jcb3k0hUsvKLl2masF05GLIkhBtKlHOXmKhXsl0LKugz1zBw7Ob/FQMkC2aaGUewBtbh7svJXNJ d4j11EZWXqSpE/zA6rqUy+EBl5bZL4rF1jQgfsCAK+yTWt91jaqRPw5H9s7mPLrXV1SHTxeCVu1 pvY6CABlY= X-Google-Smtp-Source: ABdhPJyOPRMxcPubQGdiMeVPnIW7PaSf5KHBFZi9KPsKEL3S8efRzWHeUc5pDRGW0+mfw9LbZ7ou49Hoc7WKQJn/hT4= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:90b:e04:: with SMTP id ge4mr46315pjb.0.1605714079596; Wed, 18 Nov 2020 07:41:19 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:15 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-30-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 29/61] e2fsck: serialize fix operations From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Allow different threads to fix at the same time could be dangerous and error-prone now, and most of time parallel scanning and checking is important. So this patch adds a mutex to serialize fix operations during pass1. And the good benefit of this, we don't need block allocations and free, superblock updates protection any more, since only fix operations during pass1 could touch them. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 6 +++++ e2fsck/pass1.c | 63 +++++++++++++++++++++++++++++++++++++++++++------ e2fsck/util.c | 38 +++++++++++++++++++++++++++++ 3 files changed, 100 insertions(+), 7 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index c3b0af34..777d8b96 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -447,6 +447,10 @@ struct e2fsck_struct { /* Undo file */ char *undo_file; +#ifdef CONFIG_PFSCK + /* serialize fix operation for multiple threads */ + pthread_mutex_t fs_fix_mutex; +#endif }; #ifdef CONFIG_PFSCK @@ -744,6 +748,8 @@ extern errcode_t e2fsck_allocate_subcluster_bitmap(ext2_filsys fs, const char *profile_name, ext2fs_block_bitmap *ret); unsigned long long get_memory_size(void); +extern void e2fsck_pass1_fix_lock(e2fsck_t ctx); +extern void e2fsck_pass1_fix_unlock(e2fsck_t ctx); /* unix.c */ extern void e2fsck_clear_progbar(e2fsck_t ctx); diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index f2476261..594571a7 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -380,8 +380,10 @@ static problem_t check_large_ea_inode(e2fsck_t ctx, pctx->num = entry->e_value_inum; if (fix_problem(ctx, PR_1_ATTR_SET_EA_INODE_FL, pctx)) { inode.i_flags |= EXT4_EA_INODE_FL; + e2fsck_pass1_fix_lock(ctx); ext2fs_write_inode(ctx->fs, entry->e_value_inum, &inode); + e2fsck_pass1_fix_unlock(ctx); } else { return PR_1_ATTR_NO_EA_INODE_FL; } @@ -872,8 +874,11 @@ static errcode_t recheck_bad_inode_checksum(ext2_filsys fs, ext2_ino_t ino, if (!fix_problem(ctx, PR_1_INODE_ONLY_CSUM_INVALID, pctx)) return 0; + + e2fsck_pass1_fix_lock(ctx); retval = ext2fs_write_inode_full(fs, ino, (struct ext2_inode *)&inode, sizeof(inode)); + e2fsck_pass1_fix_unlock(ctx); return retval; } @@ -883,15 +888,19 @@ static void reserve_block_for_root_repair(e2fsck_t ctx) errcode_t err; ext2_filsys fs = ctx->fs; + e2fsck_pass1_fix_lock(ctx); ctx->root_repair_block = 0; if (ext2fs_test_inode_bitmap2(ctx->inode_used_map, EXT2_ROOT_INO)) - return; + goto out; err = ext2fs_new_block2(fs, 0, ctx->block_found_map, &blk); if (err) - return; + goto out; ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); ctx->root_repair_block = blk; +out: + e2fsck_pass1_fix_unlock(ctx); + return; } static void reserve_block_for_lnf_repair(e2fsck_t ctx) @@ -902,15 +911,19 @@ static void reserve_block_for_lnf_repair(e2fsck_t ctx) static const char name[] = "lost+found"; ext2_ino_t ino; + e2fsck_pass1_fix_lock(ctx); ctx->lnf_repair_block = 0; if (!ext2fs_lookup(fs, EXT2_ROOT_INO, name, sizeof(name)-1, 0, &ino)) - return; + goto out; err = ext2fs_new_block2(fs, 0, ctx->block_found_map, &blk); if (err) - return; + goto out; ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); ctx->lnf_repair_block = blk; +out: + e2fsck_pass1_fix_unlock(ctx); + return; } static errcode_t get_inline_data_ea_size(ext2_filsys fs, ext2_ino_t ino, @@ -1531,8 +1544,10 @@ void e2fsck_pass1_run(e2fsck_t ctx) pctx.errcode = get_inline_data_ea_size(fs, ino, &size); if (!pctx.errcode && fix_problem(ctx, PR_1_INLINE_DATA_FEATURE, &pctx)) { + e2fsck_pass1_fix_lock(ctx); ext2fs_set_feature_inline_data(sb); ext2fs_mark_super_dirty(fs); + e2fsck_pass1_fix_unlock(ctx); inlinedata_fs = 1; } else if (fix_problem(ctx, PR_1_INLINE_DATA_SET, &pctx)) { e2fsck_clear_inode(ctx, ino, inode, 0, "pass1"); @@ -1620,9 +1635,11 @@ void e2fsck_pass1_run(e2fsck_t ctx) if ((ext2fs_extent_header_verify(inode->i_block, sizeof(inode->i_block)) == 0) && fix_problem(ctx, PR_1_EXTENT_FEATURE, &pctx)) { + e2fsck_pass1_fix_lock(ctx); ext2fs_set_feature_extents(sb); ext2fs_mark_super_dirty(fs); extent_fs = 1; + e2fsck_pass1_fix_unlock(ctx); } else if (fix_problem(ctx, PR_1_EXTENTS_SET, &pctx)) { clear_inode: e2fsck_clear_inode(ctx, ino, inode, 0, "pass1"); @@ -2031,8 +2048,11 @@ void e2fsck_pass1_run(e2fsck_t ctx) ctx->ea_block_quota_inodes = 0; } - if (ctx->invalid_bitmaps) + if (ctx->invalid_bitmaps) { + e2fsck_pass1_fix_lock(ctx); handle_fs_bad_blocks(ctx); + e2fsck_pass1_fix_unlock(ctx); + } /* We don't need the block_ea_map any more */ if (ctx->block_ea_map) { @@ -2045,7 +2065,9 @@ void e2fsck_pass1_run(e2fsck_t ctx) if (ctx->flags & E2F_FLAG_RESIZE_INODE) { clear_problem_context(&pctx); + e2fsck_pass1_fix_lock(ctx); pctx.errcode = ext2fs_create_resize_inode(fs); + e2fsck_pass1_fix_unlock(ctx); if (pctx.errcode) { if (!fix_problem(ctx, PR_1_RESIZE_INODE_CREATE, &pctx)) { @@ -2874,6 +2896,7 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) int num_threads = 1; errcode_t retval; + pthread_mutex_init(&global_ctx->fs_fix_mutex, NULL); retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); if (retval) { com_err(global_ctx->program_name, retval, @@ -3172,6 +3195,18 @@ static _INLINE_ void mark_blocks_used(e2fsck_t ctx, blk64_t block, } } +static errcode_t _INLINE_ e2fsck_write_ext_attr3(e2fsck_t ctx, blk64_t block, + void *inbuf, ext2_ino_t inum) +{ + errcode_t retval; + ext2_filsys fs = ctx->fs; + + e2fsck_pass1_fix_lock(ctx); + retval = ext2fs_write_ext_attr3(fs, block, inbuf, inum); + e2fsck_pass1_fix_unlock(ctx); + + return retval; +} /* * Adjust the extended attribute block's reference counts at the end * of pass 1, either by subtracting out references for EA blocks that @@ -3208,7 +3243,7 @@ static void adjust_extattr_refcount(e2fsck_t ctx, ext2_refcount_t refcount, pctx.num = should_be; if (fix_problem(ctx, PR_1_EXTATTR_REFCOUNT, &pctx)) { header->h_refcount = should_be; - pctx.errcode = ext2fs_write_ext_attr3(fs, blk, + pctx.errcode = e2fsck_write_ext_attr3(ctx, blk, block_buf, pctx.ino); if (pctx.errcode) { @@ -3439,7 +3474,7 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, */ if (failed_csum && fix_problem(ctx, PR_1_EA_BLOCK_ONLY_CSUM_INVALID, pctx)) { - pctx->errcode = ext2fs_write_ext_attr3(fs, blk, block_buf, + pctx->errcode = e2fsck_write_ext_attr3(ctx, blk, block_buf, pctx->ino); if (pctx->errcode) return 0; @@ -3717,10 +3752,12 @@ static void scan_extent_node(e2fsck_t ctx, struct problem_context *pctx, if (try_repairs && is_dir && problem == 0 && (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) && fix_problem(ctx, PR_1_UNINIT_DBLOCK, pctx)) { + e2fsck_pass1_fix_lock(ctx); extent.e_flags &= ~EXT2_EXTENT_FLAGS_UNINIT; pb->inode_modified = 1; pctx->errcode = ext2fs_extent_replace(ehandle, 0, &extent); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) return; failed_csum = 0; @@ -3764,13 +3801,17 @@ report_problem: } e2fsck_read_bitmaps(ctx); pb->inode_modified = 1; + e2fsck_pass1_fix_lock(ctx); pctx->errcode = ext2fs_extent_delete(ehandle, 0); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) { pctx->str = "ext2fs_extent_delete"; return; } + e2fsck_pass1_fix_lock(ctx); pctx->errcode = ext2fs_extent_fix_parents(ehandle); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode && pctx->errcode != EXT2_ET_NO_CURRENT_NODE) { pctx->str = "ext2fs_extent_fix_parents"; @@ -3834,9 +3875,11 @@ report_problem: pctx->num = e_info.curr_level - 1; problem = PR_1_EXTENT_INDEX_START_INVALID; if (fix_problem(ctx, problem, pctx)) { + e2fsck_pass1_fix_lock(ctx); pb->inode_modified = 1; pctx->errcode = ext2fs_extent_fix_parents(ehandle); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) { pctx->str = "ext2fs_extent_fix_parents"; return; @@ -3900,15 +3943,19 @@ report_problem: pctx->blk = extent.e_lblk; pctx->blk2 = new_lblk; if (fix_problem(ctx, PR_1_COLLAPSE_DBLOCK, pctx)) { + e2fsck_pass1_fix_lock(ctx); extent.e_lblk = new_lblk; pb->inode_modified = 1; pctx->errcode = ext2fs_extent_replace(ehandle, 0, &extent); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) { pctx->errcode = 0; goto alloc_later; } + e2fsck_pass1_fix_lock(ctx); pctx->errcode = ext2fs_extent_fix_parents(ehandle); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) goto failed_add_dir_block; pctx->errcode = ext2fs_extent_goto(ehandle, @@ -4004,8 +4051,10 @@ alloc_later: /* Failed csum but passes checks? Ask to fix checksum. */ if (failed_csum && fix_problem(ctx, PR_1_EXTENT_ONLY_CSUM_INVALID, pctx)) { + e2fsck_pass1_fix_lock(ctx); pb->inode_modified = 1; pctx->errcode = ext2fs_extent_replace(ehandle, 0, &extent); + e2fsck_pass1_fix_unlock(ctx); if (pctx->errcode) return; } diff --git a/e2fsck/util.c b/e2fsck/util.c index a388bd70..8eec477c 100644 --- a/e2fsck/util.c +++ b/e2fsck/util.c @@ -42,6 +42,10 @@ #include #endif +#ifdef CONFIG_PFSCK +#include +#endif + #include "e2fsck.h" extern e2fsck_t e2fsck_global_ctx; /* Try your very best not to use this! */ @@ -565,13 +569,45 @@ void e2fsck_read_inode_full(e2fsck_t ctx, unsigned long ino, } } +#ifdef CONFIG_PFSCK +void e2fsck_pass1_fix_lock(e2fsck_t ctx) +{ + e2fsck_t global_ctx = ctx->global_ctx; + if (!global_ctx) + global_ctx = ctx; + + pthread_mutex_lock(&global_ctx->fs_fix_mutex); +} + +void e2fsck_pass1_fix_unlock(e2fsck_t ctx) +{ + e2fsck_t global_ctx = ctx->global_ctx; + if (!global_ctx) + global_ctx = ctx; + + pthread_mutex_unlock(&global_ctx->fs_fix_mutex); +} +#else +void e2fsck_pass1_fix_lock(e2fsck_t ctx) +{ + +} + +void e2fsck_pass1_fix_unlock(e2fsck_t ctx) +{ + +} +#endif + void e2fsck_write_inode_full(e2fsck_t ctx, unsigned long ino, struct ext2_inode * inode, int bufsize, const char *proc) { errcode_t retval; + e2fsck_pass1_fix_lock(ctx); retval = ext2fs_write_inode_full(ctx->fs, ino, inode, bufsize); + e2fsck_pass1_fix_unlock(ctx); if (retval) { com_err("ext2fs_write_inode", retval, _("while writing inode %lu in %s"), ino, proc); @@ -584,7 +620,9 @@ void e2fsck_write_inode(e2fsck_t ctx, unsigned long ino, { errcode_t retval; + e2fsck_pass1_fix_lock(ctx); retval = ext2fs_write_inode(ctx->fs, ino, inode); + e2fsck_pass1_fix_unlock(ctx); if (retval) { com_err("ext2fs_write_inode", retval, _("while writing inode %lu in %s"), ino, proc); From patchwork Wed Nov 18 15:39:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402337 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=f8DPkwFd; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn972tQKz9sVs for ; Thu, 19 Nov 2020 02:41:31 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727530AbgKRPlY (ORCPT ); Wed, 18 Nov 2020 10:41:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726524AbgKRPlW (ORCPT ); Wed, 18 Nov 2020 10:41:22 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AEA07C0613D4 for ; Wed, 18 Nov 2020 07:41:22 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id v188so2873189ybc.19 for ; Wed, 18 Nov 2020 07:41:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=pHlJ0i5ksTG/QCCZPUMiKyeLB7mnIPuIMptsj+rv/IY=; b=f8DPkwFdC6oK5hbVg1i66kFYT3FtznB3/VFBJyf1YTLEt7z96xnK1Llzu7R3POmufE If/0PoC/SJLzk+uXYKz2TY1Gr+JJbbalh9hrVoc5IYBbanAtHfjKlzjzh25cx752DYZ+ RiR1I8xWZqlJGAi+UYddKbV5DZ9DFN5Vmzs/X6DqYecVnGfvkNjD4Q6IfIrq1jgg9evU OePoU6pRS7NI1GT+l3vS2d/yJcrXk2E4J8t6r6P/YT70fY0GLYu1LPaVMcJnLQyMFnrs eq/wH+VErzG+VqdjnDowEbatFh3El568YcUxqszR70luIrh1pvsPLq3+8i/LFBoYo9TG fvhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=pHlJ0i5ksTG/QCCZPUMiKyeLB7mnIPuIMptsj+rv/IY=; b=cRKxwqw/WSg+R2i+JZXTnSUidEz6geFH+OtH8SF6RrsDN5g3drdGmyIhMOwu2Mkn68 v5XctFVmuKuQ9+0o+rbTrXZcUmHbNyIqvca6Hn8jo/L7AvT641vjV4TbsX54RqYp6GJO iWn6/W8yPRYu5UURghwxfTEBaghzrmeXdyKwOuVu7RAzv3v7BheKacDj4MLI2zNjNk5k S+dSmS8jQZNXKDHUh1aO+jt+l/TrOuXT1djtdmt0MT+ZgYs0imOIRqkFI4S0+RZguGDH VaJmc9/13XDn9J1NSheZkJJM+WRIvt46HQEWFj9jHfqM/PHogHV1Sno0AQmU/ez9Bsl/ 7tmg== X-Gm-Message-State: AOAM533E+iHVYoffU6+kZgpXYqYSIW7gvJOKECIcf854Ofwz4xgG0hFR z7OXUSQ9gg//PovQP70VozuX3XG7dlKiuyBpAlkU4AuHTpHqRrrNQs5y2pQudYF0sPliRirpTP+ 84b5DmhbBZTEHW52pU/OOEnTDH9fuHhERFFUtUFvdIuAPR81t9Qeoi2GH8IzwiAJBre5/knu3dj pN9fnjgFU= X-Google-Smtp-Source: ABdhPJxGC6dLr+drLsizPgXLkUgGeEPLimdIZIxr3tyaROQQkele7MRQ6n+MOdq3jQBJy4UeE2DeCDNq5GJL0KML7vE= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:c74b:: with SMTP id w72mr10495275ybe.117.1605714081827; Wed, 18 Nov 2020 07:41:21 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:16 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-31-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 30/61] e2fsck: move some fixes out of parallel pthreads From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong We could only use @found_map_block to find free blocks after we have collectd all used blocks, so something like handle_fs_bad_blocks(), ext2fs_create_resize_inode(), e2fsck_pass1_dupblocks() really should be handled after all threads has been finished. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 4 + e2fsck/pass1.c | 306 ++++++++++++++++----------- e2fsck/util.c | 36 +++- tests/f_multithread/expect.1 | 2 +- tests/f_multithread_logfile/expect.1 | 2 +- tests/f_multithread_no/expect.1 | 2 +- 6 files changed, 216 insertions(+), 136 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 777d8b96..fecc8bbf 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -450,6 +450,8 @@ struct e2fsck_struct { #ifdef CONFIG_PFSCK /* serialize fix operation for multiple threads */ pthread_mutex_t fs_fix_mutex; + /* protect block_found_map, block_dup_map */ + pthread_mutex_t fs_block_map_mutex; #endif }; @@ -750,6 +752,8 @@ extern errcode_t e2fsck_allocate_subcluster_bitmap(ext2_filsys fs, unsigned long long get_memory_size(void); extern void e2fsck_pass1_fix_lock(e2fsck_t ctx); extern void e2fsck_pass1_fix_unlock(e2fsck_t ctx); +extern void e2fsck_pass1_block_map_lock(e2fsck_t ctx); +extern void e2fsck_pass1_block_map_unlock(e2fsck_t ctx); /* unix.c */ extern void e2fsck_clear_progbar(e2fsck_t ctx); diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 594571a7..49bdba21 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -750,11 +750,15 @@ static void check_is_really_dir(e2fsck_t ctx, struct problem_context *pctx, if (i >= 4) not_device++; + e2fsck_pass1_block_map_lock(ctx); if (blk < ctx->fs->super->s_first_data_block || blk >= ext2fs_blocks_count(ctx->fs->super) || ext2fs_fast_test_block_bitmap2(ctx->block_found_map, - blk)) + blk)) { + e2fsck_pass1_block_map_unlock(ctx); return; /* Invalid block, can't be dir */ + } + e2fsck_pass1_block_map_unlock(ctx); } blk = inode->i_block[0]; } @@ -888,19 +892,15 @@ static void reserve_block_for_root_repair(e2fsck_t ctx) errcode_t err; ext2_filsys fs = ctx->fs; - e2fsck_pass1_fix_lock(ctx); ctx->root_repair_block = 0; if (ext2fs_test_inode_bitmap2(ctx->inode_used_map, EXT2_ROOT_INO)) - goto out; + return; err = ext2fs_new_block2(fs, 0, ctx->block_found_map, &blk); if (err) - goto out; + return; ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); ctx->root_repair_block = blk; -out: - e2fsck_pass1_fix_unlock(ctx); - return; } static void reserve_block_for_lnf_repair(e2fsck_t ctx) @@ -911,18 +911,15 @@ static void reserve_block_for_lnf_repair(e2fsck_t ctx) static const char name[] = "lost+found"; ext2_ino_t ino; - e2fsck_pass1_fix_lock(ctx); ctx->lnf_repair_block = 0; if (!ext2fs_lookup(fs, EXT2_ROOT_INO, name, sizeof(name)-1, 0, &ino)) - goto out; + return; err = ext2fs_new_block2(fs, 0, ctx->block_found_map, &blk); if (err) - goto out; + return; ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); ctx->lnf_repair_block = blk; -out: - e2fsck_pass1_fix_unlock(ctx); return; } @@ -1177,6 +1174,118 @@ static int e2fsck_should_abort(e2fsck_t ctx) return 0; } +/* + * We need call mark_table_blocks() before multiple + * thread start, since all known system blocks should be + * marked and checked later. + */ +static errcode_t e2fsck_pass1_prepare(e2fsck_t ctx) +{ + struct problem_context pctx; + ext2_filsys fs = ctx->fs; + + clear_problem_context(&pctx); + if (!(ctx->options & E2F_OPT_PREEN)) + fix_problem(ctx, PR_1_PASS_HEADER, &pctx); + + pctx.errcode = e2fsck_allocate_subcluster_bitmap(ctx->fs, + _("in-use block map"), EXT2FS_BMAP64_RBTREE, + "block_found_map", &ctx->block_found_map); + if (pctx.errcode) { + pctx.num = 1; + fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); + ctx->flags |= E2F_FLAG_ABORT; + return pctx.errcode; + } + pctx.errcode = e2fsck_allocate_block_bitmap(ctx->fs, + _("metadata block map"), EXT2FS_BMAP64_RBTREE, + "block_metadata_map", &ctx->block_metadata_map); + if (pctx.errcode) { + pctx.num = 1; + fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); + ctx->flags |= E2F_FLAG_ABORT; + return pctx.errcode; + } + + mark_table_blocks(ctx); + pctx.errcode = ext2fs_convert_subcluster_bitmap(ctx->fs, + &ctx->block_found_map); + if (pctx.errcode) { + fix_problem(ctx, PR_1_CONVERT_SUBCLUSTER, &pctx); + ctx->flags |= E2F_FLAG_ABORT; + return pctx.errcode; + } + + if (ext2fs_has_feature_mmp(fs->super) && + fs->super->s_mmp_block > fs->super->s_first_data_block && + fs->super->s_mmp_block < ext2fs_blocks_count(fs->super)) + ext2fs_mark_block_bitmap2(ctx->block_found_map, + fs->super->s_mmp_block); + + return 0; +} + +static void e2fsck_pass1_post(e2fsck_t ctx) +{ + struct problem_context pctx; + ext2_filsys fs = ctx->fs; + char *block_buf; + + reserve_block_for_root_repair(ctx); + reserve_block_for_lnf_repair(ctx); + + if (ctx->invalid_bitmaps) + handle_fs_bad_blocks(ctx); + + if (ctx->flags & E2F_FLAG_RESIZE_INODE) { + struct ext2_inode *inode; + int inode_size = EXT2_INODE_SIZE(fs->super); + inode = e2fsck_allocate_memory(ctx, inode_size, + "scratch inode"); + + clear_problem_context(&pctx); + pctx.errcode = ext2fs_create_resize_inode(fs); + if (pctx.errcode) { + if (!fix_problem(ctx, PR_1_RESIZE_INODE_CREATE, + &pctx)) { + ctx->flags |= E2F_FLAG_ABORT; + ext2fs_free_mem(&inode); + ext2fs_free_mem(&block_buf); + return; + } + pctx.errcode = 0; + } + if (!pctx.errcode) { + e2fsck_read_inode(ctx, EXT2_RESIZE_INO, inode, + "recreate inode"); + inode->i_mtime = ctx->now; + e2fsck_write_inode(ctx, EXT2_RESIZE_INO, inode, + "recreate inode"); + } + ctx->flags &= ~E2F_FLAG_RESIZE_INODE; + ext2fs_free_mem(&inode); + } + + if (ctx->flags & E2F_FLAG_RESTART) { + ext2fs_free_mem(&block_buf); + return; + } + + if (ctx->block_dup_map) { + if (ctx->options & E2F_OPT_PREEN) { + clear_problem_context(&pctx); + fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); + } + block_buf = + (char *)e2fsck_allocate_memory(ctx, + ctx->fs->blocksize * 3, + "block interate buffer"); + e2fsck_pass1_dupblocks(ctx, block_buf); + ext2fs_free_mem(&block_buf); + } +} + + void e2fsck_pass1_run(e2fsck_t ctx) { int i; @@ -1215,9 +1324,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs); pass1_readahead(ctx, &ra_group, &ino_threshold); - if (!(ctx->options & E2F_OPT_PREEN)) - fix_problem(ctx, PR_1_PASS_HEADER, &pctx); - if (ext2fs_has_feature_dir_index(fs->super) && !(ctx->options & E2F_OPT_NO)) { if (ext2fs_u32_list_create(&ctx->dirs_to_hash, 50)) @@ -1266,24 +1372,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) ctx->flags |= E2F_FLAG_ABORT; return; } - pctx.errcode = e2fsck_allocate_subcluster_bitmap(fs, - _("in-use block map"), EXT2FS_BMAP64_RBTREE, - "block_found_map", &ctx->block_found_map); - if (pctx.errcode) { - pctx.num = 1; - fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); - ctx->flags |= E2F_FLAG_ABORT; - return; - } - pctx.errcode = e2fsck_allocate_block_bitmap(fs, - _("metadata block map"), EXT2FS_BMAP64_RBTREE, - "block_metadata_map", &ctx->block_metadata_map); - if (pctx.errcode) { - pctx.num = 1; - fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); - ctx->flags |= E2F_FLAG_ABORT; - return; - } pctx.errcode = e2fsck_setup_icount(ctx, "inode_link_info", 0, NULL, &ctx->inode_link_info); if (pctx.errcode) { @@ -1325,14 +1413,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) } } - mark_table_blocks(ctx); - pctx.errcode = ext2fs_convert_subcluster_bitmap(fs, - &ctx->block_found_map); - if (pctx.errcode) { - fix_problem(ctx, PR_1_CONVERT_SUBCLUSTER, &pctx); - ctx->flags |= E2F_FLAG_ABORT; - goto endit; - } block_buf = (char *) e2fsck_allocate_memory(ctx, fs->blocksize * 3, "block interate buffer"); if (EXT2_INODE_SIZE(fs->super) == EXT2_GOOD_OLD_INODE_SIZE) @@ -1366,12 +1446,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) fs->super->s_mkfs_time < fs->super->s_inodes_count)) low_dtime_check = 0; - if (ext2fs_has_feature_mmp(fs->super) && - fs->super->s_mmp_block > fs->super->s_first_data_block && - fs->super->s_mmp_block < ext2fs_blocks_count(fs->super)) - ext2fs_mark_block_bitmap2(ctx->block_found_map, - fs->super->s_mmp_block); - /* Set up ctx->lost_and_found if possible */ (void) e2fsck_get_lost_and_found(ctx, 0); @@ -1701,8 +1775,10 @@ void e2fsck_pass1_run(e2fsck_t ctx) failed_csum = 0; } + e2fsck_pass1_block_map_lock(ctx); pctx.errcode = ext2fs_copy_bitmap(ctx->block_found_map, &pb.fs_meta_blocks); + e2fsck_pass1_block_map_unlock(ctx); if (pctx.errcode) { pctx.num = 4; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); @@ -2018,9 +2094,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) ext2fs_close_inode_scan(scan); scan = NULL; - reserve_block_for_root_repair(ctx); - reserve_block_for_lnf_repair(ctx); - /* * If any extended attribute blocks' reference counts need to * be adjusted, either up (ctx->refcount_extra), or down @@ -2048,11 +2121,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) ctx->ea_block_quota_inodes = 0; } - if (ctx->invalid_bitmaps) { - e2fsck_pass1_fix_lock(ctx); - handle_fs_bad_blocks(ctx); - e2fsck_pass1_fix_unlock(ctx); - } /* We don't need the block_ea_map any more */ if (ctx->block_ea_map) { @@ -2063,29 +2131,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) /* We don't need the encryption policy => ID map any more */ destroy_encryption_policy_map(ctx); - if (ctx->flags & E2F_FLAG_RESIZE_INODE) { - clear_problem_context(&pctx); - e2fsck_pass1_fix_lock(ctx); - pctx.errcode = ext2fs_create_resize_inode(fs); - e2fsck_pass1_fix_unlock(ctx); - if (pctx.errcode) { - if (!fix_problem(ctx, PR_1_RESIZE_INODE_CREATE, - &pctx)) { - ctx->flags |= E2F_FLAG_ABORT; - goto endit; - } - pctx.errcode = 0; - } - if (!pctx.errcode) { - e2fsck_read_inode(ctx, EXT2_RESIZE_INO, inode, - "recreate inode"); - inode->i_mtime = ctx->now; - e2fsck_write_inode(ctx, EXT2_RESIZE_INO, inode, - "recreate inode"); - } - ctx->flags &= ~E2F_FLAG_RESIZE_INODE; - } - if (ctx->flags & E2F_FLAG_RESTART) { /* * Only the master copy of the superblock and block @@ -2097,13 +2142,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) goto endit; } - if (ctx->block_dup_map) { - if (ctx->options & E2F_OPT_PREEN) { - clear_problem_context(&pctx); - fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); - } - e2fsck_pass1_dupblocks(ctx, block_buf); - } ctx->flags |= E2F_FLAG_ALLOC_OK; endit: e2fsck_use_inode_shortcuts(ctx, 0); @@ -2358,10 +2396,10 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre assert(global_ctx->inode_reg_map == NULL); assert(global_ctx->inodes_to_rebuild == NULL); - assert(global_ctx->block_found_map == NULL); + assert(global_ctx->block_found_map != NULL); + assert(global_ctx->block_metadata_map != NULL); assert(global_ctx->block_dup_map == NULL); assert(global_ctx->block_ea_map == NULL); - assert(global_ctx->block_metadata_map == NULL); assert(global_ctx->fs->dblist == NULL); retval = ext2fs_get_mem(sizeof(struct e2fsck_struct), &thread_context); @@ -2525,10 +2563,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; - ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; - ext2fs_block_bitmap block_metadata_map = global_ctx->block_metadata_map; ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; ext2_icount_t inode_count = global_ctx->inode_count; ext2_icount_t inode_link_info = global_ctx->inode_link_info; @@ -2571,10 +2607,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->inode_imagic_map = inode_imagic_map; global_ctx->inodes_to_rebuild = inodes_to_rebuild; global_ctx->inode_reg_map = inode_reg_map; - global_ctx->block_found_map = block_found_map; - global_ctx->block_dup_map = block_dup_map; global_ctx->block_ea_map = block_ea_map; - global_ctx->block_metadata_map = block_metadata_map; + global_ctx->block_dup_map = block_dup_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); global_ctx->dx_dir_info = dx_dir_info; @@ -2670,26 +2704,11 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx &global_ctx->inodes_to_rebuild); if (retval) return retval; - retval = e2fsck_pass1_merge_bitmap(global_fs, - &thread_ctx->block_found_map, - &global_ctx->block_found_map); - if (retval) - return retval; - retval = e2fsck_pass1_merge_bitmap(global_fs, - &thread_ctx->block_dup_map, - &global_ctx->block_dup_map); - if (retval) - return retval; retval = e2fsck_pass1_merge_bitmap(global_fs, &thread_ctx->block_ea_map, &global_ctx->block_ea_map); if (retval) return retval; - retval = e2fsck_pass1_merge_bitmap(global_fs, - &thread_ctx->block_metadata_map, - &global_ctx->block_metadata_map); - if (retval) - return retval; return 0; } @@ -2713,10 +2732,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) e2fsck_pass1_free_bitmap(&thread_ctx->inode_imagic_map); e2fsck_pass1_free_bitmap(&thread_ctx->inode_reg_map); e2fsck_pass1_free_bitmap(&thread_ctx->inodes_to_rebuild); - e2fsck_pass1_free_bitmap(&thread_ctx->block_found_map); - e2fsck_pass1_free_bitmap(&thread_ctx->block_dup_map); e2fsck_pass1_free_bitmap(&thread_ctx->block_ea_map); - e2fsck_pass1_free_bitmap(&thread_ctx->block_metadata_map); e2fsck_free_dir_info(thread_ctx); ext2fs_free_icount(thread_ctx->inode_count); ext2fs_free_icount(thread_ctx->inode_link_info); @@ -2897,6 +2913,7 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) errcode_t retval; pthread_mutex_init(&global_ctx->fs_fix_mutex, NULL); + pthread_mutex_init(&global_ctx->fs_block_map_mutex, NULL); retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); if (retval) { com_err(global_ctx->program_name, retval, @@ -2948,17 +2965,25 @@ static int multiple_threads_supported(e2fsck_t ctx) void e2fsck_pass1(e2fsck_t ctx) { + errcode_t retval; + int multiple = 0; init_ext2_max_sizes(); + retval = e2fsck_pass1_prepare(ctx); + if (retval) + return; #ifdef CONFIG_PFSCK if (multiple_threads_supported(ctx)) { + multiple = 1; e2fsck_pass1_multithread(ctx); - return; + } else { + fprintf(stderr, "Fall through single thread for pass1 " + "because tdb could not handle properly\n"); } - fprintf(stderr, "Fall through single thread for pass1 " - "because tdb could not handle properly\n"); #endif - e2fsck_pass1_run(ctx); + if (!multiple) + e2fsck_pass1_run(ctx); + e2fsck_pass1_post(ctx); } #undef FINISH_INODE_LOOP @@ -3147,9 +3172,14 @@ static void alloc_imagic_map(e2fsck_t ctx) * WARNING: Assumes checks have already been done to make sure block * is valid. This is true in both process_block and process_bad_block. */ -static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) +static _INLINE_ void mark_block_used_unlocked(e2fsck_t ctx, blk64_t block) { - struct problem_context pctx; + struct problem_context pctx; + e2fsck_t global_ctx; + + global_ctx = ctx->global_ctx; + if (!global_ctx) + global_ctx = ctx; clear_problem_context(&pctx); @@ -3158,11 +3188,15 @@ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) !(ctx->options & E2F_OPT_UNSHARE_BLOCKS)) { return; } - if (!ctx->block_dup_map) { + /** + * this should be safe because this operation has + * been serialized by mutex. + */ + if (!global_ctx->block_dup_map) { pctx.errcode = e2fsck_allocate_block_bitmap(ctx->fs, _("multiply claimed block map"), EXT2FS_BMAP64_RBTREE, "block_dup_map", - &ctx->block_dup_map); + &global_ctx->block_dup_map); if (pctx.errcode) { pctx.num = 3; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, @@ -3172,12 +3206,20 @@ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) return; } } - ext2fs_fast_mark_block_bitmap2(ctx->block_dup_map, block); + ext2fs_fast_mark_block_bitmap2(global_ctx->block_dup_map, block); } else { ext2fs_fast_mark_block_bitmap2(ctx->block_found_map, block); } } +static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) +{ + e2fsck_pass1_block_map_lock(ctx); + mark_block_used_unlocked(ctx, block); + e2fsck_pass1_block_map_unlock(ctx); + +} + /* * When cluster size is greater than one block, it is caller's responsibility * to make sure block parameter starts at a cluster boundary. @@ -3185,14 +3227,16 @@ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) static _INLINE_ void mark_blocks_used(e2fsck_t ctx, blk64_t block, unsigned int num) { - if (ext2fs_test_block_bitmap_range2(ctx->block_found_map, block, num)) + e2fsck_pass1_block_map_lock(ctx); + if (ext2fs_test_block_bitmap_range2(ctx->block_found_map, block, num)) { ext2fs_mark_block_bitmap_range2(ctx->block_found_map, block, num); - else { + } else { unsigned int i; for (i = 0; i < num; i += EXT2FS_CLUSTER_RATIO(ctx->fs)) - mark_block_used(ctx, block + i); + mark_block_used_unlocked(ctx, block + i); } + e2fsck_pass1_block_map_unlock(ctx); } static errcode_t _INLINE_ e2fsck_write_ext_attr3(e2fsck_t ctx, blk64_t block, @@ -4759,10 +4803,12 @@ static int process_bad_block(ext2_filsys fs, } if (blockcnt < 0) { + e2fsck_pass1_block_map_lock(ctx); if (ext2fs_test_block_bitmap2(p->fs_meta_blocks, blk)) { p->bbcheck = 1; if (fix_problem(ctx, PR_1_BB_FS_BLOCK, pctx)) { *block_nr = 0; + e2fsck_pass1_block_map_unlock(ctx); return BLOCK_CHANGED; } } else if (ext2fs_test_block_bitmap2(ctx->block_found_map, @@ -4771,12 +4817,17 @@ static int process_bad_block(ext2_filsys fs, if (fix_problem(ctx, PR_1_BBINODE_BAD_METABLOCK, pctx)) { *block_nr = 0; + e2fsck_pass1_block_map_unlock(ctx); return BLOCK_CHANGED; } - if (e2fsck_should_abort(ctx)) + if (e2fsck_should_abort(ctx)) { + e2fsck_pass1_block_map_unlock(ctx); return BLOCK_ABORT; - } else - mark_block_used(ctx, blk); + } + } else { + mark_block_used_unlocked(ctx, blk); + } + e2fsck_pass1_block_map_unlock(ctx); return 0; } #if 0 @@ -4789,10 +4840,13 @@ static int process_bad_block(ext2_filsys fs, * there's an overlap between the filesystem table blocks * (bitmaps and inode table) and the bad block list. */ + e2fsck_pass1_block_map_lock(ctx); if (!ext2fs_test_block_bitmap2(ctx->block_found_map, blk)) { ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); + e2fsck_pass1_block_map_unlock(ctx); return 0; } + e2fsck_pass1_block_map_unlock(ctx); /* * Try to find the where the filesystem block was used... */ @@ -4947,6 +5001,7 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group, fix_problem(ctx, (old_block ? PR_1_RELOC_FROM_TO : PR_1_RELOC_TO), &pctx); pctx.blk2 = 0; + e2fsck_pass1_block_map_lock(ctx); for (i = 0; i < num; i++) { pctx.blk = i; ext2fs_mark_block_bitmap2(ctx->block_found_map, (*new_block)+i); @@ -4967,6 +5022,7 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group, if (pctx.errcode) fix_problem(ctx, PR_1_RELOC_WRITE_ERR, &pctx); } + e2fsck_pass1_block_map_unlock(ctx); ext2fs_free_mem(&buf); } diff --git a/e2fsck/util.c b/e2fsck/util.c index 8eec477c..3dcfa86a 100644 --- a/e2fsck/util.c +++ b/e2fsck/util.c @@ -570,23 +570,34 @@ void e2fsck_read_inode_full(e2fsck_t ctx, unsigned long ino, } #ifdef CONFIG_PFSCK +#define e2fsck_get_lock_context(ctx) \ + e2fsck_t global_ctx = ctx->global_ctx; \ + if (!global_ctx) \ + global_ctx = ctx; \ + void e2fsck_pass1_fix_lock(e2fsck_t ctx) { - e2fsck_t global_ctx = ctx->global_ctx; - if (!global_ctx) - global_ctx = ctx; - + e2fsck_get_lock_context(ctx); pthread_mutex_lock(&global_ctx->fs_fix_mutex); } void e2fsck_pass1_fix_unlock(e2fsck_t ctx) { - e2fsck_t global_ctx = ctx->global_ctx; - if (!global_ctx) - global_ctx = ctx; - + e2fsck_get_lock_context(ctx); pthread_mutex_unlock(&global_ctx->fs_fix_mutex); } + +void e2fsck_pass1_block_map_lock(e2fsck_t ctx) +{ + e2fsck_get_lock_context(ctx); + pthread_mutex_lock(&global_ctx->fs_block_map_mutex); +} + +void e2fsck_pass1_block_map_unlock(e2fsck_t ctx) +{ + e2fsck_get_lock_context(ctx); + pthread_mutex_unlock(&global_ctx->fs_block_map_mutex); +} #else void e2fsck_pass1_fix_lock(e2fsck_t ctx) { @@ -596,6 +607,15 @@ void e2fsck_pass1_fix_lock(e2fsck_t ctx) void e2fsck_pass1_fix_unlock(e2fsck_t ctx) { +} + +void e2fsck_pass1_block_map_lock(e2fsck_t ctx) +{ + +} + +void e2fsck_pass1_block_map_unlock(e2fsck_t ctx) +{ } #endif diff --git a/tests/f_multithread/expect.1 b/tests/f_multithread/expect.1 index 8d2acd2b..4db68d9e 100644 --- a/tests/f_multithread/expect.1 +++ b/tests/f_multithread/expect.1 @@ -1,7 +1,7 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) -[Thread 0] Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity diff --git a/tests/f_multithread_logfile/expect.1 b/tests/f_multithread_logfile/expect.1 index 8d2acd2b..4db68d9e 100644 --- a/tests/f_multithread_logfile/expect.1 +++ b/tests/f_multithread_logfile/expect.1 @@ -1,7 +1,7 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) -[Thread 0] Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity diff --git a/tests/f_multithread_no/expect.1 b/tests/f_multithread_no/expect.1 index f85a3382..eda2fcac 100644 --- a/tests/f_multithread_no/expect.1 +++ b/tests/f_multithread_no/expect.1 @@ -1,7 +1,7 @@ ext2fs_open2: Bad magic number in super-block ../e2fsck/e2fsck: Superblock invalid, trying backup blocks... +Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 2) -[Thread 0] Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scanned group range [0, 2), inodes 3008 Pass 2: Checking directory structure Pass 3: Checking directory connectivity From patchwork Wed Nov 18 15:39:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402338 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=VGT6lZby; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn976dgxz9sVq for ; Thu, 19 Nov 2020 02:41:31 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727533AbgKRPl0 (ORCPT ); Wed, 18 Nov 2020 10:41:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727443AbgKRPl0 (ORCPT ); Wed, 18 Nov 2020 10:41:26 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 788A4C0613D4 for ; Wed, 18 Nov 2020 07:41:24 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id w8so2922397ybj.14 for ; Wed, 18 Nov 2020 07:41:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ah/XmUUyjFZp1jiQfzR6JQ8mjXk0RTYLoGUqXyyD6vI=; b=VGT6lZbySZI/nnLx13YxQ9osEpSJrRfznaD8bDKv9FNHt+HiNKp+mDwqXNNN/tz/HA rzL01A51MF12D/B/WiI0D8WuBdERWveSe0Z0rKJJQ8KqGNw7cIiFGXWPH/mdw+0a85YY W+zAewQ4PVSuDp7Y5tgUVrnvI9zYzgsvbPwy7d63vf56kbc5yRmp2zNMb4LrLnQHmMX9 VPzlLwaPu3iGnNojzvQR6P/0io092IHCNE9XFVrGskT30fbd0vcHFZ+ppK9bGQ4W19ug 3F9bS096KtxGQHS6MAi6EvWZinB6pNU0BgtDQgiLiix2dhiuCad3Z1oviTojrQRHsaYo BNQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ah/XmUUyjFZp1jiQfzR6JQ8mjXk0RTYLoGUqXyyD6vI=; b=MWc8rxngWAunnRpDapmG571TUYj7glcscqR/W0YL08EJWXjdNxINpfvYF8GmmOROs5 OVt15wShNr7HpW6QL60MmnVuQPxggo3aTlZpki6Lo08X7+MZ5Qa7KstRUnReKLqhtlcn 8WcGB9fufzAiIC1TgRHejRGrGlgIixaZpTcIzlAXNHqKCaWEJO+ecftvJ1GOUeEIpYKC wGVy5KQpOJZ2oNcpTqlVMrpk6142YRwAsMJuikI+0NicJP6c5MDiyflOg5vTfmhWSyoM YJnyeeuBX+JvmSplaBiuMmmA5XC7aYrOZyRNVOshpEBl6OS833noqRuY21Bmxg5/A2dR jwlw== X-Gm-Message-State: AOAM531jRGcVq5KDekXmU31/YKDbFL1O0rvfOuXp1djpeJYcs/MAGamF 62CLYzBLg9sWPAVU/YsRfcddFaPZeOI5cryJPN33xi4JoffkmQyTaJzq/Sx2bG0wSt46fYEtPI0 999UssgD/WZ6HwyYJY2tc6Po2QWFawwzeUNrw2R4L0+/kWyllb2lDz/Dyu4Yrj4PVbSKuVqAlIs 8iX59F17M= X-Google-Smtp-Source: ABdhPJy0at4wlUgRbMtvX6WvkzouiiJ07TfPe5XiFNMKspgV5vGqGCPJP65eXGm6JoCjX9r1Yoi1coFzy20U6bzCbgk= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:3614:: with SMTP id d20mr2751607yba.415.1605714083648; Wed, 18 Nov 2020 07:41:23 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:17 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-32-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 31/61] e2fsck: split and merge invalid bitmaps From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Invalid bitmaps are splitted per thread, and we should merge them after thread finish. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 49bdba21..29954e88 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2379,6 +2379,62 @@ out: return retval; } +static void e2fsck_pass1_copy_invalid_bitmaps(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + dgrp_t i, j; + dgrp_t grp_start = thread_ctx->thread_info.et_group_start; + dgrp_t grp_end = thread_ctx->thread_info.et_group_end; + dgrp_t total = grp_end - grp_start; + + thread_ctx->invalid_inode_bitmap_flag = + e2fsck_allocate_memory(global_ctx, sizeof(int) * total, + "invalid_inode_bitmap"); + thread_ctx->invalid_block_bitmap_flag = + e2fsck_allocate_memory(global_ctx, sizeof(int) * total, + "invalid_block_bitmap"); + thread_ctx->invalid_inode_table_flag = + e2fsck_allocate_memory(global_ctx, sizeof(int) * total, + "invalid_inode_table"); + + memcpy(thread_ctx->invalid_block_bitmap_flag, + &global_ctx->invalid_block_bitmap_flag[grp_start], + total * sizeof(int)); + memcpy(thread_ctx->invalid_inode_bitmap_flag, + &global_ctx->invalid_inode_bitmap_flag[grp_start], + total * sizeof(int)); + memcpy(thread_ctx->invalid_inode_table_flag, + &global_ctx->invalid_inode_table_flag[grp_start], + total * sizeof(int)); + + thread_ctx->invalid_bitmaps = 0; + for (i = grp_start, j = 0; i < grp_end; i++, j++) { + if (thread_ctx->invalid_block_bitmap_flag[j]) + thread_ctx->invalid_bitmaps++; + if (thread_ctx->invalid_inode_bitmap_flag[j]) + thread_ctx->invalid_bitmaps++; + if (thread_ctx->invalid_inode_table_flag[j]) + thread_ctx->invalid_bitmaps++; + } +} + +static void e2fsck_pass1_merge_invalid_bitmaps(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + dgrp_t i, j; + dgrp_t grp_start = thread_ctx->thread_info.et_group_start; + dgrp_t grp_end = thread_ctx->thread_info.et_group_end; + dgrp_t total = grp_end - grp_start; + + memcpy(&global_ctx->invalid_block_bitmap_flag[grp_start], + thread_ctx->invalid_block_bitmap_flag, total * sizeof(int)); + memcpy(&global_ctx->invalid_inode_bitmap_flag[grp_start], + thread_ctx->invalid_inode_bitmap_flag, total * sizeof(int)); + memcpy(&global_ctx->invalid_inode_table_flag[grp_start], + thread_ctx->invalid_inode_table_flag, total * sizeof(int)); + global_ctx->invalid_bitmaps += thread_ctx->invalid_bitmaps; +} + static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx, int thread_index, int num_threads) { @@ -2455,6 +2511,7 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre goto out_fs; } *thread_ctx = thread_context; + e2fsck_pass1_copy_invalid_bitmaps(global_ctx, thread_context); return 0; out_fs: ext2fs_free_mem(&thread_fs); @@ -2589,6 +2646,10 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_ino_t dx_dir_info_count = global_ctx->dx_dir_info_count; ext2_u32_list dirs_to_hash = global_ctx->dirs_to_hash; quota_ctx_t qctx = global_ctx->qctx; + int *invalid_block_bitmap_flag = global_ctx->invalid_block_bitmap_flag; + int *invalid_inode_bitmap_flag = global_ctx->invalid_inode_bitmap_flag; + int *invalid_inode_table_flag = global_ctx->invalid_inode_table_flag; + int invalid_bitmaps = global_ctx->invalid_bitmaps; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2667,6 +2728,11 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx thread_ctx->qctx); if (retval) return retval; + global_ctx->invalid_block_bitmap_flag = invalid_block_bitmap_flag; + global_ctx->invalid_inode_bitmap_flag = invalid_inode_bitmap_flag; + global_ctx->invalid_inode_table_flag = invalid_inode_table_flag; + global_ctx->invalid_bitmaps = invalid_bitmaps; + e2fsck_pass1_merge_invalid_bitmaps(global_ctx, thread_ctx); retval = e2fsck_pass1_merge_bitmap(global_fs, &thread_ctx->inode_used_map, @@ -2739,6 +2805,9 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) if (thread_ctx->dirs_to_hash) ext2fs_badblocks_list_free(thread_ctx->dirs_to_hash); quota_release_context(&thread_ctx->qctx); + ext2fs_free_mem(&thread_ctx->invalid_block_bitmap_flag); + ext2fs_free_mem(&thread_ctx->invalid_inode_bitmap_flag); + ext2fs_free_mem(&thread_ctx->invalid_inode_table_flag); ext2fs_free_mem(&thread_ctx); return retval; @@ -2752,6 +2821,8 @@ static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, int i; struct e2fsck_thread_info *pinfo; + /* merge invalid bitmaps will recalculate it */ + global_ctx->invalid_bitmaps = 0; for (i = 0; i < num_threads; i++) { pinfo = &infos[i]; From patchwork Wed Nov 18 15:39:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402339 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=AO09Ptta; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn982674z9sW4 for ; Thu, 19 Nov 2020 02:41:32 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727534AbgKRPl2 (ORCPT ); Wed, 18 Nov 2020 10:41:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727443AbgKRPl1 (ORCPT ); Wed, 18 Nov 2020 10:41:27 -0500 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CE20C0613D4 for ; Wed, 18 Nov 2020 07:41:26 -0800 (PST) Received: by mail-qk1-x749.google.com with SMTP id q25so1749056qkm.17 for ; Wed, 18 Nov 2020 07:41:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ZIUX778P7BRRnxlKsN/61V0caaEGkj/6yu/1n19ZbCw=; b=AO09PttaA6ki8Sf81OwZjjVWhsFbfVvZj2EgcyyIAY8St6PcViI/PU20Ear+nNxpQk jPGCDxAmXpbTYfbhbHNJfnExBJPHRSnbpxIUIlopwNKmap4i15t7VVZUOsTBP/5smCnQ 68knrJMam7fDcQsG6WeINV37MPDExfjBvRNTgUsENBE8x3LvHpHYXDH1qNspGDcZpzW1 D7slzxzo+Ebd4JeR3s0eUdXsgi+D7cFVptUe2H47JT2A42iDt06n8yZoRo7KY7lr/SIw b+DH+/2htP3xkCI9zHih9EtEA3d0bF82S/wDcZ1ctPBXnzA4wmkC0zq/TvXZPQnnja0Z puzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ZIUX778P7BRRnxlKsN/61V0caaEGkj/6yu/1n19ZbCw=; b=NGmqUNYwTnkKO3+x3ebBfE9p7xbFio7TGI1gCCssdGr0XNee1NN0IglHYsfqLsuqJ5 TnG2HIL7wZVhp9NLCefEnc3IRFMDbmhuXU3KZd8PofLHv2fM8BuyWRQ2h+34duzg+G8L QiSA88MGK2xqIr2iNIKsVtsN8yiZMharrlIvS39Y2c0RAaGv/oL+1C9TBsobA0RY+9Ef 7+t7QR+bOV1szszx9fAdomfCNA4MKSkxsCk1t65qq+RFSL87H4ow6HGFddu4koS8FUGu Oz+rOHNHvhVMflf4uKvxqBMPtBBL/yzMH65yQ5fbJ4uTYzdKwvt16/YBAEsWPvpmgjUS ZkdQ== X-Gm-Message-State: AOAM531JlKo+gKxEdT1mcbVVDQ8BreIyEhr9pw2zVoYWbd8rvawsPnAe Kyv9tBHcE3RiQrEPefUXu3Em1rO2/G7ciQt3mZHqlvPROqPoSaEul2nwsUK4mYCqWDNA+zsOhte Z4Nx0Ge8jwZj1uFmeIYdEesg51FMUmU2lMhsGf/sUqve7pYxFq6U8MAmtjo3Q3umY+g6Fj5mXNI 4eDO7psh4= X-Google-Smtp-Source: ABdhPJycKQTQKSCvsWwDXuo8iS7VXEbY63r/zj5YZ2R6Q6FZLq3gxsSWKWCjPayLbAq4DduntRD31n8oCbGKnrOTFG4= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:e88c:: with SMTP id b12mr4984612qvo.42.1605714085381; Wed, 18 Nov 2020 07:41:25 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:18 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-33-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 32/61] e2fsck: merge EA blocks properly From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong EA blocks might be shared, merge them carefully. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 1 + e2fsck/pass1.c | 245 +++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 213 insertions(+), 33 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index fecc8bbf..192a534c 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -299,6 +299,7 @@ struct e2fsck_struct { ext2_refcount_t refcount; ext2_refcount_t refcount_extra; + ext2_refcount_t refcount_orig; /* * Quota blocks and inodes to be charged for each ea block. diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 29954e88..8b03b6f9 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -677,14 +677,14 @@ static void check_is_really_dir(e2fsck_t ctx, struct problem_context *pctx, LINUX_S_ISLNK(inode->i_mode) || inode->i_block[0] == 0) return; - /* + /* * Check the block numbers in the i_block array for validity: * zero blocks are skipped (but the first one cannot be zero - * see above), other blocks are checked against the first and * max data blocks (from the the superblock) and against the * block bitmap. Any invalid block found means this cannot be * a directory. - * + * * If there are non-zero blocks past the fourth entry, then * this cannot be a device file: we remember that for the next * check. @@ -1229,14 +1229,39 @@ static void e2fsck_pass1_post(e2fsck_t ctx) { struct problem_context pctx; ext2_filsys fs = ctx->fs; - char *block_buf; + char *block_buf = + (char *)e2fsck_allocate_memory(ctx, ctx->fs->blocksize * 3, + "block interate buffer"); reserve_block_for_root_repair(ctx); reserve_block_for_lnf_repair(ctx); + /* + * If any extended attribute blocks' reference counts need to + * be adjusted, either up (ctx->refcount_extra), or down + * (ctx->refcount), then fix them. + */ + if (ctx->refcount) { + adjust_extattr_refcount(ctx, ctx->refcount, block_buf, -1); + ea_refcount_free(ctx->refcount); + ctx->refcount = 0; + } + if (ctx->refcount_extra) { + adjust_extattr_refcount(ctx, ctx->refcount_extra, + block_buf, +1); + ea_refcount_free(ctx->refcount_extra); + ctx->refcount_extra = 0; + } + if (ctx->invalid_bitmaps) handle_fs_bad_blocks(ctx); + /* We don't need the block_ea_map any more */ + if (ctx->block_ea_map) { + ext2fs_free_block_bitmap(ctx->block_ea_map); + ctx->block_ea_map = 0; + } + if (ctx->flags & E2F_FLAG_RESIZE_INODE) { struct ext2_inode *inode; int inode_size = EXT2_INODE_SIZE(fs->super); @@ -1276,10 +1301,6 @@ static void e2fsck_pass1_post(e2fsck_t ctx) clear_problem_context(&pctx); fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); } - block_buf = - (char *)e2fsck_allocate_memory(ctx, - ctx->fs->blocksize * 3, - "block interate buffer"); e2fsck_pass1_dupblocks(ctx, block_buf); ext2fs_free_mem(&block_buf); } @@ -2094,23 +2115,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) ext2fs_close_inode_scan(scan); scan = NULL; - /* - * If any extended attribute blocks' reference counts need to - * be adjusted, either up (ctx->refcount_extra), or down - * (ctx->refcount), then fix them. - */ - if (ctx->refcount) { - adjust_extattr_refcount(ctx, ctx->refcount, block_buf, -1); - ea_refcount_free(ctx->refcount); - ctx->refcount = 0; - } - if (ctx->refcount_extra) { - adjust_extattr_refcount(ctx, ctx->refcount_extra, - block_buf, +1); - ea_refcount_free(ctx->refcount_extra); - ctx->refcount_extra = 0; - } - if (ctx->ea_block_quota_blocks) { ea_refcount_free(ctx->ea_block_quota_blocks); ctx->ea_block_quota_blocks = 0; @@ -2121,13 +2125,6 @@ void e2fsck_pass1_run(e2fsck_t ctx) ctx->ea_block_quota_inodes = 0; } - - /* We don't need the block_ea_map any more */ - if (ctx->block_ea_map) { - ext2fs_free_block_bitmap(ctx->block_ea_map); - ctx->block_ea_map = 0; - } - /* We don't need the encryption policy => ID map any more */ destroy_encryption_policy_map(ctx); @@ -2604,6 +2601,156 @@ static errcode_t e2fsck_pass1_merge_dirs_to_hash(e2fsck_t global_ctx, return retval; } +static errcode_t e2fsck_pass1_merge_ea_inode_refs(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + ea_value_t count; + blk64_t blk; + errcode_t retval; + + if (!thread_ctx->ea_inode_refs) + return 0; + + if (!global_ctx->ea_inode_refs) { + global_ctx->ea_inode_refs = thread_ctx->ea_inode_refs; + thread_ctx->ea_inode_refs = NULL; + return 0; + } + + ea_refcount_intr_begin(thread_ctx->ea_inode_refs); + while (1) { + if ((blk = ea_refcount_intr_next(thread_ctx->ea_inode_refs, + &count)) == 0) + break; + if (!global_ctx->block_ea_map || + !ext2fs_fast_test_block_bitmap2(global_ctx->block_ea_map, + blk)) { + retval = ea_refcount_store(global_ctx->ea_inode_refs, + blk, count); + if (retval) + return retval; + } + } + + return retval; +} + +static ea_value_t ea_refcount_usage(e2fsck_t ctx, blk64_t blk, + ea_value_t *orig) +{ + ea_value_t count_cur; + ea_value_t count_extra = 0; + ea_value_t count_orig; + + ea_refcount_fetch(ctx->refcount_orig, blk, &count_orig); + ea_refcount_fetch(ctx->refcount, blk, &count_cur); + /* most of time this is not needed */ + if (ctx->refcount_extra && count_cur == 0) + ea_refcount_fetch(ctx->refcount_extra, blk, &count_extra); + + if (!count_orig) + count_orig = *orig; + else if (orig) + *orig = count_orig; + + return count_orig + count_extra - count_cur; +} + +static errcode_t e2fsck_pass1_merge_ea_refcount(e2fsck_t global_ctx, + e2fsck_t thread_ctx) +{ + ea_value_t count; + blk64_t blk; + errcode_t retval = 0; + + if (!thread_ctx->refcount) + return 0; + + if (!global_ctx->refcount) { + global_ctx->refcount = thread_ctx->refcount; + thread_ctx->refcount = NULL; + global_ctx->refcount_extra = thread_ctx->refcount; + thread_ctx->refcount_extra = NULL; + return 0; + } + + ea_refcount_intr_begin(thread_ctx->refcount); + while (1) { + if ((blk = ea_refcount_intr_next(thread_ctx->refcount, + &count)) == 0) + break; + /** + * this EA has never seen before, so just store its + * refcount and refcount_extra into global_ctx if needed. + */ + if (!global_ctx->block_ea_map || + !ext2fs_fast_test_block_bitmap2(global_ctx->block_ea_map, + blk)) { + ea_value_t extra; + + retval = ea_refcount_store(global_ctx->refcount, + blk, count); + if (retval) + return retval; + + if (count > 0 || !thread_ctx->refcount_extra) + continue; + ea_refcount_fetch(thread_ctx->refcount_extra, blk, + &extra); + if (extra == 0) + continue; + + if (!global_ctx->refcount_extra) { + retval = ea_refcount_create(0, + &global_ctx->refcount_extra); + if (retval) + return retval; + } + retval = ea_refcount_store(global_ctx->refcount_extra, + blk, extra); + if (retval) + return retval; + } else { + ea_value_t orig; + ea_value_t thread_usage; + ea_value_t global_usage; + ea_value_t new; + + thread_usage = ea_refcount_usage(thread_ctx, + blk, &orig); + global_usage = ea_refcount_usage(global_ctx, + blk, &orig); + if (thread_usage + global_usage <= orig) { + new = orig - thread_usage - global_usage; + retval = ea_refcount_store(global_ctx->refcount, + blk, new); + if (retval) + return retval; + continue; + } + /* update it is as zero */ + retval = ea_refcount_store(global_ctx->refcount, + blk, 0); + if (retval) + return retval; + /* Ooops, this EA was referenced more than it stated */ + if (!global_ctx->refcount_extra) { + retval = ea_refcount_create(0, + &global_ctx->refcount_extra); + if (retval) + return retval; + } + new = global_usage + thread_usage - orig; + retval = ea_refcount_store(global_ctx->refcount_extra, + blk, new); + if (retval) + return retval; + } + } + + return retval; +} + static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; @@ -2621,7 +2768,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; - ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; ext2_icount_t inode_count = global_ctx->inode_count; ext2_icount_t inode_link_info = global_ctx->inode_link_info; @@ -2650,6 +2796,13 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx int *invalid_inode_bitmap_flag = global_ctx->invalid_inode_bitmap_flag; int *invalid_inode_table_flag = global_ctx->invalid_inode_table_flag; int invalid_bitmaps = global_ctx->invalid_bitmaps; + ext2_refcount_t refcount = global_ctx->refcount; + ext2_refcount_t refcount_extra = global_ctx->refcount_extra; + ext2_refcount_t refcount_orig = global_ctx->refcount_orig; + ext2_refcount_t ea_block_quota_blocks = global_ctx->ea_block_quota_blocks; + ext2_refcount_t ea_block_quota_inodes = global_ctx->ea_block_quota_inodes; + ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; + ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2668,7 +2821,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->inode_imagic_map = inode_imagic_map; global_ctx->inodes_to_rebuild = inodes_to_rebuild; global_ctx->inode_reg_map = inode_reg_map; - global_ctx->block_ea_map = block_ea_map; global_ctx->block_dup_map = block_dup_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); @@ -2678,6 +2830,13 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); global_ctx->inode_count = inode_count; global_ctx->inode_link_info = inode_link_info; + global_ctx->refcount = refcount; + global_ctx->refcount_extra = refcount_extra; + global_ctx->refcount_orig = refcount_orig; + global_ctx->ea_block_quota_blocks = ea_block_quota_blocks; + global_ctx->ea_block_quota_inodes = ea_block_quota_inodes; + global_ctx->block_ea_map = block_ea_map; + global_ctx->ea_inode_refs = ea_inode_refs; global_ctx->fs_directory_count += fs_directory_count; global_ctx->fs_regular_count += fs_regular_count; global_ctx->fs_blockdev_count += fs_blockdev_count; @@ -2723,6 +2882,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx return retval; } + e2fsck_pass1_merge_ea_inode_refs(global_ctx, thread_ctx); + e2fsck_pass1_merge_ea_refcount(global_ctx, thread_ctx); global_ctx->qctx = qctx; retval = quota_merge_and_update_usage(global_ctx->qctx, thread_ctx->qctx); @@ -2799,6 +2960,14 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) e2fsck_pass1_free_bitmap(&thread_ctx->inode_reg_map); e2fsck_pass1_free_bitmap(&thread_ctx->inodes_to_rebuild); e2fsck_pass1_free_bitmap(&thread_ctx->block_ea_map); + if (thread_ctx->refcount) + ea_refcount_free(thread_ctx->refcount); + if (thread_ctx->refcount_extra) + ea_refcount_free(thread_ctx->refcount_extra); + if (thread_ctx->ea_inode_refs) + ea_refcount_free(thread_ctx->ea_inode_refs); + if (thread_ctx->refcount_orig) + ea_refcount_free(thread_ctx->refcount_orig); e2fsck_free_dir_info(thread_ctx); ext2fs_free_icount(thread_ctx->inode_count); ext2fs_free_icount(thread_ctx->inode_link_info); @@ -3425,6 +3594,15 @@ static int check_ext_attr(e2fsck_t ctx, struct problem_context *pctx, /* Create the EA refcount structure if necessary */ if (!ctx->refcount) { + pctx->errcode = ea_refcount_create(0, + &ctx->refcount_orig); + if (pctx->errcode) { + pctx->num = 1; + fix_problem(ctx, PR_1_ALLOCATE_REFCOUNT, pctx); + ctx->flags |= E2F_FLAG_ABORT; + return 0; + } + pctx->errcode = ea_refcount_create(0, &ctx->refcount); if (pctx->errcode) { pctx->num = 1; @@ -3629,6 +3807,7 @@ refcount_fail: inc_ea_inode_refs(ctx, pctx, first, end); ea_refcount_store(ctx->refcount, blk, header->h_refcount - 1); + ea_refcount_store(ctx->refcount_orig, blk, header->h_refcount); mark_block_used(ctx, blk); ext2fs_fast_mark_block_bitmap2(ctx->block_ea_map, blk); return 1; From patchwork Wed Nov 18 15:39:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402340 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=bLwWo5Xt; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn985t2Xz9sW8 for ; Thu, 19 Nov 2020 02:41:32 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727543AbgKRPl2 (ORCPT ); Wed, 18 Nov 2020 10:41:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727443AbgKRPl2 (ORCPT ); Wed, 18 Nov 2020 10:41:28 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2478DC0613D4 for ; Wed, 18 Nov 2020 07:41:28 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id g129so2904309ybf.20 for ; Wed, 18 Nov 2020 07:41:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=V1e/3CITZ0z8WAi/fer1yOneLGk2EQ4KLkYZwrwSFxQ=; b=bLwWo5XtTj32vn19PVOsrDefCcEI3LLBhajVvalvz2gO1YuMMJrpsBvoDIkXkn5rm5 Vxwj7vlAu+pnCp/zJqV8eytNLu+C1Hx7Y2VQFs9qPBu5P8GSQiZClZVtEux+ORXK5e7F 5QrZJ6m4tS2qocpD5IBzfR3EcdwhodoiyKHW1aXzUUuthGzJl/SI4D8LNsO90hrJxg1B dn5aCXZXuG7f4GHmljJx5hrz6ANyxRNa68T/LJcCSNmUPtU+wONheAXyTgwctMfFadLH 6JGTedpuH5Y646hXViyjDucl3vuvBsmN/Sq2oF5EW+eTpNkUzJ6+UoVkbK+YEyxZ/FHv k3Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=V1e/3CITZ0z8WAi/fer1yOneLGk2EQ4KLkYZwrwSFxQ=; b=Skn+/q7oK9UTt7FhgFrk2r8DMwt915o+YXJlSEiFXd/SPfH2BqyFmC7HfJSDGHsihY toZCND5DP8QWxRPbn/aJ0V+V9CeZmKYjzntJA3G/yUKYuRSaAkyDl1SdABaxSmwIjYvt BKKBg6945dltjJ20uQlH04HBakr0WRiLLS4dU6EH04N+Xrcd8mmltC4EKG6uDEtYkpSc ufQA70Wk6IdFhlBtuGKzfcobEToqhivjJU11rLWDRmaFKHDZoEaS8nEkNuTA57awiO1P 1RDCRQiS+ZK1dUKpOMeB9Jbzw0ov9ERUbDWRAg1CoDekTvx12vi60MaNRcS+dnddC4lp CBGA== X-Gm-Message-State: AOAM530s0MC6AqBRc+a90kdAWO8TB3/BC+GV/+YZ2llAxa6REcPdgnuf tjf/b5/eZYs3tIFt6bVLwyvkL6lfhQ2urO2OUiKl5bFTgXNWfJ6N738bOnU3GXbLjehVanQdzjL Baix45e1UrsDf3AEJkl+3Do/JRO6IyPnEffmzI5HZ/IraQXYFapai+yeWbChuJbewkx+42dLrlX G1s8O2fds= X-Google-Smtp-Source: ABdhPJxArmqqXf2hy+/yGqmbnTSD2ium0IDyGn0aR8+TsfqxfOWpch8g8q8yIhxNpwg0aY2WbkDfY3qyTgEGWn4VutQ= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:d412:: with SMTP id m18mr6651149ybf.361.1605714087265; Wed, 18 Nov 2020 07:41:27 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:19 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-34-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 33/61] e2fsck: kickoff mutex lock for block found map From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Now @block_found_map is no longer shared by multiple threads, and @block_dup_map need be checked again after threads finish. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 9 +- e2fsck/pass1.c | 172 ++++++++++++++++++------------ e2fsck/util.c | 34 ++++-- lib/ext2fs/bitops.h | 2 + lib/ext2fs/gen_bitmap64.c | 33 ++++++ tests/f_itable_collision/expect.1 | 3 - 6 files changed, 172 insertions(+), 81 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 192a534c..d4b472f5 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -202,6 +202,7 @@ struct resource_track { #define E2F_FLAG_TIME_INSANE 0x2000 /* Time is insane */ #define E2F_FLAG_PROBLEMS_FIXED 0x4000 /* At least one problem was fixed */ #define E2F_FLAG_ALLOC_OK 0x8000 /* Can we allocate blocks? */ +#define E2F_FLAG_DUP_BLOCK 0x20000 /* dup block found during pass1 */ #define E2F_RESET_FLAGS (E2F_FLAG_TIME_INSANE | E2F_FLAG_PROBLEMS_FIXED) @@ -452,7 +453,7 @@ struct e2fsck_struct { /* serialize fix operation for multiple threads */ pthread_mutex_t fs_fix_mutex; /* protect block_found_map, block_dup_map */ - pthread_mutex_t fs_block_map_mutex; + pthread_rwlock_t fs_block_map_rwlock; #endif }; @@ -753,8 +754,10 @@ extern errcode_t e2fsck_allocate_subcluster_bitmap(ext2_filsys fs, unsigned long long get_memory_size(void); extern void e2fsck_pass1_fix_lock(e2fsck_t ctx); extern void e2fsck_pass1_fix_unlock(e2fsck_t ctx); -extern void e2fsck_pass1_block_map_lock(e2fsck_t ctx); -extern void e2fsck_pass1_block_map_unlock(e2fsck_t ctx); +extern void e2fsck_pass1_block_map_w_lock(e2fsck_t ctx); +extern void e2fsck_pass1_block_map_w_unlock(e2fsck_t ctx); +extern void e2fsck_pass1_block_map_r_lock(e2fsck_t ctx); +extern void e2fsck_pass1_block_map_r_unlock(e2fsck_t ctx); /* unix.c */ extern void e2fsck_clear_progbar(e2fsck_t ctx); diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 8b03b6f9..6dba6d1b 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -647,6 +647,31 @@ static void check_inode_extra_space(e2fsck_t ctx, struct problem_context *pctx, } +static _INLINE_ int is_blocks_used(e2fsck_t ctx, blk64_t block, + unsigned int num) +{ + int retval; + + /* used to avoid duplicate output from below */ + retval = ext2fs_test_block_bitmap_range2_valid(ctx->block_found_map, + block, num); + if (!retval) + return 0; + + retval = ext2fs_test_block_bitmap_range2(ctx->block_found_map, block, num); + if (retval) { + e2fsck_pass1_block_map_r_lock(ctx); + if (ctx->global_ctx) + retval = ext2fs_test_block_bitmap_range2( + ctx->global_ctx->block_found_map, block, num); + e2fsck_pass1_block_map_r_unlock(ctx); + if (retval) + return 0; + } + + return 1; +} + /* * Check to see if the inode might really be a directory, despite i_mode * @@ -750,15 +775,10 @@ static void check_is_really_dir(e2fsck_t ctx, struct problem_context *pctx, if (i >= 4) not_device++; - e2fsck_pass1_block_map_lock(ctx); if (blk < ctx->fs->super->s_first_data_block || blk >= ext2fs_blocks_count(ctx->fs->super) || - ext2fs_fast_test_block_bitmap2(ctx->block_found_map, - blk)) { - e2fsck_pass1_block_map_unlock(ctx); + is_blocks_used(ctx, blk, 1)) return; /* Invalid block, can't be dir */ - } - e2fsck_pass1_block_map_unlock(ctx); } blk = inode->i_block[0]; } @@ -1216,11 +1236,28 @@ static errcode_t e2fsck_pass1_prepare(e2fsck_t ctx) return pctx.errcode; } + pctx.errcode = e2fsck_allocate_block_bitmap(ctx->fs, + _("multiply claimed block map"), + EXT2FS_BMAP64_RBTREE, "block_dup_map", + &ctx->block_dup_map); + if (pctx.errcode) { + pctx.num = 3; + fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, + &pctx); + /* Should never get here */ + ctx->flags |= E2F_FLAG_ABORT; + return pctx.errcode; + } + if (ext2fs_has_feature_mmp(fs->super) && fs->super->s_mmp_block > fs->super->s_first_data_block && fs->super->s_mmp_block < ext2fs_blocks_count(fs->super)) ext2fs_mark_block_bitmap2(ctx->block_found_map, fs->super->s_mmp_block); +#ifdef CONFIG_PFSCK + pthread_mutex_init(&ctx->fs_fix_mutex, NULL); + pthread_rwlock_init(&ctx->fs_block_map_rwlock, NULL); +#endif return 0; } @@ -1297,12 +1334,17 @@ static void e2fsck_pass1_post(e2fsck_t ctx) } if (ctx->block_dup_map) { + if (!(ctx->flags & E2F_FLAG_DUP_BLOCK)) { + ext2fs_free_mem(&block_buf); + return; + } if (ctx->options & E2F_OPT_PREEN) { clear_problem_context(&pctx); fix_problem(ctx, PR_1_DUP_BLOCKS_PREENSTOP, &pctx); } e2fsck_pass1_dupblocks(ctx, block_buf); ext2fs_free_mem(&block_buf); + ctx->flags &= ~E2F_FLAG_DUP_BLOCK; } } @@ -1796,10 +1838,11 @@ void e2fsck_pass1_run(e2fsck_t ctx) failed_csum = 0; } - e2fsck_pass1_block_map_lock(ctx); - pctx.errcode = ext2fs_copy_bitmap(ctx->block_found_map, - &pb.fs_meta_blocks); - e2fsck_pass1_block_map_unlock(ctx); + e2fsck_pass1_block_map_r_lock(ctx); + pctx.errcode = ext2fs_copy_bitmap(ctx->global_ctx ? + ctx->global_ctx->block_found_map : + ctx->block_found_map, &pb.fs_meta_blocks); + e2fsck_pass1_block_map_r_unlock(ctx); if (pctx.errcode) { pctx.num = 4; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); @@ -2451,7 +2494,7 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre assert(global_ctx->block_found_map != NULL); assert(global_ctx->block_metadata_map != NULL); - assert(global_ctx->block_dup_map == NULL); + assert(global_ctx->block_dup_map != NULL); assert(global_ctx->block_ea_map == NULL); assert(global_ctx->fs->dblist == NULL); @@ -2461,8 +2504,15 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre return retval; } memcpy(thread_context, global_ctx, sizeof(struct e2fsck_struct)); - thread_context->global_ctx = global_ctx; + thread_context->block_dup_map = NULL; + retval = e2fsck_allocate_block_bitmap(global_ctx->fs, + _("in-use block map"), EXT2FS_BMAP64_RBTREE, + "block_found_map", &thread_context->block_found_map); + if (retval) + goto out_context; + + thread_context->global_ctx = global_ctx; retval = ext2fs_get_mem(sizeof(struct struct_ext2_filsys), &thread_fs); if (retval) { com_err(global_ctx->program_name, retval, "while allocating memory"); @@ -2513,6 +2563,8 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre out_fs: ext2fs_free_mem(&thread_fs); out_context: + if (thread_context->block_found_map) + ext2fs_free_mem(&thread_context->block_found_map); ext2fs_free_mem(&thread_context); return retval; } @@ -2767,7 +2819,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; - ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; ext2_icount_t inode_count = global_ctx->inode_count; ext2_icount_t inode_link_info = global_ctx->inode_link_info; @@ -2803,6 +2854,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_refcount_t ea_block_quota_inodes = global_ctx->ea_block_quota_inodes; ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; + ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; + ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2822,6 +2875,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->inodes_to_rebuild = inodes_to_rebuild; global_ctx->inode_reg_map = inode_reg_map; global_ctx->block_dup_map = block_dup_map; + global_ctx->block_found_map = block_found_map; global_ctx->dir_info = dir_info; e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); global_ctx->dx_dir_info = dx_dir_info; @@ -2937,6 +2991,23 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx if (retval) return retval; + if (ext2fs_has_feature_shared_blocks(global_fs->super) && + !(global_ctx->options & E2F_OPT_UNSHARE_BLOCKS)) + return 0; + /* + * This need be done after merging block_ea_map + * because ea block might be shared, we need exclude + * them from dup blocks. + */ + e2fsck_pass1_block_map_w_lock(thread_ctx); + retval = ext2fs_merge_bitmap(thread_ctx->block_found_map, + global_ctx->block_found_map, + global_ctx->block_dup_map, + global_ctx->block_ea_map); + e2fsck_pass1_block_map_w_unlock(thread_ctx); + if (retval == EEXIST) + global_ctx->flags |= E2F_FLAG_DUP_BLOCK; + return 0; } @@ -2959,6 +3030,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) e2fsck_pass1_free_bitmap(&thread_ctx->inode_imagic_map); e2fsck_pass1_free_bitmap(&thread_ctx->inode_reg_map); e2fsck_pass1_free_bitmap(&thread_ctx->inodes_to_rebuild); + e2fsck_pass1_free_bitmap(&thread_ctx->block_found_map); e2fsck_pass1_free_bitmap(&thread_ctx->block_ea_map); if (thread_ctx->refcount) ea_refcount_free(thread_ctx->refcount); @@ -3152,8 +3224,6 @@ static void e2fsck_pass1_multithread(e2fsck_t global_ctx) int num_threads = 1; errcode_t retval; - pthread_mutex_init(&global_ctx->fs_fix_mutex, NULL); - pthread_mutex_init(&global_ctx->fs_block_map_mutex, NULL); retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); if (retval) { com_err(global_ctx->program_name, retval, @@ -3412,54 +3482,27 @@ static void alloc_imagic_map(e2fsck_t ctx) * WARNING: Assumes checks have already been done to make sure block * is valid. This is true in both process_block and process_bad_block. */ -static _INLINE_ void mark_block_used_unlocked(e2fsck_t ctx, blk64_t block) +static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) { struct problem_context pctx; - e2fsck_t global_ctx; - - global_ctx = ctx->global_ctx; - if (!global_ctx) - global_ctx = ctx; + e2fsck_t global_ctx = ctx->global_ctx ? ctx->global_ctx : ctx; clear_problem_context(&pctx); - if (ext2fs_fast_test_block_bitmap2(ctx->block_found_map, block)) { + if (is_blocks_used(ctx, block, 1)) { if (ext2fs_has_feature_shared_blocks(ctx->fs->super) && !(ctx->options & E2F_OPT_UNSHARE_BLOCKS)) { return; } - /** - * this should be safe because this operation has - * been serialized by mutex. - */ - if (!global_ctx->block_dup_map) { - pctx.errcode = e2fsck_allocate_block_bitmap(ctx->fs, - _("multiply claimed block map"), - EXT2FS_BMAP64_RBTREE, "block_dup_map", - &global_ctx->block_dup_map); - if (pctx.errcode) { - pctx.num = 3; - fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, - &pctx); - /* Should never get here */ - ctx->flags |= E2F_FLAG_ABORT; - return; - } - } + ctx->flags |= E2F_FLAG_DUP_BLOCK; + e2fsck_pass1_block_map_w_lock(ctx); ext2fs_fast_mark_block_bitmap2(global_ctx->block_dup_map, block); + e2fsck_pass1_block_map_w_unlock(ctx); } else { ext2fs_fast_mark_block_bitmap2(ctx->block_found_map, block); } } -static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) -{ - e2fsck_pass1_block_map_lock(ctx); - mark_block_used_unlocked(ctx, block); - e2fsck_pass1_block_map_unlock(ctx); - -} - /* * When cluster size is greater than one block, it is caller's responsibility * to make sure block parameter starts at a cluster boundary. @@ -3467,16 +3510,14 @@ static _INLINE_ void mark_block_used(e2fsck_t ctx, blk64_t block) static _INLINE_ void mark_blocks_used(e2fsck_t ctx, blk64_t block, unsigned int num) { - e2fsck_pass1_block_map_lock(ctx); - if (ext2fs_test_block_bitmap_range2(ctx->block_found_map, block, num)) { + if (!is_blocks_used(ctx, block, num)) { ext2fs_mark_block_bitmap_range2(ctx->block_found_map, block, num); } else { unsigned int i; for (i = 0; i < num; i += EXT2FS_CLUSTER_RATIO(ctx->fs)) - mark_block_used_unlocked(ctx, block + i); + mark_block_used(ctx, block + i); } - e2fsck_pass1_block_map_unlock(ctx); } static errcode_t _INLINE_ e2fsck_write_ext_attr3(e2fsck_t ctx, blk64_t block, @@ -3808,7 +3849,12 @@ refcount_fail: inc_ea_inode_refs(ctx, pctx, first, end); ea_refcount_store(ctx->refcount, blk, header->h_refcount - 1); ea_refcount_store(ctx->refcount_orig, blk, header->h_refcount); - mark_block_used(ctx, blk); + /** + * It might be racy that this block has been merged in the + * global found map. + */ + if (!is_blocks_used(ctx, blk, 1)) + ext2fs_fast_mark_block_bitmap2(ctx->block_found_map, blk); ext2fs_fast_mark_block_bitmap2(ctx->block_ea_map, blk); return 1; @@ -5053,31 +5099,24 @@ static int process_bad_block(ext2_filsys fs, } if (blockcnt < 0) { - e2fsck_pass1_block_map_lock(ctx); if (ext2fs_test_block_bitmap2(p->fs_meta_blocks, blk)) { p->bbcheck = 1; if (fix_problem(ctx, PR_1_BB_FS_BLOCK, pctx)) { *block_nr = 0; - e2fsck_pass1_block_map_unlock(ctx); return BLOCK_CHANGED; } - } else if (ext2fs_test_block_bitmap2(ctx->block_found_map, - blk)) { + } else if (is_blocks_used(ctx, blk, 1)) { p->bbcheck = 1; if (fix_problem(ctx, PR_1_BBINODE_BAD_METABLOCK, pctx)) { *block_nr = 0; - e2fsck_pass1_block_map_unlock(ctx); return BLOCK_CHANGED; } - if (e2fsck_should_abort(ctx)) { - e2fsck_pass1_block_map_unlock(ctx); + if (e2fsck_should_abort(ctx)) return BLOCK_ABORT; - } } else { - mark_block_used_unlocked(ctx, blk); + mark_block_used(ctx, blk); } - e2fsck_pass1_block_map_unlock(ctx); return 0; } #if 0 @@ -5090,13 +5129,10 @@ static int process_bad_block(ext2_filsys fs, * there's an overlap between the filesystem table blocks * (bitmaps and inode table) and the bad block list. */ - e2fsck_pass1_block_map_lock(ctx); - if (!ext2fs_test_block_bitmap2(ctx->block_found_map, blk)) { + if (!is_blocks_used(ctx, blk, 1)) { ext2fs_mark_block_bitmap2(ctx->block_found_map, blk); - e2fsck_pass1_block_map_unlock(ctx); return 0; } - e2fsck_pass1_block_map_unlock(ctx); /* * Try to find the where the filesystem block was used... */ @@ -5251,7 +5287,6 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group, fix_problem(ctx, (old_block ? PR_1_RELOC_FROM_TO : PR_1_RELOC_TO), &pctx); pctx.blk2 = 0; - e2fsck_pass1_block_map_lock(ctx); for (i = 0; i < num; i++) { pctx.blk = i; ext2fs_mark_block_bitmap2(ctx->block_found_map, (*new_block)+i); @@ -5272,7 +5307,6 @@ static void new_table_block(e2fsck_t ctx, blk64_t first_block, dgrp_t group, if (pctx.errcode) fix_problem(ctx, PR_1_RELOC_WRITE_ERR, &pctx); } - e2fsck_pass1_block_map_unlock(ctx); ext2fs_free_mem(&buf); } diff --git a/e2fsck/util.c b/e2fsck/util.c index 3dcfa86a..3a84abb6 100644 --- a/e2fsck/util.c +++ b/e2fsck/util.c @@ -587,17 +587,29 @@ void e2fsck_pass1_fix_unlock(e2fsck_t ctx) pthread_mutex_unlock(&global_ctx->fs_fix_mutex); } -void e2fsck_pass1_block_map_lock(e2fsck_t ctx) +void e2fsck_pass1_block_map_w_lock(e2fsck_t ctx) { e2fsck_get_lock_context(ctx); - pthread_mutex_lock(&global_ctx->fs_block_map_mutex); + pthread_rwlock_wrlock(&global_ctx->fs_block_map_rwlock); } -void e2fsck_pass1_block_map_unlock(e2fsck_t ctx) +void e2fsck_pass1_block_map_w_unlock(e2fsck_t ctx) { e2fsck_get_lock_context(ctx); - pthread_mutex_unlock(&global_ctx->fs_block_map_mutex); + pthread_rwlock_unlock(&global_ctx->fs_block_map_rwlock); } + +void e2fsck_pass1_block_map_r_lock(e2fsck_t ctx) +{ + e2fsck_get_lock_context(ctx); + pthread_rwlock_rdlock(&global_ctx->fs_block_map_rwlock); +} + +void e2fsck_pass1_block_map_r_unlock(e2fsck_t ctx) +{ + e2fsck_get_lock_context(ctx); + pthread_rwlock_unlock(&global_ctx->fs_block_map_rwlock); + } #else void e2fsck_pass1_fix_lock(e2fsck_t ctx) { @@ -608,14 +620,24 @@ void e2fsck_pass1_fix_unlock(e2fsck_t ctx) { } +void e2fsck_pass1_block_map_w_lock(e2fsck_t ctx) +{ -void e2fsck_pass1_block_map_lock(e2fsck_t ctx) +} + +void e2fsck_pass1_block_map_w_unlock(e2fsck_t ctx) { } -void e2fsck_pass1_block_map_unlock(e2fsck_t ctx) +void e2fsck_pass1_block_map_r_lock(e2fsck_t ctx) { + +} + +void e2fsck_pass1_block_map_r_unlock(e2fsck_t ctx) +{ + } #endif diff --git a/lib/ext2fs/bitops.h b/lib/ext2fs/bitops.h index 505b3c9c..1facc8dd 100644 --- a/lib/ext2fs/bitops.h +++ b/lib/ext2fs/bitops.h @@ -120,6 +120,8 @@ extern int ext2fs_test_block_bitmap_range(ext2fs_block_bitmap bitmap, extern void ext2fs_set_bitmap_padding(ext2fs_generic_bitmap map); extern __u32 ext2fs_get_generic_bitmap_start(ext2fs_generic_bitmap bitmap); extern __u32 ext2fs_get_generic_bitmap_end(ext2fs_generic_bitmap bitmap); +extern int ext2fs_test_block_bitmap_range2_valid(ext2fs_block_bitmap bitmap, + blk64_t block, unsigned int num); /* 64-bit versions */ diff --git a/lib/ext2fs/gen_bitmap64.c b/lib/ext2fs/gen_bitmap64.c index 50617a34..bdfed633 100644 --- a/lib/ext2fs/gen_bitmap64.c +++ b/lib/ext2fs/gen_bitmap64.c @@ -725,6 +725,39 @@ int ext2fs_test_block_bitmap_range2(ext2fs_block_bitmap gen_bmap, return bmap->bitmap_ops->test_clear_bmap_extent(bmap, block, num); } +int ext2fs_test_block_bitmap_range2_valid(ext2fs_block_bitmap bitmap, + blk64_t block, unsigned int num) +{ + ext2fs_generic_bitmap_64 bmap = (ext2fs_generic_bitmap_64)bitmap; + __u64 end = block + num; + + if (!bmap) + return 0; + + if (EXT2FS_IS_32_BITMAP(bmap)) { + if ((block & ~0xffffffffULL) || + ((block+num-1) & ~0xffffffffULL)) { + return 0; + } + } + + if (!EXT2FS_IS_64_BITMAP(bmap)) + return 0; + + /* convert to clusters if necessary */ + block >>= bmap->cluster_bits; + end += (1 << bmap->cluster_bits) - 1; + end >>= bmap->cluster_bits; + num = end - block; + + if ((block < bmap->start) || (block > bmap->end) || + (block+num-1 > bmap->end)) + return 0; + + return 1; +} + + void ext2fs_mark_block_bitmap_range2(ext2fs_block_bitmap gen_bmap, blk64_t block, unsigned int num) { diff --git a/tests/f_itable_collision/expect.1 b/tests/f_itable_collision/expect.1 index 01c85d4d..7e98baa8 100644 --- a/tests/f_itable_collision/expect.1 +++ b/tests/f_itable_collision/expect.1 @@ -1,6 +1,5 @@ Pass 1: Checking inodes, blocks, and sizes Inode 12 block 37 conflicts with critical metadata, skipping block checks. -Illegal block number passed to ext2fs_test_block_bitmap #268435455 for in-use block map Illegal block number passed to ext2fs_mark_block_bitmap #268435455 for in-use block map Inode 12, i_blocks is 48, should be 56. Fix? yes @@ -27,9 +26,7 @@ Clear inode? yes Restarting e2fsck from the beginning... Pass 1: Checking inodes, blocks, and sizes Inode 12 block 37 conflicts with critical metadata, skipping block checks. -Illegal block number passed to ext2fs_test_block_bitmap #4294967294 for in-use block map Illegal block number passed to ext2fs_mark_block_bitmap #4294967294 for in-use block map -Illegal block number passed to ext2fs_test_block_bitmap #268435455 for in-use block map Illegal block number passed to ext2fs_mark_block_bitmap #268435455 for in-use block map Running additional passes to resolve blocks claimed by more than one inode... From patchwork Wed Nov 18 15:39:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402341 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=rHZZnZax; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn996NY8z9sVq for ; Thu, 19 Nov 2020 02:41:33 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727550AbgKRPlb (ORCPT ); Wed, 18 Nov 2020 10:41:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727438AbgKRPla (ORCPT ); Wed, 18 Nov 2020 10:41:30 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F649C0613D4 for ; Wed, 18 Nov 2020 07:41:30 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id c137so2904038ybf.21 for ; Wed, 18 Nov 2020 07:41:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=8LU0YxAerh4POqUdK05x0LXM/TPnR/ysn8K7JZeGK1E=; b=rHZZnZaxluMOW3UBeHuhozNpkeJjUC/oHh3NG+JwZsw/13LQQZvbJgZtF+w0jmZUaN b8pUS48laux+hLkA9cG8aHysXu0cvjBV899/WcADgHp/Le8KiyEM9iJJeO2PD9RuMGZY NvXQ/qOAJC+8B2YGs2lHKbVkzWO4CXFp9j44TJH13Q/RD+zpG4NilrCRWsBkJU5Hknbq 4Ble+w2PJbkGQsW0WiEm+qmwjinpdozmVO2THlDab4pBSnTsE9SpN8L27Qf+oBCpL0WX JWquHB7AEP04M71GR9+usLqF05wprTkxeDgf7I3NaLtLWbI+H1M5kwnlO/NK9hECoApJ YliA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8LU0YxAerh4POqUdK05x0LXM/TPnR/ysn8K7JZeGK1E=; b=Ozhpj8ToHfcawjAwzcy5rEUNQAJaMh/L/yeAEwTH+zS/MkjREMc//aGI9f8hk9ky6h VxxW5W+E17oTl6c7TX4yR3Xz66Z1eZ9rlU/zOrJkhoTcEb1of4F8LF1wosCtqTmXkmjs L27duzCE5H5zIQk5CIvtiXrAQSDl1MnrxfCRywu6LLvF+3kFylwhl6UJCkLI7aIfMQE/ nne1dDhEO6x3WEuxEQmPFC/ohZ6uKp+bH1Mf2vDXs/f5cTKtLX716I5j6oqYmO/TLxLH q8AqEDtR4NBnt1XS6WaYQeyPDTnIAGQxk8AtMhF5B8aHG21+J6m0Y8aO2HOUEN8RV1Jt dA8Q== X-Gm-Message-State: AOAM533ORvcMzI5n9Dinoq9AP8ttU35AhMF+lPc+dsy4JZ9+u7cqbxa9 0XtIg/CaxLBNEx3GCZhxcSCNd8EcIOmY1cZ+kORuLrcxdTXUbLNnGg/+GxKUga5mBJui6fDGdkv g4W4MxzLkL5rnrw64z/FGy2P5o6GTWA7pE4d5kLVCuzPPvohkv0P1D4/+VEF9GTVtYivQ6ply7a lTPOHbQu0= X-Google-Smtp-Source: ABdhPJwLUDBO86KnpjCzLu/P7FhWHFIl4ex5K0Q93MoWR9FrJLha1IxSnjW+GAKMrRie1hecg9SfzxFXhGVF6fItFjI= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:4946:: with SMTP id w67mr11695609yba.143.1605714089269; Wed, 18 Nov 2020 07:41:29 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:20 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-35-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 34/61] e2fsck: allow admin specify number of threads From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong -m option is introduced to specify number of threads for pfsck. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 2 + e2fsck/pass1.c | 165 ++++++++++++++++---------- e2fsck/unix.c | 17 ++- tests/f_multithread/script | 2 +- tests/f_multithread_completion/script | 2 +- tests/f_multithread_logfile/script | 2 +- tests/f_multithread_no/script | 2 +- tests/f_multithread_preen/script | 2 +- tests/f_multithread_yes/script | 2 +- 9 files changed, 128 insertions(+), 68 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index d4b472f5..f46a95ef 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -450,6 +450,7 @@ struct e2fsck_struct { /* Undo file */ char *undo_file; #ifdef CONFIG_PFSCK + __u32 fs_num_threads; /* serialize fix operation for multiple threads */ pthread_mutex_t fs_fix_mutex; /* protect block_found_map, block_dup_map */ @@ -684,6 +685,7 @@ int check_backup_super_block(e2fsck_t ctx); void check_resize_inode(e2fsck_t ctx); /* util.c */ +#define E2FSCK_MAX_THREADS (65536) extern void *e2fsck_allocate_memory(e2fsck_t ctx, unsigned long size, const char *description); extern int ask(e2fsck_t ctx, const char * string, int def); diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 6dba6d1b..30365d23 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1194,6 +1194,97 @@ static int e2fsck_should_abort(e2fsck_t ctx) return 0; } +static void init_ext2_max_sizes() +{ + int i; + __u64 max_sizes; + + /* + * Init ext2_max_sizes which will be immutable and shared between + * threads + */ +#define EXT2_BPP(bits) (1ULL << ((bits) - 2)) + + for (i = EXT2_MIN_BLOCK_LOG_SIZE; i <= EXT2_MAX_BLOCK_LOG_SIZE; i++) { + max_sizes = EXT2_NDIR_BLOCKS + EXT2_BPP(i); + max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i); + max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i) * EXT2_BPP(i); + max_sizes = (max_sizes * (1UL << i)); + ext2_max_sizes[i - EXT2_MIN_BLOCK_LOG_SIZE] = max_sizes; + } +#undef EXT2_BPP +} + +#ifdef CONFIG_PFSCK +/* TODO: tdb needs to be handled properly for multiple threads*/ +static int multiple_threads_supported(e2fsck_t ctx) +{ +#ifdef CONFIG_TDB + unsigned int threshold; + ext2_ino_t num_dirs; + errcode_t retval; + char *tdb_dir; + int enable; + + profile_get_string(ctx->profile, "scratch_files", "directory", 0, 0, + &tdb_dir); + profile_get_uint(ctx->profile, "scratch_files", + "numdirs_threshold", 0, 0, &threshold); + profile_get_boolean(ctx->profile, "scratch_files", + "icount", 0, 1, &enable); + + retval = ext2fs_get_num_dirs(ctx->fs, &num_dirs); + if (retval) + num_dirs = 1024; /* Guess */ + + /* tdb is unsupported now */ + if (enable && tdb_dir && !access(tdb_dir, W_OK) && + (!threshold || num_dirs > threshold)) + return 0; +#endif + return 1; +} + +/** + * Even though we could specify number of threads, + * but it might be more than the whole filesystem + * block groups, correct it here. + */ +static void e2fsck_pass1_set_thread_num(e2fsck_t ctx) +{ + unsigned flexbg_size = 1; + ext2_filsys fs = ctx->fs; + int num_threads = ctx->fs_num_threads; + int max_threads; + + if (num_threads < 1) { + num_threads = 1; + goto out; + } + + if (!multiple_threads_supported(ctx)) { + num_threads = 1; + fprintf(stderr, "Fall through single thread for pass1 " + "because tdb could not handle properly\n"); + goto out; + } + + if (ext2fs_has_feature_flex_bg(fs->super)) + flexbg_size = 1 << fs->super->s_log_groups_per_flex; + max_threads = fs->group_desc_count / flexbg_size; + if (max_threads == 0) + max_threads = 1; + + if (num_threads > max_threads) { + fprintf(stderr, "Use max possible thread num: %d instead\n", + max_threads); + num_threads = max_threads; + } +out: + ctx->fs_num_threads = num_threads; +} +#endif + /* * We need call mark_table_blocks() before multiple * thread start, since all known system blocks should be @@ -1204,6 +1295,11 @@ static errcode_t e2fsck_pass1_prepare(e2fsck_t ctx) struct problem_context pctx; ext2_filsys fs = ctx->fs; + init_ext2_max_sizes(); +#ifdef CONFIG_PFSCK + e2fsck_pass1_set_thread_num(ctx); +#endif + clear_problem_context(&pctx); if (!(ctx->options & E2F_OPT_PREEN)) fix_problem(ctx, PR_1_PASS_HEADER, &pctx); @@ -2207,27 +2303,6 @@ endit: ctx->invalid_bitmaps++; } -static void init_ext2_max_sizes() -{ - int i; - __u64 max_sizes; - - /* - * Init ext2_max_sizes which will be immutable and shared between - * threads - */ -#define EXT2_BPP(bits) (1ULL << ((bits) - 2)) - - for (i = EXT2_MIN_BLOCK_LOG_SIZE; i <= EXT2_MAX_BLOCK_LOG_SIZE; i++) { - max_sizes = EXT2_NDIR_BLOCKS + EXT2_BPP(i); - max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i); - max_sizes = max_sizes + EXT2_BPP(i) * EXT2_BPP(i) * EXT2_BPP(i); - max_sizes = (max_sizes * (1UL << i)); - ext2_max_sizes[i - EXT2_MIN_BLOCK_LOG_SIZE] = max_sizes; - } -#undef EXT2_BPP -} - #ifdef CONFIG_PFSCK static errcode_t e2fsck_pass1_copy_bitmap(ext2_filsys fs, ext2fs_generic_bitmap *src, ext2fs_generic_bitmap *dest) @@ -3220,9 +3295,9 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, static void e2fsck_pass1_multithread(e2fsck_t global_ctx) { - struct e2fsck_thread_info *infos = NULL; - int num_threads = 1; - errcode_t retval; + struct e2fsck_thread_info *infos = NULL; + int num_threads = global_ctx->fs_num_threads; + errcode_t retval; retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); if (retval) { @@ -3244,54 +3319,22 @@ out_abort: } #endif -/* TODO: tdb needs to be handled properly for multiple threads*/ -static int multiple_threads_supported(e2fsck_t ctx) -{ -#ifdef CONFIG_TDB - unsigned int threshold; - ext2_ino_t num_dirs; - errcode_t retval; - char *tdb_dir; - int enable; - - profile_get_string(ctx->profile, "scratch_files", "directory", 0, 0, - &tdb_dir); - profile_get_uint(ctx->profile, "scratch_files", - "numdirs_threshold", 0, 0, &threshold); - profile_get_boolean(ctx->profile, "scratch_files", - "icount", 0, 1, &enable); - - retval = ext2fs_get_num_dirs(ctx->fs, &num_dirs); - if (retval) - num_dirs = 1024; /* Guess */ - - /* tdb is unsupported now */ - if (enable && tdb_dir && !access(tdb_dir, W_OK) && - (!threshold || num_dirs > threshold)) - return 0; - #endif - return 1; -} - void e2fsck_pass1(e2fsck_t ctx) { errcode_t retval; - int multiple = 0; + int need_single = 1; - init_ext2_max_sizes(); retval = e2fsck_pass1_prepare(ctx); if (retval) return; #ifdef CONFIG_PFSCK - if (multiple_threads_supported(ctx)) { - multiple = 1; + if (ctx->fs_num_threads > 1 || + ctx->options & E2F_OPT_MULTITHREAD) { + need_single = 0; e2fsck_pass1_multithread(ctx); - } else { - fprintf(stderr, "Fall through single thread for pass1 " - "because tdb could not handle properly\n"); } #endif - if (!multiple) + if (need_single) e2fsck_pass1_run(ctx); e2fsck_pass1_post(ctx); } diff --git a/e2fsck/unix.c b/e2fsck/unix.c index 30c2bf31..cd31bcd5 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -819,6 +819,10 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) int res; /* result of sscanf */ #ifdef CONFIG_JBD_DEBUG char *jbd_debug; +#endif +#ifdef CONFIG_PFSCK + char *pm; + unsigned long thread_num; #endif unsigned long long phys_mem_kb; @@ -852,7 +856,7 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) ctx->readahead_kb = ~0ULL; #ifdef CONFIG_PFSCK - while ((c = getopt(argc, argv, "pamnyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) + while ((c = getopt(argc, argv, "pam:nyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) #else while ((c = getopt(argc, argv, "panyrcC:B:dE:fvtFVM:b:I:j:P:l:L:N:SsDkz:")) != EOF) #endif @@ -898,7 +902,18 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) break; #ifdef CONFIG_PFSCK case 'm': + thread_num = strtoul(optarg, &pm, 0); + if (*pm) + fatal_error(ctx, + _("Invalid multiple thread num.\n")); + if (thread_num > E2FSCK_MAX_THREADS) { + fprintf(stderr, + _("threads %lu too large (max %lu)\n"), + thread_num, E2FSCK_MAX_THREADS); + fatal_error(ctx, 0); + } ctx->options |= E2F_OPT_MULTITHREAD; + ctx->fs_num_threads = thread_num; break; #endif case 'n': diff --git a/tests/f_multithread/script b/tests/f_multithread/script index 0fe96cd0..83cd0f03 100644 --- a/tests/f_multithread/script +++ b/tests/f_multithread/script @@ -1,4 +1,4 @@ -FSCK_OPT="-fy -m" +FSCK_OPT="-fy -m1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_completion/script b/tests/f_multithread_completion/script index bf23cd61..0ec13816 100644 --- a/tests/f_multithread_completion/script +++ b/tests/f_multithread_completion/script @@ -1,4 +1,4 @@ -FSCK_OPT="-fy -m -C 1" +FSCK_OPT="-fy -m1 -C 1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_logfile/script b/tests/f_multithread_logfile/script index 4f9ca6f8..dbb65319 100644 --- a/tests/f_multithread_logfile/script +++ b/tests/f_multithread_logfile/script @@ -1,5 +1,5 @@ LOG_FNAME="f_multithread_logfile_xxx" -FSCK_OPT="-fy -m -y -E log_filename=$LOG_FNAME" +FSCK_OPT="-fy -m1 -y -E log_filename=$LOG_FNAME" SKIP_VERIFY="true" ONE_PASS_ONLY="true" SKIP_CLEANUP="true" diff --git a/tests/f_multithread_no/script b/tests/f_multithread_no/script index b93deb3a..db791e11 100644 --- a/tests/f_multithread_no/script +++ b/tests/f_multithread_no/script @@ -1,4 +1,4 @@ -FSCK_OPT="-fn -m" +FSCK_OPT="-fn -m1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_preen/script b/tests/f_multithread_preen/script index ecb79cd6..8965f4a7 100644 --- a/tests/f_multithread_preen/script +++ b/tests/f_multithread_preen/script @@ -1,4 +1,4 @@ -FSCK_OPT="-fp -m" +FSCK_OPT="-fp -m1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck diff --git a/tests/f_multithread_yes/script b/tests/f_multithread_yes/script index 38891f6a..8b4aa9b8 100644 --- a/tests/f_multithread_yes/script +++ b/tests/f_multithread_yes/script @@ -1,4 +1,4 @@ -FSCK_OPT="-f -m" +FSCK_OPT="-f -m1" SECOND_FSCK_OPT=-yf . $cmd_dir/run_e2fsck From patchwork Wed Nov 18 15:39:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402342 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=DxT3gGsB; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9B2Pkjz9sW8 for ; Thu, 19 Nov 2020 02:41:34 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727553AbgKRPlc (ORCPT ); Wed, 18 Nov 2020 10:41:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727438AbgKRPlc (ORCPT ); Wed, 18 Nov 2020 10:41:32 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F06CC0613D4 for ; Wed, 18 Nov 2020 07:41:32 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id e19so2957233ybc.5 for ; Wed, 18 Nov 2020 07:41:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=qZZIcbpfoZ3m5j0ryKxAM+L6rfSrz5CTtp6iMM9vl70=; b=DxT3gGsBvZEs769Ey0GfpQtNGqdljKoJULyNGPSJB3IE9VrS7G82EUZaIxorwDCk4a U4SPsI1JhTrSrpUCUhKmkLdZgPinaU6xdQMom56QFvm6WoDlke49tzgApQrlnK1EnNP9 wmDZ0eR0CoMYn92qQmcdn9D7AymZP0TTrBfVQwuW+q9UC0bmIlm/C/OG3gG1z300bARI UOrpHioKr/f5O8kU71n0fbMJp+hp871lN0mcQ9yg/CdPqs5C7DvpdqZh9CIQamXYgbGQ iHl4ooH468qENUsIgpNPJWw+KR/qBcXI+ytVxngW6zVeAv8RQaBLgeuOberoVFk9ehac Rqhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qZZIcbpfoZ3m5j0ryKxAM+L6rfSrz5CTtp6iMM9vl70=; b=RiG/fy5itkAlnwkxe/OLrngqwVVOeDmlBt7hzVXEquvlqKLKODYQw+hrc8T6v1M739 NJM3iOdGOch2GT6wDfSsacibwIDvAblM6cwg58uOfe4aBFOdmfkVVYTvs/QA7EOgcvTi SBHIrobXqJrtA6BTt9fpX7Aj8IDm7CBtgm5BOg79mH81NaCfwHpX/toffIhhG4dHpr/G a3unToxM8WHXmKDqnfj/c+L8DrW57drYmJAPXypR1zXy0dWbOVC7c7Pw4YRgGDtjZGmH 5RBlcGEhejCgCtcVUXgi/nD1dtPIYe/+sEdXjNt805RoFdCpICXxdB2dPPACue6r0HEA 2EZA== X-Gm-Message-State: AOAM530zHqHspkun3xRpOXQaUlFSc3YfawWkH3XUt/p/r8IQ64nhhs1f n/X32+EomwgSplYcmcJ1T415BH0kaKrk6ubRgS0y5XIa23NmNfIrsi4DQB222DFxjp269HNStyv ErzJMHiGq61X8RBwvm1cUhpNDblQXBKeEIm8RPZcqIjprZGk5e8Tn2NKftj75Gr5OoEvlBNmcgm KmoSGC/38= X-Google-Smtp-Source: ABdhPJwcES15BGXpsJ8tcVUQHaeD3YWagyiSa84tvARagFp8FnCwMSC2fQwp4ON8INJzAkFP7aZfZw1c8pCfsnyyD9Y= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:d981:: with SMTP id q123mr8607734ybg.50.1605714091166; Wed, 18 Nov 2020 07:41:31 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:21 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-36-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 35/61] e2fsck: adjust number of threads From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong number of threads should not exceed flex bg numbers, and output messages if we adjust threads number. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 32 +++++++++++++++----------------- lib/ext2fs/ext2fs.h | 32 +++++++++++++++++++++++++++++++- 2 files changed, 46 insertions(+), 18 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 30365d23..e2387fe3 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1282,6 +1282,7 @@ static void e2fsck_pass1_set_thread_num(e2fsck_t ctx) } out: ctx->fs_num_threads = num_threads; + ctx->fs->fs_num_threads = num_threads; } #endif @@ -2551,14 +2552,14 @@ static void e2fsck_pass1_merge_invalid_bitmaps(e2fsck_t global_ctx, } static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thread_ctx, - int thread_index, int num_threads) + int thread_index, int num_threads, + dgrp_t average_group) { errcode_t retval; e2fsck_t thread_context; ext2_filsys thread_fs; ext2_filsys global_fs = global_ctx->fs; struct e2fsck_thread *tinfo; - dgrp_t average_group; assert(global_ctx->inode_used_map == NULL); assert(global_ctx->inode_dir_map == NULL); @@ -2605,16 +2606,9 @@ static errcode_t e2fsck_pass1_thread_prepare(e2fsck_t global_ctx, e2fsck_t *thre thread_context->thread_info.et_thread_index = thread_index; set_up_logging(thread_context); - /* - * Distribute work to multiple threads: - * Each thread work on fs->group_desc_count / nthread groups. - */ tinfo = &thread_context->thread_info; - average_group = thread_fs->group_desc_count / num_threads; - if (average_group == 0) - average_group = 1; tinfo->et_group_start = average_group * thread_index; - if (thread_index == num_threads - 1) + if (thread_index == global_fs->fs_num_threads - 1) tinfo->et_group_end = thread_fs->group_desc_count; else tinfo->et_group_end = average_group * (thread_index + 1); @@ -3130,12 +3124,13 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) } static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, - int num_threads, e2fsck_t global_ctx) + e2fsck_t global_ctx) { errcode_t rc; errcode_t ret = 0; int i; struct e2fsck_thread_info *pinfo; + int num_threads = global_ctx->fs_num_threads; /* merge invalid bitmaps will recalculate it */ global_ctx->invalid_bitmaps = 0; @@ -3217,7 +3212,7 @@ out: } static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, - int num_threads, e2fsck_t global_ctx) + e2fsck_t global_ctx) { struct e2fsck_thread_info *infos; pthread_attr_t attr; @@ -3226,6 +3221,8 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, struct e2fsck_thread_info *tmp_pinfo; int i; e2fsck_t thread_ctx; + dgrp_t average_group; + int num_threads = global_ctx->fs_num_threads; #ifdef DEBUG_THREADS struct e2fsck_thread_debug thread_debug = {PTHREAD_MUTEX_INITIALIZER, PTHREAD_COND_INITIALIZER, 0}; @@ -3249,6 +3246,7 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, return retval; } + average_group = ext2fs_get_avg_group(global_ctx->fs); for (i = 0; i < num_threads; i++) { tmp_pinfo = &infos[i]; tmp_pinfo->eti_thread_index = i; @@ -3256,7 +3254,8 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, tmp_pinfo->eti_debug = &thread_debug; #endif retval = e2fsck_pass1_thread_prepare(global_ctx, &thread_ctx, - i, num_threads); + i, num_threads, + average_group); if (retval) { com_err(global_ctx->program_name, retval, _("while preparing pass1 thread\n")); @@ -3286,7 +3285,7 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, } if (retval) { - e2fsck_pass1_threads_join(infos, num_threads, global_ctx); + e2fsck_pass1_threads_join(infos, global_ctx); return retval; } *pinfo = infos; @@ -3296,17 +3295,16 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, static void e2fsck_pass1_multithread(e2fsck_t global_ctx) { struct e2fsck_thread_info *infos = NULL; - int num_threads = global_ctx->fs_num_threads; errcode_t retval; - retval = e2fsck_pass1_threads_start(&infos, num_threads, global_ctx); + retval = e2fsck_pass1_threads_start(&infos, global_ctx); if (retval) { com_err(global_ctx->program_name, retval, _("while starting pass1 threads\n")); goto out_abort; } - retval = e2fsck_pass1_threads_join(infos, num_threads, global_ctx); + retval = e2fsck_pass1_threads_join(infos, global_ctx); if (retval) { com_err(global_ctx->program_name, retval, _("while joining pass1 threads\n")); diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 0fa0e22f..83f2af07 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -255,10 +255,11 @@ struct struct_ext2_filsys { int cluster_ratio_bits; __u16 default_bitmap_type; __u16 pad; + __u32 fs_num_threads; /* * Reserved for future expansion */ - __u32 reserved[5]; + __u32 reserved[4]; /* * Reserved for the use of the calling application. @@ -2121,6 +2122,35 @@ ext2fs_const_inode(const struct ext2_inode_large * large_inode) return (const struct ext2_inode *) large_inode; } +static dgrp_t ext2fs_get_avg_group(ext2_filsys fs) +{ +#ifdef CONFIG_PFSCK + dgrp_t average_group; + unsigned flexbg_size; + + if (fs->fs_num_threads <= 1) + return fs->group_desc_count; + + average_group = fs->group_desc_count / fs->fs_num_threads; + if (average_group <= 1) + return 1; + + if (ext2fs_has_feature_flex_bg(fs->super)) { + int times = 1; + + flexbg_size = 1 << fs->super->s_log_groups_per_flex; + if (average_group % flexbg_size) { + times = average_group / flexbg_size; + average_group = times * flexbg_size; + } + } + + return average_group; +#else + return fs->group_desc_count; +#endif +} + #undef _INLINE_ #endif From patchwork Wed Nov 18 15:39:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402343 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=Ct6JKe4I; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9D0DxTz9sVk for ; Thu, 19 Nov 2020 02:41:36 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726518AbgKRPlf (ORCPT ); Wed, 18 Nov 2020 10:41:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725772AbgKRPlf (ORCPT ); Wed, 18 Nov 2020 10:41:35 -0500 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B53CDC0613D4 for ; Wed, 18 Nov 2020 07:41:33 -0800 (PST) Received: by mail-qt1-x849.google.com with SMTP id 100so1653347qtf.14 for ; Wed, 18 Nov 2020 07:41:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=iMZrbO0dmf4V31EW7iI98HkF+2Wa62X/z0L1CLIakiA=; b=Ct6JKe4IPdiP637C3/6PucYEq4uzAc/Oyhj/EAg9bNa6XOQ55HkM5yfVfUVEzblFPX TAyFecKuJYBAp5rz/EnjKXqei/aHadIosmmV+W+rKArUpV15K+8cs61BCZi1SnUimKy5 TtC7NrdSHVQ6kZ5pBk7y3j7eHEshvzdqoLrcUWuPgGMxXdQ21dx0MtCsG7PtiJURh9o4 6rbzapg2+/75F6u9HMh2Ux7ySy6mtURB987xlbpufG6zCreyMQwaiq56CtLVe64ei7ki wwrL/Fk1JoascAVr+L6MHww/1xaaRuq3cEa/ukADtfjS85J3qUiLQrgiY55dlGnkGiT/ UBcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=iMZrbO0dmf4V31EW7iI98HkF+2Wa62X/z0L1CLIakiA=; b=jllsJotA99eJrH18jDWLW74TtDtQo12doJx/hO+NnLri646dfWbiCHhNE0QlEvE7Qr bvAgVb6n+3OmovCuM/UsAAtvHTK+s0in8z9MFwOgAAyXjx+lUrPEgj/3UN69kCVSqv1b QBTnfBfgp4o5CA0UmA0rPXWY4Plh9lEwqMOqLeENO4Vphj7nrAq5UGLTFMFYjsvaSL7J DY//kjh34B5iZPeaO1ksBCaoeq33pvh5GscoHnSv9qpHrIyvSXp/lFpHrOf2800UecDx 3VvQgvpDA8onga9dhZxNBNiEu1V7qhDGZdlm0Tmo1iNMFx287KC0u2JzhZsIsZBhg1f4 J0Nw== X-Gm-Message-State: AOAM532Ek747lqCNqML6j4blxFuJMvGpAtlbxPxm8U+WC0HT5+OUNR1H q2I5i//P8kIaGKin2h9THtWJm9n7spUqamCFA193t0PXp7dwNXvPNVTMmvNmIQ3zVKvbVLhJplN 5gv8ST7Th347q4YVKFf7WedHiQSdz7iy/hbKcCQZyjpMH1ooRfVfRBqcw5qfbNLOIYPv0Y4s13P a8j0F2E8w= X-Google-Smtp-Source: ABdhPJzmnBNAWIEnBtQLGe2cva5+ZBHqgYEYStyTskraz/j0iHWF/k2kI1tSj9+CsIkFd6GvX9amFDLLM7t+uVJMaUM= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:9e65:: with SMTP id z37mr5049609qve.39.1605714092824; Wed, 18 Nov 2020 07:41:32 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:22 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-37-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 36/61] e2fsck: fix readahead for pfsck of pass1 From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Several improvments for this patch: 1) move readahead_kb detection to preparing phase. 2) inode readahead should be aware of thread block group boundary. 3) make readahead_kb aware of multiple threads. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index e2387fe3..ad3bd8be 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1093,16 +1093,19 @@ out: static void pass1_readahead(e2fsck_t ctx, dgrp_t *group, ext2_ino_t *next_ino) { ext2_ino_t inodes_in_group = 0, inodes_per_block, inodes_per_buffer; - dgrp_t start = *group, grp; + dgrp_t start = *group, grp, grp_end = ctx->fs->group_desc_count; blk64_t blocks_to_read = 0; errcode_t err = EXT2_ET_INVALID_ARGUMENT; +#ifdef CONFIG_PFSCK + grp_end = ctx->thread_info.et_group_end; +#endif if (ctx->readahead_kb == 0) goto out; /* Keep iterating groups until we have enough to readahead */ inodes_per_block = EXT2_INODES_PER_BLOCK(ctx->fs->super); - for (grp = start; grp < ctx->fs->group_desc_count; grp++) { + for (grp = start; grp < grp_end; grp++) { if (ext2fs_bg_flags_test(ctx->fs, grp, EXT2_BG_INODE_UNINIT)) continue; inodes_in_group = ctx->fs->super->s_inodes_per_group - @@ -1295,12 +1298,25 @@ static errcode_t e2fsck_pass1_prepare(e2fsck_t ctx) { struct problem_context pctx; ext2_filsys fs = ctx->fs; + unsigned long long readahead_kb; init_ext2_max_sizes(); -#ifdef CONFIG_PFSCK +#ifdef CONFIG_PFSCK e2fsck_pass1_set_thread_num(ctx); #endif + /* If we can do readahead, figure out how many groups to pull in. */ + if (!e2fsck_can_readahead(ctx->fs)) + ctx->readahead_kb = 0; + else if (ctx->readahead_kb == ~0ULL) + ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs); +#ifdef CONFIG_PFSCK + /* don't use more than 1/10 of memory for threads checking */ + readahead_kb = get_memory_size() / (10 * ctx->fs_num_threads); + /* maybe better disable RA if this is too small? */ + if (ctx->readahead_kb > readahead_kb) + ctx->readahead_kb = readahead_kb; +#endif clear_problem_context(&pctx); if (!(ctx->options & E2F_OPT_PREEN)) fix_problem(ctx, PR_1_PASS_HEADER, &pctx); @@ -1477,13 +1493,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) init_resource_track(&rtrack, ctx->fs->io); clear_problem_context(&pctx); - /* If we can do readahead, figure out how many groups to pull in. */ - if (!e2fsck_can_readahead(ctx->fs)) - ctx->readahead_kb = 0; - else if (ctx->readahead_kb == ~0ULL) - ctx->readahead_kb = e2fsck_guess_readahead(ctx->fs); pass1_readahead(ctx, &ra_group, &ino_threshold); - if (ext2fs_has_feature_dir_index(fs->super) && !(ctx->options & E2F_OPT_NO)) { if (ext2fs_u32_list_create(&ctx->dirs_to_hash, 50)) From patchwork Wed Nov 18 15:39:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402367 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=AjQEOCJV; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CbnBN2PkJz9sPB for ; Thu, 19 Nov 2020 02:42:36 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727554AbgKRPlg (ORCPT ); Wed, 18 Nov 2020 10:41:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727438AbgKRPlf (ORCPT ); Wed, 18 Nov 2020 10:41:35 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7EDB8C0613D4 for ; Wed, 18 Nov 2020 07:41:35 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id z29so2885179ybi.23 for ; Wed, 18 Nov 2020 07:41:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=6e3QHpxWYS5pREffVmkDbMng1wHtMagVt6U7jho/w5M=; b=AjQEOCJVRTZHcPDd3WEh0lncuncx1Mngl6692ZuxuZzC2FIocCQnbvVasczjQ9y5P4 BMyJbitPUuKWv6Eekzu6+qJhjmPW0u59qrjlRw1OW9rN2vwlcNX4QWcYw+oT2oLF6RDD 6obH1sgpnB/Fa7mzQTbbc5CwTOdsCFDXp0cjg86J4nCb38meiT/0QN4rKVeyWcGwA0fg LES7c1wNP3+T34p7OfbeJmL59Z/FL0dou+o+T5o7AB8cwECrf5RMdBtmoHBVxuUzoQDv BCZ1u0vVql9BN4TqxWtNQSwfBR3e/9r9mKeNwboFsQD7mMDP7aSJ0Hj7dDR83EFJmgCl k4Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=6e3QHpxWYS5pREffVmkDbMng1wHtMagVt6U7jho/w5M=; b=pI6e85AvMMM5ONYOkpiv/iDD46g7vRDWbzS+/cScB4MZIYhlGubY7IBqevZzjau03M nU2UvG3Vd/g5WK3wDCJATOSYEHoRPdQBgmkAxcFucKY3HhKrwNVqqLsDPKCEXZKD2qYZ tbE4Yhad5To39VXdmd7GyXxerqsGoGBkDZ8aXNvQNCvY62aFJ5ttyfWaM8P6gTh/eOc1 lmmUJvYtdTMZNl9/QWLJRQqy6y55TgOHiVUmR8LZCtffAo1qq7NfkEKGaNKf5j2BYzDg 6MqmWSZwNYoenor1ajcxSxp6Lp3rSVbkDp0DBiRNwr6mTwlka4dWCCQZKFC9UhUgi3UL P4VQ== X-Gm-Message-State: AOAM533TMaP9MOa2bLJ4muQfkdz4ky3aGIwd0VIoG7KCVbwLKtK57p6w zvH/0cYx23yv18seE/wnitn9Exup6fMIsjL0nl8miRw6zh4td+E1raZ7HJS1dpYZ/PxSt0b1aap D49/VVIh02pDm89hvMezsMvXj8OiAlCoIsUq4ziNkIjTrr41a6EmUPZ3zTk/jAlYKtTuBMBk0jV TtsBEeg/0= X-Google-Smtp-Source: ABdhPJxXlzvqgP8nctltuybtGM4MQuKY1GNhWxB9jLm1vEiyypvEzxRAzkhiGiwANGboaS/7j12J/k/o+K2Te68Sdrw= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:7e42:: with SMTP id z63mr8602623ybc.63.1605714094707; Wed, 18 Nov 2020 07:41:34 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:23 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-38-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 37/61] e2fsck: merge options after threads finish From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong It will be possible that threads might append E2F_OPT_YES, so we need merge options to global, test f_yesall cover this. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index ad3bd8be..1a68a2fb 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2935,6 +2935,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; + int options = global_ctx->options; #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2987,7 +2988,8 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx global_ctx->fs_fragmented += fs_fragmented; global_ctx->fs_fragmented_dir += fs_fragmented_dir; global_ctx->large_files += large_files; - + /* threads might enable E2F_OPT_YES */ + global_ctx->options |= options; global_ctx->flags |= flags; retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); @@ -3022,10 +3024,7 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx thread_ctx->qctx); if (retval) return retval; - global_ctx->invalid_block_bitmap_flag = invalid_block_bitmap_flag; - global_ctx->invalid_inode_bitmap_flag = invalid_inode_bitmap_flag; - global_ctx->invalid_inode_table_flag = invalid_inode_table_flag; - global_ctx->invalid_bitmaps = invalid_bitmaps; + e2fsck_pass1_merge_invalid_bitmaps(global_ctx, thread_ctx); retval = e2fsck_pass1_merge_bitmap(global_fs, From patchwork Wed Nov 18 15:39:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402344 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=O6haHNlO; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9G2ZTKz9sWD for ; Thu, 19 Nov 2020 02:41:38 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727438AbgKRPli (ORCPT ); Wed, 18 Nov 2020 10:41:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726209AbgKRPlh (ORCPT ); Wed, 18 Nov 2020 10:41:37 -0500 Received: from mail-qt1-x84a.google.com (mail-qt1-x84a.google.com [IPv6:2607:f8b0:4864:20::84a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 697DEC0613D4 for ; Wed, 18 Nov 2020 07:41:37 -0800 (PST) Received: by mail-qt1-x84a.google.com with SMTP id z14so1659382qto.8 for ; Wed, 18 Nov 2020 07:41:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=sq/Wt/N6vZNBn3ie6Kd7N2wURpAwp2rrAhDC4aN8dBw=; b=O6haHNlOzhj1DRgwZioYmkBwY8/quVYgptj9yUSUl/3lqKhOw+1vtbsiSb06WNShGR Z7JZ6aVwoKZY328j77S92QsTrKeUBc65jGHkvGIJbTN2YLx0VdJTPTggTukBP6mOK98s 3Tx5q/4wUL0dU9b75lyF/9yUd4sr3PBcua8BJl0Hpx351nqxEJv3o2g1zfrS0Vngmwc8 Wdf51P6WsMZmtKZjfnoY2hlA3aPRVF7hDp2B2er/Q9uo82XOjIsoDsObECFMAJgdusmY UzSSzhlE/xxlm+4LXjAoI5fyGz0pLV/Cy6lJqyO4S7kZJjDh0shWT64zlX6RpyHPuAtV 3pdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=sq/Wt/N6vZNBn3ie6Kd7N2wURpAwp2rrAhDC4aN8dBw=; b=WbaT5f9Ew8sH0ADwDL0RWiydvB/7H4IYncJyKkAmacGKWv8Qhp/r0RVfhoaH99+LCQ BvPtDqGgq6twF9JYt6GSMfYsAsz2gur/dHeqsrq1L8RGFX78sOBddhr6Acmf6ZiVO+Kf M590uavMYrQUGIPDBP9oT9O0tqG44eaiMP75yDKUMZKi8R7Rj6utLSBY5cJ+8AD16d1J QlJ/RDc8VgHqTDQ8BpOd1cOTcr0ZrpfXcyPBGwnSx+7L6xT1Tnnnvb99VPi9H0SLQTHC 60l6S9cBy7I7pJqn24ZsaXmsDyPc8s5qb26EBOdvubQrmMhu74knXXTyqcPx6vxWdxJi z+Ag== X-Gm-Message-State: AOAM533AqqfZ3+u/m2GXamv7FxDkZbkgsvHLp/8iC/ERZbSlQomBXJuG LHABXTocRQoZo9ZvvfMnQAD3rBg+hfJe48q8nZFjqJXjqLExndPagX40dVNEOaY63rrovgluQge ahSPTWJVtm7Ah8xksGzWNgUv95lPaSWuPvRJ/4ZonHaf+AcXYaMnrpe24ovYaLmG2lxItBA+wWG ZyLPNFzaI= X-Google-Smtp-Source: ABdhPJymAya9IYyoW2ZE6fX3KEASj3dZyV5UNK+Zttos1qcvGGA3ePAfbLqEgCJ2bjcGG3RdpuTcOEFXKaSWediu6MY= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:ad4:548b:: with SMTP id q11mr4911153qvy.44.1605714096518; Wed, 18 Nov 2020 07:41:36 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:24 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-39-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 38/61] e2fsck: reset lost_and_found after threads finish From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong This should not be kept, the reaons is similar to what e2fsck_pass1 has done before. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 1a68a2fb..09bfef44 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2991,6 +2991,11 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx /* threads might enable E2F_OPT_YES */ global_ctx->options |= options; global_ctx->flags |= flags; + /* + * The l+f inode may have been cleared, so zap it now and + * later passes will recalculate it if necessary + */ + global_ctx->lost_and_found = 0; retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); if (retval) { From patchwork Wed Nov 18 15:39:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402345 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=knMVGkzn; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9H6v7rz9sWM for ; Thu, 19 Nov 2020 02:41:39 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727556AbgKRPlj (ORCPT ); Wed, 18 Nov 2020 10:41:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727453AbgKRPlj (ORCPT ); Wed, 18 Nov 2020 10:41:39 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12761C0613D4 for ; Wed, 18 Nov 2020 07:41:39 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id k7so2893301ybm.13 for ; Wed, 18 Nov 2020 07:41:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=N9eDhLjAK0lXjDIxv8fRi1NvR4Rb1NkH/1Lwpae+OkM=; b=knMVGkzn9iDTAsLdt0JTlH5fJS9NgYczDqyP3bpCRHh+18z1sMryFYkxLGvpEm+ZwK Iff6ShNzDCQCrJcUW/v5Y1+65PZV272znx2N8hnaRhoeGB+Vfkp2u0qiNsgyLPPAA4O6 6lNIZ7EXhPZhfUqtLfYLudHx1i0ZIq6ROA6OlOspw1UtfTSGQXIwDDfz8Ocgh/rfk+Hf y7AAEUHXovmjjyE9l/sOxT9d+lTOUyxW7S/eess7il8+Sp+kcFB/CAMD6ywmaTJoo7qU fm2x/BgBM/lEquywaGm+gK1ND1MJdW5LwVq2P3Uugrct6sjCUYe76hw9Xnm1e+KcAmXS obBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=N9eDhLjAK0lXjDIxv8fRi1NvR4Rb1NkH/1Lwpae+OkM=; b=NqIcysE8qPgCFYVlg4apPYco9t+YEErMRoTV2wbGUXvTRFBlIqIzEAz6SEoNgPvVGr JdzIuul+wl1YirUGgdOlJqVDuFbfnTSvLjHXplY0bEUtd7zFjtmRk6v9sN1irz+jt0KE Dj+UeYoko/iL8ax+ULiooiD2hKTBlhsQSQBlQDfWoiYn17gpIWYEwH/HxZdD8E3Ro+7T DVMD6hyoWnl9bLXJP9FppE6w3wQGLAUfJ8fqiVDYlX73oPeejHnkwonHBDKefswT/ECk XTxPTsX/FRCr89NPM6RXk2qp/9OdTvI7BvHL2ybJnlUyZFructcio1ol1mudH6oVtTWh uQuQ== X-Gm-Message-State: AOAM530MGQfZk1RLOiVwHIcgG1Ufx3+CngRTWt78y4J97Z67q1BBB/Mb HAynASffTCzK3aefftBl4dfqXu2aETer/iLe56MNO0r91+sRjhzVhJHISzJO/rgdEhzQJH0yFhf /6rXkU9O9TUcLqw6+GEyWGdaR0gWv2uDc5+UJ7oBVJMP0ZsNIVrprrVaNm8QttQJJVHtPh8P/je vPUTpkvh8= X-Google-Smtp-Source: ABdhPJxfUhGyBptjYbSLizRSJb7QxlYRXnCBoa28RoW6Pper4W1PFd5HlQCJJTYOeDPHU6PvNPjW4YyHx+XyGStGDE8= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a5b:850:: with SMTP id v16mr5558343ybq.4.1605714098192; Wed, 18 Nov 2020 07:41:38 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:25 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-40-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 39/61] e2fsck: merge extent depth count after threads finish From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong tests covered by f_extent_htree. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 09bfef44..0a872028 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2935,8 +2935,11 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; - int options = global_ctx->options; + int options = global_ctx->options, i; + __u32 extent_depth_count[MAX_EXTENT_DEPTH_COUNT]; + memcpy(extent_depth_count, global_ctx->extent_depth_count, + sizeof(extent_depth_count)); #ifdef HAVE_SETJMP_H jmp_buf old_jmp; @@ -2996,6 +2999,12 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx * later passes will recalculate it if necessary */ global_ctx->lost_and_found = 0; + memcpy(global_ctx->extent_depth_count, extent_depth_count, + sizeof(extent_depth_count)); + /* merge extent depth count */ + for (i = 0; i < MAX_EXTENT_DEPTH_COUNT; i++) + global_ctx->extent_depth_count[i] += + thread_ctx->extent_depth_count[i]; retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); if (retval) { From patchwork Wed Nov 18 15:39:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402346 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=dISW5vyJ; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9K58WJz9sWQ for ; Thu, 19 Nov 2020 02:41:41 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727557AbgKRPll (ORCPT ); Wed, 18 Nov 2020 10:41:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726209AbgKRPlk (ORCPT ); Wed, 18 Nov 2020 10:41:40 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8FECC0613D4 for ; Wed, 18 Nov 2020 07:41:40 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id o10so1409943pjr.1 for ; Wed, 18 Nov 2020 07:41:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=l3Zjzi7p0M6YZou7JBYanihfRqqLv3MLERAvxMffuZI=; b=dISW5vyJiz3fP8/TiCalReqv7CX1jgiCZFky5i/CLLlnM45Ze59ZKjC1eGKI2f7rZ/ kKcXzruG8i94q2HIDe1bZEfnFU4l6dnVry6/T8dIENumScT9wULfKX+7gn919AdJJ9S9 t9Q2QleaaXRuMHidzJUqoQvyJSI2ZClNOSkSG5tqpY+h25qE/sLRYqh0dWLl4SvRDJzB iPBf76lQfv9L8i7kOpoJFzpeCVLsLp5DEFhOZhhr7YC3+YWySqlpwCRDQatCYHdxKbp4 IM79PlizbgOiaQe3T/69F/VFBz38JFuiR4PbmqxIt9RTNm2XFCMw62oJPHMVal3nsvdX f1MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=l3Zjzi7p0M6YZou7JBYanihfRqqLv3MLERAvxMffuZI=; b=L3lU2tZRIEHwacguzfJNwaaKt4WaPMvBR/UwJV/QQv19+1/kWhusF9OUb0YyGiiIoc Sd0cGbT/rRnbu0r+jtfx5Uzc2l2qOtLJuGWNYqlMHS1lkBX5qzdTBDm3kq1vTaUvrHL2 qdD6gj3MyB64nUZK5sMuRGdrKZzuNw7urgwIeq7lsdhX+UBloMrkSdJNZFbpxH7SQUZn ndvkBjo61IfEBU+YYxQjqTltlQcaDYr0vJvw8AJ1yMAbYAQzgaTh4CT0fPUB8H1CVpXs cLuA911P2DyiCe4glJTDSJWs3ZeeMia0JcWJtuD+c6IHojmtGtiJo1A8oL6+j96k5G8R 588g== X-Gm-Message-State: AOAM5333DGOtrIulOUn2peAkVJJjTIWTTuBJXfSlSkPUy4wyVoT9pT3N oXjtoJUY6iaxlgOLT42cq5Y2HoP5A8NQkOX65umG83zNbyN/sPqF0NX2KZByOkueREA8skgir2X 2O+1jSNZruTAecMSNuBLLkQe70JxVYTssqXscdOQO5vMaf4UkW2Dy1RShc+S2+DoqbdglpSioYa INYzSF+kA= X-Google-Smtp-Source: ABdhPJx+MN2ojE/9FynzzhO4iNOU5UpFgbvx0YX3px2vVBfIbLc9N6ywo5SM6sn4ZNnmadNLpJNhC4jzrqlrFhu2D0U= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a62:ee06:0:b029:164:20d:183b with SMTP id e6-20020a62ee060000b0290164020d183bmr5170370pfi.4.1605714100119; Wed, 18 Nov 2020 07:41:40 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:26 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-41-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 40/61] e2fsck: simplify e2fsck context merging codes From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong We tried to copy thread context to global context directly and then copy back some saved variables before merging. Since we have finished almost all necessary variables in the e2fsck context, we could simplify codes, and this could help us understand what is missing rather than hide problems. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 151 ++++++++++--------------------------------------- 1 file changed, 31 insertions(+), 120 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 0a872028..03d7f455 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -2432,6 +2432,7 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) ext2_badblocks_list badblocks; ext2_dblist dblist; int flags; + e2fsck_t dest_ctx = dest->priv_data; dest_io = dest->io; dest_image_io = dest->image_io; @@ -2449,6 +2450,7 @@ static int e2fsck_pass1_merge_fs(ext2_filsys dest, ext2_filsys src) dest->block_map = block_map; dest->badblocks = badblocks; dest->dblist = dblist; + dest->priv_data = dest_ctx; if (dest->dblist) dest->dblist->fs = dest; dest->flags = src->flags | flags; @@ -2882,140 +2884,51 @@ static errcode_t e2fsck_pass1_merge_ea_refcount(e2fsck_t global_ctx, return retval; } -static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx) +static errcode_t e2fsck_pass1_merge_context(e2fsck_t global_ctx, + e2fsck_t thread_ctx) { - errcode_t retval; - int flags = global_ctx->flags; - ext2_filsys thread_fs = thread_ctx->fs; - ext2_filsys global_fs = global_ctx->fs; - FILE *global_logf = global_ctx->logf; - FILE *global_problem_logf = global_ctx->problem_logf; - ext2fs_inode_bitmap inode_bad_map = global_ctx->inode_bad_map; - struct dir_info_db *dir_info = global_ctx->dir_info; - struct dx_dir_info *dx_dir_info = global_ctx->dx_dir_info; - ext2fs_inode_bitmap inode_used_map = global_ctx->inode_used_map; - ext2fs_inode_bitmap inode_dir_map = global_ctx->inode_dir_map; - ext2fs_inode_bitmap inode_bb_map = global_ctx->inode_bb_map; - ext2fs_inode_bitmap inode_imagic_map = global_ctx->inode_imagic_map; - ext2fs_inode_bitmap inode_reg_map = global_ctx->inode_reg_map; - ext2fs_block_bitmap inodes_to_rebuild = global_ctx->inodes_to_rebuild; - ext2_icount_t inode_count = global_ctx->inode_count; - ext2_icount_t inode_link_info = global_ctx->inode_link_info; - __u32 fs_directory_count = global_ctx->fs_directory_count; - __u32 fs_regular_count = global_ctx->fs_regular_count; - __u32 fs_blockdev_count = global_ctx->fs_blockdev_count; - __u32 fs_chardev_count = global_ctx->fs_chardev_count; - __u32 fs_links_count = global_ctx->fs_links_count; - __u32 fs_symlinks_count = global_ctx->fs_symlinks_count; - __u32 fs_fast_symlinks_count = global_ctx->fs_fast_symlinks_count; - __u32 fs_fifo_count = global_ctx->fs_fifo_count; - __u32 fs_total_count = global_ctx->fs_total_count; - __u32 fs_badblocks_count = global_ctx->fs_badblocks_count; - __u32 fs_sockets_count = global_ctx->fs_sockets_count; - __u32 fs_ind_count = global_ctx->fs_ind_count; - __u32 fs_dind_count = global_ctx->fs_dind_count; - __u32 fs_tind_count = global_ctx->fs_tind_count; - __u32 fs_fragmented = global_ctx->fs_fragmented; - __u32 fs_fragmented_dir = global_ctx->fs_fragmented_dir; - __u32 large_files = global_ctx->large_files; - ext2_ino_t dx_dir_info_size = global_ctx->dx_dir_info_size; - ext2_ino_t dx_dir_info_count = global_ctx->dx_dir_info_count; - ext2_u32_list dirs_to_hash = global_ctx->dirs_to_hash; - quota_ctx_t qctx = global_ctx->qctx; - int *invalid_block_bitmap_flag = global_ctx->invalid_block_bitmap_flag; - int *invalid_inode_bitmap_flag = global_ctx->invalid_inode_bitmap_flag; - int *invalid_inode_table_flag = global_ctx->invalid_inode_table_flag; - int invalid_bitmaps = global_ctx->invalid_bitmaps; - ext2_refcount_t refcount = global_ctx->refcount; - ext2_refcount_t refcount_extra = global_ctx->refcount_extra; - ext2_refcount_t refcount_orig = global_ctx->refcount_orig; - ext2_refcount_t ea_block_quota_blocks = global_ctx->ea_block_quota_blocks; - ext2_refcount_t ea_block_quota_inodes = global_ctx->ea_block_quota_inodes; - ext2fs_block_bitmap block_ea_map = global_ctx->block_ea_map; - ext2_refcount_t ea_inode_refs = global_ctx->ea_inode_refs; - ext2fs_block_bitmap block_found_map = global_ctx->block_found_map; - ext2fs_block_bitmap block_dup_map = global_ctx->block_dup_map; - int options = global_ctx->options, i; - __u32 extent_depth_count[MAX_EXTENT_DEPTH_COUNT]; - - memcpy(extent_depth_count, global_ctx->extent_depth_count, - sizeof(extent_depth_count)); -#ifdef HAVE_SETJMP_H - jmp_buf old_jmp; - - memcpy(old_jmp, global_ctx->abort_loc, sizeof(jmp_buf)); -#endif - memcpy(global_ctx, thread_ctx, sizeof(struct e2fsck_struct)); -#ifdef HAVE_SETJMP_H - memcpy(global_ctx->abort_loc, old_jmp, sizeof(jmp_buf)); -#endif + ext2_filsys global_fs = global_ctx->fs; + errcode_t retval; + int i; - global_ctx->inode_used_map = inode_used_map; - global_ctx->inode_bad_map = inode_bad_map; - global_ctx->inode_dir_map = inode_dir_map; - global_ctx->inode_bb_map = inode_bb_map; - global_ctx->inode_imagic_map = inode_imagic_map; - global_ctx->inodes_to_rebuild = inodes_to_rebuild; - global_ctx->inode_reg_map = inode_reg_map; - global_ctx->block_dup_map = block_dup_map; - global_ctx->block_found_map = block_found_map; - global_ctx->dir_info = dir_info; - e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); - global_ctx->dx_dir_info = dx_dir_info; - global_ctx->dx_dir_info_count = dx_dir_info_count; - global_ctx->dx_dir_info_size = dx_dir_info_size; - e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); - global_ctx->inode_count = inode_count; - global_ctx->inode_link_info = inode_link_info; - global_ctx->refcount = refcount; - global_ctx->refcount_extra = refcount_extra; - global_ctx->refcount_orig = refcount_orig; - global_ctx->ea_block_quota_blocks = ea_block_quota_blocks; - global_ctx->ea_block_quota_inodes = ea_block_quota_inodes; - global_ctx->block_ea_map = block_ea_map; - global_ctx->ea_inode_refs = ea_inode_refs; - global_ctx->fs_directory_count += fs_directory_count; - global_ctx->fs_regular_count += fs_regular_count; - global_ctx->fs_blockdev_count += fs_blockdev_count; - global_ctx->fs_chardev_count += fs_chardev_count; - global_ctx->fs_links_count += fs_links_count; - global_ctx->fs_symlinks_count += fs_symlinks_count; - global_ctx->fs_fast_symlinks_count += fs_fast_symlinks_count; - global_ctx->fs_fifo_count += fs_fifo_count; - global_ctx->fs_total_count += fs_total_count; - global_ctx->fs_badblocks_count += fs_badblocks_count; - global_ctx->fs_sockets_count += fs_sockets_count; - global_ctx->fs_ind_count += fs_ind_count; - global_ctx->fs_dind_count += fs_dind_count; - global_ctx->fs_tind_count += fs_tind_count; - global_ctx->fs_fragmented += fs_fragmented; - global_ctx->fs_fragmented_dir += fs_fragmented_dir; - global_ctx->large_files += large_files; + global_ctx->fs_directory_count += thread_ctx->fs_directory_count; + global_ctx->fs_regular_count += thread_ctx->fs_regular_count; + global_ctx->fs_blockdev_count += thread_ctx->fs_blockdev_count; + global_ctx->fs_chardev_count += thread_ctx->fs_chardev_count; + global_ctx->fs_links_count += thread_ctx->fs_links_count; + global_ctx->fs_symlinks_count += thread_ctx->fs_symlinks_count; + global_ctx->fs_fast_symlinks_count += thread_ctx->fs_fast_symlinks_count; + global_ctx->fs_fifo_count += thread_ctx->fs_fifo_count; + global_ctx->fs_total_count += thread_ctx->fs_total_count; + global_ctx->fs_badblocks_count += thread_ctx->fs_badblocks_count; + global_ctx->fs_sockets_count += thread_ctx->fs_sockets_count; + global_ctx->fs_ind_count += thread_ctx->fs_ind_count; + global_ctx->fs_dind_count += thread_ctx->fs_dind_count; + global_ctx->fs_tind_count += thread_ctx->fs_tind_count; + global_ctx->fs_fragmented += thread_ctx->fs_fragmented; + global_ctx->fs_fragmented_dir += thread_ctx->fs_fragmented_dir; + global_ctx->large_files += thread_ctx->large_files; /* threads might enable E2F_OPT_YES */ - global_ctx->options |= options; - global_ctx->flags |= flags; + global_ctx->options |= thread_ctx->options; + global_ctx->flags |= thread_ctx->flags; /* * The l+f inode may have been cleared, so zap it now and * later passes will recalculate it if necessary */ global_ctx->lost_and_found = 0; - memcpy(global_ctx->extent_depth_count, extent_depth_count, - sizeof(extent_depth_count)); /* merge extent depth count */ for (i = 0; i < MAX_EXTENT_DEPTH_COUNT; i++) global_ctx->extent_depth_count[i] += thread_ctx->extent_depth_count[i]; - retval = e2fsck_pass1_merge_fs(global_fs, thread_fs); + e2fsck_pass1_merge_dir_info(global_ctx, thread_ctx); + e2fsck_pass1_merge_dx_dir(global_ctx, thread_ctx); + + retval = e2fsck_pass1_merge_fs(global_ctx->fs, thread_ctx->fs); if (retval) { com_err(global_ctx->program_name, 0, _("while merging fs\n")); return retval; } - global_fs->priv_data = global_ctx; - global_ctx->fs = global_fs; - global_ctx->logf = global_logf; - global_ctx->problem_logf = global_problem_logf; - global_ctx->global_ctx = NULL; retval = e2fsck_pass1_merge_icounts(global_ctx, thread_ctx); if (retval) { com_err(global_ctx->program_name, 0, @@ -3023,7 +2936,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx return retval; } - global_ctx->dirs_to_hash = dirs_to_hash; retval = e2fsck_pass1_merge_dirs_to_hash(global_ctx, thread_ctx); if (retval) { com_err(global_ctx->program_name, 0, @@ -3033,7 +2945,6 @@ static int e2fsck_pass1_thread_join_one(e2fsck_t global_ctx, e2fsck_t thread_ctx e2fsck_pass1_merge_ea_inode_refs(global_ctx, thread_ctx); e2fsck_pass1_merge_ea_refcount(global_ctx, thread_ctx); - global_ctx->qctx = qctx; retval = quota_merge_and_update_usage(global_ctx->qctx, thread_ctx->qctx); if (retval) @@ -3107,7 +3018,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) { errcode_t retval; - retval = e2fsck_pass1_thread_join_one(global_ctx, thread_ctx); + retval = e2fsck_pass1_merge_context(global_ctx, thread_ctx); ext2fs_free_mem(&thread_ctx->fs); if (thread_ctx->logf) fclose(thread_ctx->logf); From patchwork Wed Nov 18 15:39:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402347 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=t+YOMAhu; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9P01qVz9sWT for ; Thu, 19 Nov 2020 02:41:44 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727558AbgKRPlo (ORCPT ); Wed, 18 Nov 2020 10:41:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726209AbgKRPlo (ORCPT ); Wed, 18 Nov 2020 10:41:44 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5E87C0613D4 for ; Wed, 18 Nov 2020 07:41:42 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id a20so1502535pgb.21 for ; Wed, 18 Nov 2020 07:41:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Cy+36TZ+jm/k9a0XwFzOM5dMptqvuuXClEnH7B0HQRo=; b=t+YOMAhuTz7hAEFJSfe1yLvMEJGZgijAfP2u1w35kzOml6u9bHy59ymERFcmmgzsw+ h2S8A8urTeQ2mpF+Hc/KXRFsSXTQ/9b5exfNBNCkhr7PJfdw53GUDiBJrjFTmNv5NRMK xRG39Bk0PchpHpHIatVZidi90X4WtbvFw3ZC4Pa8Imb299WLtAyO8UAcV0RsW0JcCSiG MU+2mENKdskVow55NP7ORZAyGivpP+BaK7PKVo9tnrFdvlFoFnvZcshQ8eSBEDFlMq7o roGsoabqL1v7YIzf9HOpU/Dyfw3z+EuoJESinNow2B4kkXC6Nro07UdzVhZsZ/nyZF/T xUFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Cy+36TZ+jm/k9a0XwFzOM5dMptqvuuXClEnH7B0HQRo=; b=f335+aiBm5bVHicustPzwDupWUnxNzkMbyHcVfJCp7py8bKrUwaTC8JyisH+snaqbz aC+l4vJt1mwvXQwfa3ALsHkRH63l+4Os0GG9T8bV+ao2jsB7elTlG+QTvwYGgoKltFVR Mqhm2uqjgahnlQD8a3L2xotYRVoA1O6feG7zsTq2apRl6ssOzpMiGn8uQqdx+uQAiban id3JdN+m94bN+zrVWExrIhctwG6l2wRFNEkY/IJke8bhHkivFbAptiFz9QoKATS/SroI y6k9p9dbziBGGAAeVRttqAFBlKoHIgvzbwWNKWvyJnznAOpy/r1ar3c409F5QrZ8TRuo b5/g== X-Gm-Message-State: AOAM530UAAfjQDpJew3Nq4gsRrehyCIAUeG5aINNdC4csTjkZq5jAZQs xwatDkOHUaZ195u49BpSA759CMYVteOMMF458gl8G01fDeX3/Gt+0iiqAhrQy52i6zmJ5tPHCsL PVWK6KieGiUdj6rrxHegLVU+PmBlJZq8POLJo5wL7VRyAPA/EVUvwmEPGfAHuqx5O/NUDIl1UfQ Ybn+UiJYU= X-Google-Smtp-Source: ABdhPJyQwu6cqd2zdlk1h/1H6ZvmKMO28laMsVK9l6s94eWRE2AI21JL27O5S65bTvqIesgQz3PUaT8KN1uvuXVKl8k= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:902:b947:b029:d8:ab80:1e9c with SMTP id h7-20020a170902b947b02900d8ab801e9cmr5257868pls.42.1605714101991; Wed, 18 Nov 2020 07:41:41 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:27 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-42-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 41/61] e2fsck: set E2F_FLAG_ALLOC_OK after threads From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Only flag ALLOC OK after all threads finished without problem. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 03d7f455..62345bb6 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1379,9 +1379,12 @@ static void e2fsck_pass1_post(e2fsck_t ctx) { struct problem_context pctx; ext2_filsys fs = ctx->fs; + char *block_buf; - char *block_buf = - (char *)e2fsck_allocate_memory(ctx, ctx->fs->blocksize * 3, + if (e2fsck_should_abort(ctx)) + return; + + block_buf = (char *)e2fsck_allocate_memory(ctx, ctx->fs->blocksize * 3, "block interate buffer"); reserve_block_for_root_repair(ctx); reserve_block_for_lnf_repair(ctx); @@ -1459,6 +1462,8 @@ static void e2fsck_pass1_post(e2fsck_t ctx) ext2fs_free_mem(&block_buf); ctx->flags &= ~E2F_FLAG_DUP_BLOCK; } + + ctx->flags |= E2F_FLAG_ALLOC_OK; } @@ -2290,6 +2295,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) } ctx->flags |= E2F_FLAG_ALLOC_OK; + ext2fs_free_mem(&inodes_to_process); endit: e2fsck_use_inode_shortcuts(ctx, 0); ext2fs_free_mem(&inodes_to_process); From patchwork Wed Nov 18 15:39:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402348 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=SSG+ztGe; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9P3nKTz9sWX for ; Thu, 19 Nov 2020 02:41:45 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727561AbgKRPlp (ORCPT ); Wed, 18 Nov 2020 10:41:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727560AbgKRPlo (ORCPT ); Wed, 18 Nov 2020 10:41:44 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA892C0613D4 for ; Wed, 18 Nov 2020 07:41:44 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id f20so1391522pjq.9 for ; Wed, 18 Nov 2020 07:41:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=PM5k/+pyYeYWvC2oKkTa57pw4+TAFvx0ODzORD+kTV8=; b=SSG+ztGeC6uRPYN8UmJjjmVNVqfSXpgL4NmYzgdSTPgorCRFIoqy2OZlNNXtdcZEzI wQpdN9CREMVciY6319f9UaWyD23NJ+dXs+IoeFGYS+ltWbnQ0964sgZ6TKf+adS3epBB xO55GvSUwoL7OO3AKpJiYfJpVI9FpJN2g78Nz2j/7obFQaNQnQPyY+e/CloUL4jP4jOi 6k478iIydRgyXg7LDTbAsakl37lxcweJiGxqq6MjZDCGg0V8Lh+FcJuKt1juciKQFnGv wK74Wmq76mnYM+cfIjvD0LawUZnsrdO+zsLJXdFNWSsmtyJyXmnZSdGraO3vjjM4OwEm GYdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PM5k/+pyYeYWvC2oKkTa57pw4+TAFvx0ODzORD+kTV8=; b=dUYbSjCabxBnFVk1GwAkaAD4aIFSvsZ+Ia/rhTFDv0S7Ua4RaeekiXiloxPOf5C4+5 yWCgJm2Wx4Bhm65G+cIufbMmuTqp+zwkkxCVQH2sULZFcImiGa/8yS7cVUa10AKWtdzl gDW1J+VIhJLhQv+DVA1A0D6ySGfl/NWncq3hijRATnMUg3cOPt2L9OkJZtTrYKB/Ypal B+M8wbPukeURY/Na/bOKQzDcd7KdXV4M0ZmBPElzI+XgAzmFAzZMSOANOVj6IdNSXtya wabzmpOg6Rtu8j1/GF6xwz+YccYbr//fVn/VcaO8RIrRRzs/vpr71RkVjsQgmU+NdF68 zk8g== X-Gm-Message-State: AOAM531U32SjAMiFBK8IjvidS6L0eSIE7OeJXrNJslgISMCP4aV0u2Mr EoWQnPwi5OD+IGsRmYDFP1Tv3OeQB7pDUUamspT2hQ2paDFNgHEJeRbChGZd2OWORPYiNEZb2Ji d6r2bevCY2jP+M9hlknrngp9eCrv5YKvnlfbx7Vej5FjZNVDJeZUOsLBlry8mFkBcOi4oaTDO5D M/vdKPWtU= X-Google-Smtp-Source: ABdhPJyhuVa6fqMC7HY6Pkykr5Bz53dPa+HlsiGRV1zsPtiEIRnBZR9LCvRn4sJazRozRt4ZyvhIZxOaKktIw6pZCOI= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:902:ff07:b029:d8:eea2:afae with SMTP id f7-20020a170902ff07b02900d8eea2afaemr5033556plj.18.1605714103963; Wed, 18 Nov 2020 07:41:43 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:28 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-43-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 42/61] e2fsck: wait fix thread finish before checking From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Before proceeding next inodes, waitting existed fixing finished. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.c | 3 +++ e2fsck/e2fsck.h | 5 +++- e2fsck/pass1.c | 65 +++++++++++++++++++++++++++++++++++++++++++------ e2fsck/util.c | 56 +++++++++++++++++++++++++++++++++++++++--- 4 files changed, 117 insertions(+), 12 deletions(-) diff --git a/e2fsck/e2fsck.c b/e2fsck/e2fsck.c index d8be566f..1fd57504 100644 --- a/e2fsck/e2fsck.c +++ b/e2fsck/e2fsck.c @@ -178,6 +178,9 @@ errcode_t e2fsck_reset_context(e2fsck_t ctx) ctx->fs_fragmented = 0; ctx->fs_fragmented_dir = 0; ctx->large_files = 0; +#ifdef CONFIG_PFSCK + ctx->fs_need_locking = 0; +#endif for (i=0; i < MAX_EXTENT_DEPTH_COUNT; i++) ctx->extent_depth_count[i] = 0; diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index f46a95ef..a66772c1 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -451,8 +451,9 @@ struct e2fsck_struct { char *undo_file; #ifdef CONFIG_PFSCK __u32 fs_num_threads; + int fs_need_locking; /* serialize fix operation for multiple threads */ - pthread_mutex_t fs_fix_mutex; + pthread_rwlock_t fs_fix_rwlock; /* protect block_found_map, block_dup_map */ pthread_rwlock_t fs_block_map_rwlock; #endif @@ -514,6 +515,8 @@ extern int e2fsck_strnlen(const char * s, int count); extern void e2fsck_pass1(e2fsck_t ctx); extern void e2fsck_pass1_dupblocks(e2fsck_t ctx, char *block_buf); +extern void e2fsck_pass1_check_lock(e2fsck_t ctx); +extern void e2fsck_pass1_check_unlock(e2fsck_t ctx); extern void e2fsck_pass2(e2fsck_t ctx); extern void e2fsck_pass3(e2fsck_t ctx); extern void e2fsck_pass4(e2fsck_t ctx); diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 62345bb6..5e62e357 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -986,8 +986,10 @@ static void finish_processing_inode(e2fsck_t ctx, ext2_ino_t ino, #define FINISH_INODE_LOOP(ctx, ino, pctx, failed_csum) \ do { \ finish_processing_inode((ctx), (ino), (pctx), (failed_csum)); \ - if ((ctx)->flags & E2F_FLAG_ABORT) \ + if ((ctx)->flags & E2F_FLAG_ABORT) { \ + e2fsck_pass1_check_unlock(ctx); \ return; \ + } \ } while (0) static int could_be_block_map(ext2_filsys fs, struct ext2_inode *inode) @@ -1368,8 +1370,10 @@ static errcode_t e2fsck_pass1_prepare(e2fsck_t ctx) ext2fs_mark_block_bitmap2(ctx->block_found_map, fs->super->s_mmp_block); #ifdef CONFIG_PFSCK - pthread_mutex_init(&ctx->fs_fix_mutex, NULL); + pthread_rwlock_init(&ctx->fs_fix_rwlock, NULL); pthread_rwlock_init(&ctx->fs_block_map_rwlock, NULL); + if (ctx->fs_num_threads > 1) + ctx->fs_need_locking = 1; #endif return 0; @@ -1642,6 +1646,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) #endif while (1) { + e2fsck_pass1_check_lock(ctx); if (ino % (fs->super->s_inodes_per_group * 4) == 1) { if (e2fsck_mmp_update(fs)) fatal_error(ctx, 0); @@ -1652,8 +1657,10 @@ void e2fsck_pass1_run(e2fsck_t ctx) if (ino > ino_threshold) pass1_readahead(ctx, &ra_group, &ino_threshold); ehandler_operation(old_op); - if (e2fsck_should_abort(ctx)) + if (e2fsck_should_abort(ctx)) { + e2fsck_pass1_check_unlock(ctx); goto endit; + } if (pctx.errcode == EXT2_ET_BAD_BLOCK_IN_INODE_TABLE) { /* * If badblocks says badblocks is bad, offer to clear @@ -1674,27 +1681,45 @@ void e2fsck_pass1_run(e2fsck_t ctx) fix_problem(ctx, PR_1_ISCAN_ERROR, &pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); + goto endit; } else ctx->flags |= E2F_FLAG_RESTART; - goto endit; + err = ext2fs_inode_scan_goto_blockgroup(scan, + 0); + if (err) { + fix_problem(ctx, PR_1_ISCAN_ERROR, + &pctx); + ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); + goto endit; + } + e2fsck_pass1_check_unlock(ctx); + continue; } if (!ctx->inode_bb_map) alloc_bb_map(ctx); ext2fs_mark_inode_bitmap2(ctx->inode_bb_map, ino); ext2fs_mark_inode_bitmap2(ctx->inode_used_map, ino); + e2fsck_pass1_check_unlock(ctx); continue; } - if (pctx.errcode == EXT2_ET_SCAN_FINISHED) + if (pctx.errcode == EXT2_ET_SCAN_FINISHED) { + e2fsck_pass1_check_unlock(ctx); break; + } if (pctx.errcode && pctx.errcode != EXT2_ET_INODE_CSUM_INVALID && pctx.errcode != EXT2_ET_INODE_IS_GARBAGE) { fix_problem(ctx, PR_1_ISCAN_ERROR, &pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); goto endit; } - if (!ino) + if (!ino) { + e2fsck_pass1_check_unlock(ctx); break; + } #ifdef CONFIG_PFSCK if (ctx->global_ctx) ctx->thread_info.et_inode_number++; @@ -1747,6 +1772,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) pctx.num = inode->i_links_count; fix_problem(ctx, PR_1_ICOUNT_STORE, &pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); goto endit; } } else if ((ino >= EXT2_FIRST_INODE(fs->super)) && @@ -1761,6 +1787,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) } } FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); + e2fsck_pass1_check_unlock(ctx); continue; } @@ -1781,6 +1808,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) &pctx); if (res < 0) { /* skip FINISH_INODE_LOOP */ + e2fsck_pass1_check_unlock(ctx); continue; } } @@ -1801,6 +1829,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) } else if (fix_problem(ctx, PR_1_INLINE_DATA_SET, &pctx)) { e2fsck_clear_inode(ctx, ino, inode, 0, "pass1"); /* skip FINISH_INODE_LOOP */ + e2fsck_pass1_check_unlock(ctx); continue; } } @@ -1845,6 +1874,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) if (err) { pctx.errcode = err; ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); goto endit; } inode->i_flags &= ~EXT4_INLINE_DATA_FL; @@ -1859,6 +1889,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) /* Some other kind of non-xattr error? */ pctx.errcode = err; ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); goto endit; } } @@ -1896,6 +1927,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) ext2fs_mark_inode_bitmap2(ctx->inode_used_map, ino); /* skip FINISH_INODE_LOOP */ + e2fsck_pass1_check_unlock(ctx); continue; } } @@ -1959,6 +1991,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) pctx.num = 4; fix_problem(ctx, PR_1_ALLOCATE_BBITMAP_ERROR, &pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); goto endit; } pb.ino = EXT2_BAD_INO; @@ -1976,16 +2009,19 @@ void e2fsck_pass1_run(e2fsck_t ctx) if (pctx.errcode) { fix_problem(ctx, PR_1_BLOCK_ITERATE, &pctx); ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); goto endit; } if (pb.bbcheck) if (!fix_problem(ctx, PR_1_BBINODE_BAD_METABLOCK_PROMPT, &pctx)) { ctx->flags |= E2F_FLAG_ABORT; + e2fsck_pass1_check_unlock(ctx); goto endit; } ext2fs_mark_inode_bitmap2(ctx->inode_used_map, ino); clear_problem_context(&pctx); FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); + e2fsck_pass1_check_unlock(ctx); continue; } else if (ino == EXT2_ROOT_INO) { /* @@ -2027,6 +2063,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) } check_blocks(ctx, &pctx, block_buf, NULL); FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); + e2fsck_pass1_check_unlock(ctx); continue; } if ((inode->i_links_count || @@ -2054,6 +2091,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) } check_blocks(ctx, &pctx, block_buf, NULL); FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); + e2fsck_pass1_check_unlock(ctx); continue; } if ((inode->i_links_count || @@ -2092,11 +2130,13 @@ void e2fsck_pass1_run(e2fsck_t ctx) } check_blocks(ctx, &pctx, block_buf, NULL); FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); + e2fsck_pass1_check_unlock(ctx); continue; } if (!inode->i_links_count) { FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); + e2fsck_pass1_check_unlock(ctx); continue; } /* @@ -2201,12 +2241,14 @@ void e2fsck_pass1_run(e2fsck_t ctx) ctx->fs_symlinks_count++; if (inode->i_flags & EXT4_INLINE_DATA_FL) { FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); + e2fsck_pass1_check_unlock(ctx); continue; } else if (ext2fs_is_fast_symlink(inode)) { ctx->fs_fast_symlinks_count++; check_blocks(ctx, &pctx, block_buf, &ea_ibody_quota); FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); + e2fsck_pass1_check_unlock(ctx); continue; } } @@ -2254,16 +2296,21 @@ void e2fsck_pass1_run(e2fsck_t ctx) FINISH_INODE_LOOP(ctx, ino, &pctx, failed_csum); - if (ctx->flags & E2F_FLAG_SIGNAL_MASK) + if (e2fsck_should_abort(ctx)) { + e2fsck_pass1_check_unlock(ctx); goto endit; + } if (process_inode_count >= ctx->process_inode_size) { process_inodes(ctx, block_buf, inodes_to_process, &process_inode_count); - if (e2fsck_should_abort(ctx)) + if (e2fsck_should_abort(ctx)) { + e2fsck_pass1_check_unlock(ctx); goto endit; + } } + e2fsck_pass1_check_unlock(ctx); } process_inodes(ctx, block_buf, inodes_to_process, &process_inode_count); @@ -3271,6 +3318,8 @@ void e2fsck_pass1(e2fsck_t ctx) need_single = 0; e2fsck_pass1_multithread(ctx); } + /* No lock is needed at this time */ + ctx->fs_need_locking = 0; #endif if (need_single) e2fsck_pass1_run(ctx); diff --git a/e2fsck/util.c b/e2fsck/util.c index 3a84abb6..3d85bff4 100644 --- a/e2fsck/util.c +++ b/e2fsck/util.c @@ -86,7 +86,8 @@ void fatal_error(e2fsck_t ctx, const char *msg) } out: ctx->flags |= E2F_FLAG_ABORT; - if (ctx->flags & E2F_FLAG_SETJMP_OK) + if (!(ctx->options & E2F_OPT_MULTITHREAD) && + ctx->flags & E2F_FLAG_SETJMP_OK) longjmp(ctx->abort_loc, 1); if (ctx->logf) fprintf(ctx->logf, "Exit status: %d\n", exit_value); @@ -575,38 +576,79 @@ void e2fsck_read_inode_full(e2fsck_t ctx, unsigned long ino, if (!global_ctx) \ global_ctx = ctx; \ +/** + * before we hold write lock, read lock should + * has been held. + */ void e2fsck_pass1_fix_lock(e2fsck_t ctx) { + int err; + + if (!ctx->fs_need_locking) + return; + e2fsck_get_lock_context(ctx); - pthread_mutex_lock(&global_ctx->fs_fix_mutex); + err = pthread_rwlock_trywrlock(&global_ctx->fs_fix_rwlock); + assert(err != 0); + pthread_rwlock_unlock(&global_ctx->fs_fix_rwlock); + pthread_rwlock_wrlock(&global_ctx->fs_fix_rwlock); } void e2fsck_pass1_fix_unlock(e2fsck_t ctx) { + if (!ctx->fs_need_locking) + return; e2fsck_get_lock_context(ctx); - pthread_mutex_unlock(&global_ctx->fs_fix_mutex); + /* unlock write lock */ + pthread_rwlock_unlock(&global_ctx->fs_fix_rwlock); + /* get read lock again */ + pthread_rwlock_rdlock(&global_ctx->fs_fix_rwlock); +} + +void e2fsck_pass1_check_lock(e2fsck_t ctx) +{ + if (!ctx->fs_need_locking) + return; + e2fsck_get_lock_context(ctx); + pthread_rwlock_rdlock(&global_ctx->fs_fix_rwlock); +} + +void e2fsck_pass1_check_unlock(e2fsck_t ctx) +{ + if (!ctx->fs_need_locking) + return; + e2fsck_get_lock_context(ctx); + pthread_rwlock_unlock(&global_ctx->fs_fix_rwlock); } void e2fsck_pass1_block_map_w_lock(e2fsck_t ctx) { + if (!ctx->fs_need_locking) + return; e2fsck_get_lock_context(ctx); pthread_rwlock_wrlock(&global_ctx->fs_block_map_rwlock); } void e2fsck_pass1_block_map_w_unlock(e2fsck_t ctx) { + if (!ctx->fs_need_locking) + return; e2fsck_get_lock_context(ctx); pthread_rwlock_unlock(&global_ctx->fs_block_map_rwlock); } void e2fsck_pass1_block_map_r_lock(e2fsck_t ctx) { + if (!ctx->fs_need_locking) + return; e2fsck_get_lock_context(ctx); pthread_rwlock_rdlock(&global_ctx->fs_block_map_rwlock); } void e2fsck_pass1_block_map_r_unlock(e2fsck_t ctx) { + if (!ctx->fs_need_locking) + return; e2fsck_get_lock_context(ctx); pthread_rwlock_unlock(&global_ctx->fs_block_map_rwlock); } @@ -619,6 +661,14 @@ void e2fsck_pass1_fix_lock(e2fsck_t ctx) void e2fsck_pass1_fix_unlock(e2fsck_t ctx) { +} +void e2fsck_pass1_check_lock(e2fsck_t ctx) +{ + +} +void e2fsck_pass1_check_unlock(e2fsck_t ctx) +{ + } void e2fsck_pass1_block_map_w_lock(e2fsck_t ctx) { From patchwork Wed Nov 18 15:39:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402349 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=v4e407f8; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9R5vLGz9sWX for ; Thu, 19 Nov 2020 02:41:47 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727566AbgKRPlr (ORCPT ); Wed, 18 Nov 2020 10:41:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727560AbgKRPlr (ORCPT ); Wed, 18 Nov 2020 10:41:47 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB50CC0613D4 for ; Wed, 18 Nov 2020 07:41:46 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id w8so2923546ybj.14 for ; Wed, 18 Nov 2020 07:41:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=J+qzfoSyTvz9pOQzjIGTh7s4r8If6jlOq/zMCK5+DV4=; b=v4e407f8aGXGpVFY8yg7uJWS0tYiO3NlLuF2kg5i5DGF9uA7OIXOyh4hl9P/y8ZQyz 2Cr8MI3Mc1STskg8OvPQbRd8lZcJoc6qCsRZ/H5fvp4jBGW0tz9O0GsaaW6wD8nrGhJ0 fUx0NktDcw6rMaRJKJ0a8wYjMXzIurEhuNWyh5fI6nvXQAyAQzjS5tlL0/cbb4QDaHBk u+XOiVx2WiCNTFMNscPPno2t9D0AKJCliKdNcV9IvvbldYLz/UDW12R7819Zk3//DigH 2gTzEx12A9/kgGuiopNVYThJkq8hzknnOsICOpQi6+yuAqlb8zH3g1h6Zr6iUnGtEj51 zOHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=J+qzfoSyTvz9pOQzjIGTh7s4r8If6jlOq/zMCK5+DV4=; b=ihby04zrlyQCORsz3zCsRqjYrsbGxMpsiG5B8UvXhA4B0EdH6B7GezOJkn5Uust/bf GbWpbly9IBEj2cicAbCQ3iT22lv+ek57+lEV+vqZ40eCHhzafQpM+ymBvwNZg86eXPg9 m+6zWsH3BjlVL/FAWOrVt7X/93HPi4cZaSxnSvom/K1lw360yrlb64J+alUvspOIIFdy ddsmQPi7t2TvoCbm+tRYJIgIJ+M77EjrvaFvpS6LBM9+I5HOGo7EyIA1ruu2XqbTJkP2 zEZVRquc9BCJm1gIV7NZ5u7D3IneSLu+Fv4gGi20pk74bA9+aQ3MEY/BH6Q4QSAv6h8P 20qg== X-Gm-Message-State: AOAM5335nnQHGPAJXn1Z+7fkOnx7l8Cq5o95FGA+kRW6E3RRTaKvXJHY bjyRw5WwMRa+yfxJGiuvSvHN+tdOPAMBzoMeIcsw9nnT3Te6A4ka8g7BJ4o0V/4m4+EfiEnZ3bY gxnv717l4fw6NiV0u1bb6mS2pqSqPy8aM8/iINQcEyAMzsZqTF1CIyJ0tpSAtCYzTEuHzwrsuUA YM4LX5wPo= X-Google-Smtp-Source: ABdhPJyObyeM+BE15vwDaELPAS1hrxwWEZKVPUXPrm0OlFJt3wf/IfKbmZ5BzKXfpAtu5FBS/xYcdryvJCoUlLC72MI= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:b11:: with SMTP id 17mr7188939ybl.203.1605714106018; Wed, 18 Nov 2020 07:41:46 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:29 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-44-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 43/61] e2fsck: cleanup e2fsck_pass1_thread_join() From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Use e2fsck_reset_context() to free memory to simpify codes. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 34 ++++++++-------------------------- 1 file changed, 8 insertions(+), 26 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 5e62e357..60f70111 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -3079,32 +3079,14 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) fputs("\n", thread_ctx->problem_logf); fclose(thread_ctx->problem_logf); } - e2fsck_pass1_free_bitmap(&thread_ctx->inode_used_map); - e2fsck_pass1_free_bitmap(&thread_ctx->inode_bad_map); - e2fsck_pass1_free_bitmap(&thread_ctx->inode_dir_map); - e2fsck_pass1_free_bitmap(&thread_ctx->inode_bb_map); - e2fsck_pass1_free_bitmap(&thread_ctx->inode_imagic_map); - e2fsck_pass1_free_bitmap(&thread_ctx->inode_reg_map); - e2fsck_pass1_free_bitmap(&thread_ctx->inodes_to_rebuild); - e2fsck_pass1_free_bitmap(&thread_ctx->block_found_map); - e2fsck_pass1_free_bitmap(&thread_ctx->block_ea_map); - if (thread_ctx->refcount) - ea_refcount_free(thread_ctx->refcount); - if (thread_ctx->refcount_extra) - ea_refcount_free(thread_ctx->refcount_extra); - if (thread_ctx->ea_inode_refs) - ea_refcount_free(thread_ctx->ea_inode_refs); - if (thread_ctx->refcount_orig) - ea_refcount_free(thread_ctx->refcount_orig); - e2fsck_free_dir_info(thread_ctx); - ext2fs_free_icount(thread_ctx->inode_count); - ext2fs_free_icount(thread_ctx->inode_link_info); - if (thread_ctx->dirs_to_hash) - ext2fs_badblocks_list_free(thread_ctx->dirs_to_hash); - quota_release_context(&thread_ctx->qctx); - ext2fs_free_mem(&thread_ctx->invalid_block_bitmap_flag); - ext2fs_free_mem(&thread_ctx->invalid_inode_bitmap_flag); - ext2fs_free_mem(&thread_ctx->invalid_inode_table_flag); + + /* + * @block_metadata_map and @block_dup_map are + * shared, so we don't free them. + */ + thread_ctx->block_metadata_map = NULL; + thread_ctx->block_dup_map = NULL; + e2fsck_reset_context(thread_ctx); ext2fs_free_mem(&thread_ctx); return retval; From patchwork Wed Nov 18 15:39:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402350 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=tBWtD8N5; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9T5ThGz9sW9 for ; Thu, 19 Nov 2020 02:41:49 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727571AbgKRPlt (ORCPT ); Wed, 18 Nov 2020 10:41:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727412AbgKRPls (ORCPT ); Wed, 18 Nov 2020 10:41:48 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7A45C0613D4 for ; Wed, 18 Nov 2020 07:41:48 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id u37so2942613ybi.15 for ; Wed, 18 Nov 2020 07:41:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=daegGDDQsSr9sVYzcaRO/85KlGjSfsbj2sAdl0WiLTY=; b=tBWtD8N5li5rsBCylTS2dSvTQ+lGC48YGbh8WFYT8v02IUfRKgDvGYwHDUKBOb9vs/ n+v8v7I/GqYbdPMEabzP7bpSg7We1ypuGNKT58BzOYwxOyQVSBMTnJu10wlBevmmXk+y 1SMlzpbASL26LA92XImA/9JETytzCEzSpJPJuybUMSAifAyk+qnybr+H4mRXTt2mXgI0 KLoHYslz4aMF+Gp5DaQ3Zo6y4KEHQDqiYbpw9bq/tmN+wHFgaGtjVOx0tfEMmrVuYGtk gSuJfkrK8hD/Q7dzWrgjRvSO7nMdPed1gpl1GY/XEFLTraZzAl3W67pZ3rHrbMxCBngf 3oUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=daegGDDQsSr9sVYzcaRO/85KlGjSfsbj2sAdl0WiLTY=; b=rw5bujHxau1Tvub47Fxy/u4derITxSsQ2u+f1skbRT23S7jjUh3WQmJ88G6daAcgCF ZzwgrVKIMp8eYHHM9D3vW0l3kewHPIRjB6nxL6K/n75y6+PupKJMc1x3IpkLmxepwBe8 wrqxXwR/1JdTKREWsReWDzP4eaycL3B2Iof/GJCqu5ST50xSlqI/DUt/4bVb9oXG0ebn p7x1S23NkteEr/FlpP/jO7jskwB3EAvejM3EZa98spFwUIg5LQMKor7d2BY6hjOgG+F8 mSOWOAuLBLosNIyMhU4UH4JBu0BgfKNV2Dd9SU1D8fZQSgpkxUNCIkLeKxViP2DduFKT kdAw== X-Gm-Message-State: AOAM531ryfu9iEp+5RHOVv7Ju539T9crQg46il59SDKHLD6wELmbcq40 QoYuteH+FkErAKWE3MuUl1IZOUKYwQgVBDJ7nG2leKyy2fgHCL8cQ8Ozss2HOWYm8C25yeMiu6R /nyxZj43kQ+foUESJTnUXyja2Fc47y9erpi9Dh3ug+k+VpNMJUvKUFyDUWd0NlQQ8vSPrmCKw5H whu8ny0Mg= X-Google-Smtp-Source: ABdhPJzte+9gQXYKYhQqN17c8hEWXKzQ4n50den2dujlm/cpnSzVCPAt8EYL8HZ2wCZxQ2225/jZd278gYcTeAfakL4= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:2605:: with SMTP id m5mr6624866ybm.98.1605714107888; Wed, 18 Nov 2020 07:41:47 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:30 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-45-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 44/61] e2fsck: avoid too much memory allocation for pfsck From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong e2fsck init memory according to filesystem inodes/dir numbers recorded in the superblock, this should be aware of filesystem number of threads, otherwise, oom happen. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- lib/ext2fs/dblist.c | 2 ++ lib/ext2fs/icount.c | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/lib/ext2fs/dblist.c b/lib/ext2fs/dblist.c index 1fdd8f43..c4e712fd 100644 --- a/lib/ext2fs/dblist.c +++ b/lib/ext2fs/dblist.c @@ -58,6 +58,8 @@ static errcode_t make_dblist(ext2_filsys fs, ext2_ino_t size, if (retval) goto cleanup; dblist->size = (num_dirs * 2) + 12; + if (fs->fs_num_threads) + dblist->size /= fs->fs_num_threads; } len = (size_t) sizeof(struct ext2_db_entry2) * dblist->size; dblist->count = count; diff --git a/lib/ext2fs/icount.c b/lib/ext2fs/icount.c index 766eccca..48665c7e 100644 --- a/lib/ext2fs/icount.c +++ b/lib/ext2fs/icount.c @@ -237,6 +237,8 @@ errcode_t ext2fs_create_icount_tdb(ext2_filsys fs EXT2FS_NO_TDB_UNUSED, * value. */ num_inodes = fs->super->s_inodes_count - fs->super->s_free_inodes_count; + if (fs->fs_num_threads) + num_inodes /= fs->fs_num_threads; icount->tdb = tdb_open(fn, num_inodes, TDB_NOLOCK | TDB_NOSYNC, O_RDWR | O_CREAT | O_TRUNC, 0600); @@ -288,6 +290,8 @@ errcode_t ext2fs_create_icount2(ext2_filsys fs, int flags, unsigned int size, if (retval) goto errout; icount->size += fs->super->s_inodes_count / 50; + if (fs->fs_num_threads) + icount->size /= fs->fs_num_threads; } bytes = (size_t) (icount->size * sizeof(struct ext2_icount_el)); From patchwork Wed Nov 18 15:39:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402351 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=n1QBNlOx; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9W4H8Bz9sVk for ; Thu, 19 Nov 2020 02:41:51 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727590AbgKRPlv (ORCPT ); Wed, 18 Nov 2020 10:41:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727412AbgKRPlu (ORCPT ); Wed, 18 Nov 2020 10:41:50 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9AD82C0613D4 for ; Wed, 18 Nov 2020 07:41:50 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id w8so2923730ybj.14 for ; Wed, 18 Nov 2020 07:41:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=DgnEoMuEidHu66jEvK6JXNOLWM5hp9t9dmgKJbWHLrI=; b=n1QBNlOxqeQgsE47DwPhyk00F3IKYVP7EoNUfNFHUiKuNLNq88etkqUYaf5EvnxQim 46bNYwAsct7u0kvPIBDN7fj+PSKgNniF391iumnEAmQkf/o3w20WjtePSJJvmRMrpBhd wZPOoe96kdxqT5wLaQ/ynPY6JswgbLzajOoG2pX6qg/69iFn1QN6A62rcfCnbbRFzBth 56iZ4WZrC8zZPG/B42vXPLVKu3gSTchryHUdZYBruCrPtGoes1pt7eh8RjMdWBvfcHKq ZKzt/4hfrPIqTgOUdRAgdS+U/pELwSapc9FaGebYE1iIhc5dxpc1YrNWF9pLiA0qPZ3r Wjxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=DgnEoMuEidHu66jEvK6JXNOLWM5hp9t9dmgKJbWHLrI=; b=H0AElu01TjQIikqoEUbMQqLTFr1Uj/09LzuYtq3nZNxn+oESC0SwlOW819tqZsxtAj UiICgpSJArLOc3sD5uUzSRKD9DuH52zm01WuKNYBRHKR+NDN91Bjvh+RDJwpoY7VzJvu 0ElkzV309bglA4idGp5yT5GILcSELVoUNPIYlZW947btw/1j1gKtwp0aaxwIEmQZugFH 4JcuN5l/QzaHw9MjpKEhJEjNoVpRFeEd2dHwDHlG6TjgxOpkzBXEcIjUQgTy1e1TTjVS xn/NnNtM5BTP60d2kbZoEkDTdWn3lJydlWQUlzgY3fWKiPvPo9qsfA0UsMiMFvfwcuwT QxEQ== X-Gm-Message-State: AOAM531GovnerJbDLb7LrhYu16mS7yOtZhC86oZToQEuHJEV72gci77r abF9h+bd9jEstFPBN9ownWk0ulYcA2RaqBf9EPRZu11wmA4GnFhQivDoZnZi7Wjcsj9aiwLriss yBixF9moif1K57KNt2Ko9gWjDURagMcRbh2W9MJ/V8+XXLv7AtjUY9LprUa7PJjNLjgVpirvqSQ RacHYjdNE= X-Google-Smtp-Source: ABdhPJxb8iZYvr6BQt8Axgnc0mL16S9wNbVO0E8sfSR+Im0kXoGT51HocAQCK4hWd4QqX6/jMX1Cpndx7FEHcDXBZz0= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:2c3:: with SMTP id 186mr7560017ybc.205.1605714109799; Wed, 18 Nov 2020 07:41:49 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:31 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-46-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 45/61] e2fsck: make default smallest RA size to 1M From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong If we have a smaller inodes per group, default ra size could be very small(etc 128KiB), this hurts performances. Tune above 128K to 1M, i see pass1 time drop down from 677.12 seconds to 246 secons with 32 threads. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/readahead.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/e2fsck/readahead.c b/e2fsck/readahead.c index 38d4ec42..40b73664 100644 --- a/e2fsck/readahead.c +++ b/e2fsck/readahead.c @@ -234,6 +234,8 @@ int e2fsck_can_readahead(ext2_filsys fs) return err != EXT2_ET_OP_NOT_SUPPORTED; } +#define MIN_DEFAULT_RA (1024 * 1024) + unsigned long long e2fsck_guess_readahead(ext2_filsys fs) { unsigned long long guess; @@ -245,6 +247,8 @@ unsigned long long e2fsck_guess_readahead(ext2_filsys fs) * in e2fsck runtime. */ guess = 2ULL * fs->blocksize * fs->inode_blocks_per_group; + if (guess < MIN_DEFAULT_RA) + guess = MIN_DEFAULT_RA; /* Disable RA if it'd use more 1/50th of RAM. */ if (get_memory_size() > (guess * 50)) From patchwork Wed Nov 18 15:39:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402352 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=VCjyRVQz; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9Y4fm6z9sVC for ; Thu, 19 Nov 2020 02:41:53 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727593AbgKRPlx (ORCPT ); Wed, 18 Nov 2020 10:41:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727412AbgKRPlw (ORCPT ); Wed, 18 Nov 2020 10:41:52 -0500 Received: from mail-qt1-x84a.google.com (mail-qt1-x84a.google.com [IPv6:2607:f8b0:4864:20::84a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB9C6C0613D4 for ; Wed, 18 Nov 2020 07:41:52 -0800 (PST) Received: by mail-qt1-x84a.google.com with SMTP id r29so1639244qtu.21 for ; Wed, 18 Nov 2020 07:41:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=IE9Hh3H2hu1A0A0Mt4wO8xcJjU6GkisyeoFBwJ3SNGY=; b=VCjyRVQzt+NNFZlvoOjKw7JcvyPD5sC1KpncWacxuSzSh9RdDRj3n4eXcMHFuKZpse TSF2le8PTGO/n70bWPHPG0Zk77Cj/utfOhCPxudNeua4FCa0PQgRqgLp7PjXlGiKf0JH pYoQ8piQz7UhuXooh/5Ef8WsDakV0yqjBe5NrAe5/fwDzcmpbM7lmzKGYHQahx9hxGtc KtBEVwvzRSTsHnBm4iujHu4JS3eLNiDp09szH04kGKjlfLQkgwhzYFVfY5bn5HhKTp2Y okbvmHGLzIMusLWfimsGJxE62FuW2Jqq3YiJR2ia+05CuxUg8zT28crXNLUH41mQ3o8x gUJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=IE9Hh3H2hu1A0A0Mt4wO8xcJjU6GkisyeoFBwJ3SNGY=; b=CctuBNMGALqqz1w6VIwP4jFYcX6hzsP5E3PAGtpVJPSRAG6jgsF/wp/nmRtNA4XNF6 2DEAXI47WXj3Y/IB6cQsexz2lVAuH8QDvPXSBwLjToJx0RZcg7BiFAQ51wGwnJsmFV+m alWgzBRBjlluk0uPQzXkzGER4RzvRSd3MCMlJ7GNgk74nnKklsbwIygdSXCIPcPLEMo2 GiYxvmyEaDKtNTHw6zmqdldoy7YTRC92/P28QmwXKFx5ASmPTpdCh8P3QfPBImnKHhVZ 9rvhc8Wimfw23AjXWlsdVyOj/IQ2etPDvr//D/0JQDFmr8w9SluUW+gKOVCUlej/e7ht 6PTQ== X-Gm-Message-State: AOAM531w+l0KmW7xeBywF2RILbsjxp2cgksELSy9nHCf+nijIDBmrUiQ AlK7cDrSSwqlEgX6PkyZcNVeN9rkCtXRVt4YtDkcbmoK9bB8NffRDYJWmoqkN5sWjKxbloC55ZX blBqrF2RqHyBMBCAJbeq0Nqz/I9vUH3ojbGcXxq3TnEvhGDp/LHlfRVzylhfzmmT8zOfM2ODA26 hxsqX1+oA= X-Google-Smtp-Source: ABdhPJyjUVHvp9uendqEMskWzMUR9evEWDOXGIWWo14TC2Bvg/jlJSafjX3grYP/1RJvkS2Hca703sSSHVvZyCBl7vQ= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:ab1e:: with SMTP id h30mr5140936qvb.55.1605714111776; Wed, 18 Nov 2020 07:41:51 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:32 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-47-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 46/61] ext2fs: parallel bitmap loading From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong In our benchmarking for PiB size filesystem, pass5 takes 10446s to finish and 99.5% of time takes on reading bitmaps. It makes sense to reading bitmaps using multiple threads, a quickly benchmark show 10446s to 626s with 64 threads. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- lib/ext2fs/rw_bitmaps.c | 289 ++++++++++++++++++++++++++++++++++------ 1 file changed, 250 insertions(+), 39 deletions(-) diff --git a/lib/ext2fs/rw_bitmaps.c b/lib/ext2fs/rw_bitmaps.c index d80c9eb8..960a6913 100644 --- a/lib/ext2fs/rw_bitmaps.c +++ b/lib/ext2fs/rw_bitmaps.c @@ -23,6 +23,7 @@ #ifdef HAVE_SYS_TYPES_H #include #endif +#include #include "ext2_fs.h" #include "ext2fs.h" @@ -205,22 +206,12 @@ static int bitmap_tail_verify(unsigned char *bitmap, int first, int last) return 1; } -static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) +static errcode_t read_bitmaps_range_prepare(ext2_filsys fs, int do_inode, int do_block) { - dgrp_t i; - char *block_bitmap = 0, *inode_bitmap = 0; - char *buf; errcode_t retval; int block_nbytes = EXT2_CLUSTERS_PER_GROUP(fs->super) / 8; int inode_nbytes = EXT2_INODES_PER_GROUP(fs->super) / 8; - int tail_flags = 0; - int csum_flag; - unsigned int cnt; - blk64_t blk; - blk64_t blk_itr = EXT2FS_B2C(fs, fs->super->s_first_data_block); - blk64_t blk_cnt; - ext2_ino_t ino_itr = 1; - ext2_ino_t ino_cnt; + char *buf; EXT2_CHECK_MAGIC(fs, EXT2_ET_MAGIC_EXT2FS_FILSYS); @@ -230,11 +221,10 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) fs->write_bitmaps = ext2fs_write_bitmaps; - csum_flag = ext2fs_has_group_desc_csum(fs); - retval = ext2fs_get_mem(strlen(fs->device_name) + 80, &buf); if (retval) return retval; + if (do_block) { if (fs->block_map) ext2fs_free_block_bitmap(fs->block_map); @@ -243,11 +233,8 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) retval = ext2fs_allocate_block_bitmap(fs, buf, &fs->block_map); if (retval) goto cleanup; - retval = io_channel_alloc_buf(fs->io, 0, &block_bitmap); - if (retval) - goto cleanup; - } else - block_nbytes = 0; + } + if (do_inode) { if (fs->inode_map) ext2fs_free_inode_bitmap(fs->inode_map); @@ -256,13 +243,69 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) retval = ext2fs_allocate_inode_bitmap(fs, buf, &fs->inode_map); if (retval) goto cleanup; - retval = io_channel_alloc_buf(fs->io, 0, &inode_bitmap); + } + ext2fs_free_mem(&buf); + + return retval; + +cleanup: + if (do_block) { + ext2fs_free_block_bitmap(fs->block_map); + fs->block_map = 0; + } + if (do_inode) { + ext2fs_free_inode_bitmap(fs->inode_map); + fs->inode_map = 0; + } + if (buf) + ext2fs_free_mem(&buf); + return retval; +} + +static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_block, + dgrp_t start, dgrp_t end, pthread_mutex_t *mutex, + io_channel io) +{ + dgrp_t i; + char *block_bitmap = 0, *inode_bitmap = 0; + char *buf; + errcode_t retval; + int block_nbytes = EXT2_CLUSTERS_PER_GROUP(fs->super) / 8; + int inode_nbytes = EXT2_INODES_PER_GROUP(fs->super) / 8; + int tail_flags = 0; + int csum_flag; + unsigned int cnt; + blk64_t blk; + blk64_t blk_itr = EXT2FS_B2C(fs, fs->super->s_first_data_block); + blk64_t blk_cnt; + ext2_ino_t ino_itr = 1; + ext2_ino_t ino_cnt; + io_channel this_io; + + if (!io) + this_io = fs->io; + else + this_io = io; + + csum_flag = ext2fs_has_group_desc_csum(fs); + + if (do_block) { + retval = io_channel_alloc_buf(this_io, 0, &block_bitmap); + if (retval) + goto cleanup; + } else { + block_nbytes = 0; + } + + if (do_inode) { + retval = io_channel_alloc_buf(this_io, 0, &inode_bitmap); if (retval) goto cleanup; - } else + } else { inode_nbytes = 0; - ext2fs_free_mem(&buf); + } + /* io should be null */ if (fs->flags & EXT2_FLAG_IMAGE_FILE) { blk = (ext2fs_le32_to_cpu(fs->image_header->offset_inodemap) / fs->blocksize); ino_cnt = fs->super->s_inodes_count; @@ -303,7 +346,9 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) goto success_cleanup; } - for (i = 0; i < fs->group_desc_count; i++) { + blk_itr += ((blk64_t)start * (block_nbytes << 3)); + ino_itr += ((blk64_t)start * (inode_nbytes << 3)); + for (i = start; i <= end; i++) { if (block_bitmap) { blk = ext2fs_block_bitmap_loc(fs, i); if ((csum_flag && @@ -312,7 +357,7 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) (blk >= ext2fs_blocks_count(fs->super))) blk = 0; if (blk) { - retval = io_channel_read_blk64(fs->io, blk, + retval = io_channel_read_blk64(this_io, blk, 1, block_bitmap); if (retval) { retval = EXT2_ET_BLOCK_BITMAP_READ; @@ -333,8 +378,12 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) } else memset(block_bitmap, 0, block_nbytes); cnt = block_nbytes << 3; + if (mutex) + pthread_mutex_lock(mutex); retval = ext2fs_set_block_bitmap_range2(fs->block_map, blk_itr, cnt, block_bitmap); + if (mutex) + pthread_mutex_unlock(mutex); if (retval) goto cleanup; blk_itr += block_nbytes << 3; @@ -347,7 +396,7 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) (blk >= ext2fs_blocks_count(fs->super))) blk = 0; if (blk) { - retval = io_channel_read_blk64(fs->io, blk, + retval = io_channel_read_blk64(this_io, blk, 1, inode_bitmap); if (retval) { retval = EXT2_ET_INODE_BITMAP_READ; @@ -369,29 +418,28 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) } else memset(inode_bitmap, 0, inode_nbytes); cnt = inode_nbytes << 3; + if (mutex) + pthread_mutex_lock(mutex); retval = ext2fs_set_inode_bitmap_range2(fs->inode_map, ino_itr, cnt, inode_bitmap); + if (mutex) + pthread_mutex_unlock(mutex); if (retval) goto cleanup; ino_itr += inode_nbytes << 3; } } - /* Mark group blocks for any BLOCK_UNINIT groups */ - if (do_block) { - retval = mark_uninit_bg_group_blocks(fs); - if (retval) - goto cleanup; - } - success_cleanup: - if (inode_bitmap) { - ext2fs_free_mem(&inode_bitmap); - fs->flags &= ~EXT2_FLAG_IBITMAP_TAIL_PROBLEM; - } - if (block_bitmap) { - ext2fs_free_mem(&block_bitmap); - fs->flags &= ~EXT2_FLAG_BBITMAP_TAIL_PROBLEM; + if (start == 0 && end == fs->group_desc_count - 1) { + if (inode_bitmap) { + ext2fs_free_mem(&inode_bitmap); + fs->flags &= ~EXT2_FLAG_IBITMAP_TAIL_PROBLEM; + } + if (block_bitmap) { + ext2fs_free_mem(&block_bitmap); + fs->flags &= ~EXT2_FLAG_BBITMAP_TAIL_PROBLEM; + } } fs->flags |= tail_flags; return 0; @@ -412,6 +460,169 @@ cleanup: if (buf) ext2fs_free_mem(&buf); return retval; + +} + +static errcode_t read_bitmaps_range_end(ext2_filsys fs, int do_inode, int do_block) +{ + errcode_t retval = 0; + + /* Mark group blocks for any BLOCK_UNINIT groups */ + if (do_block) { + retval = mark_uninit_bg_group_blocks(fs); + if (retval) + goto cleanup; + } + + return retval; +cleanup: + if (do_block) { + ext2fs_free_block_bitmap(fs->block_map); + fs->block_map = 0; + } + if (do_inode) { + ext2fs_free_inode_bitmap(fs->inode_map); + fs->inode_map = 0; + } + return retval; +} + +static errcode_t read_bitmaps_range(ext2_filsys fs, int do_inode, int do_block, + dgrp_t start, dgrp_t end) +{ + errcode_t retval; + + retval = read_bitmaps_range_prepare(fs, do_inode, do_block); + if (retval) + return retval; + + retval = read_bitmaps_range_start(fs, do_inode, do_block, start, end, NULL, NULL); + if (retval) + return retval; + + return read_bitmaps_range_end(fs, do_inode, do_block); +} + +#ifdef CONFIG_PFSCK +struct read_bitmaps_thread_info { + ext2_filsys rbt_fs; + int rbt_do_inode; + int rbt_do_block; + dgrp_t rbt_grp_start; + dgrp_t rbt_grp_end; + errcode_t rbt_retval; + pthread_mutex_t *rbt_mutex; + io_channel rbt_io; +}; + +static void* read_bitmaps_thread(void *data) +{ + struct read_bitmaps_thread_info *rbt = data; + + rbt->rbt_retval = read_bitmaps_range_start(rbt->rbt_fs, + rbt->rbt_do_inode, rbt->rbt_do_block, + rbt->rbt_grp_start, rbt->rbt_grp_end, + rbt->rbt_mutex, rbt->rbt_io); + return NULL; +} +#endif + +static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) +{ +#ifdef CONFIG_PFSCK + pthread_attr_t attr; + int num_threads = fs->fs_num_threads; + pthread_t *thread_ids = NULL; + struct read_bitmaps_thread_info *thread_infos = NULL; + pthread_mutex_t rbt_mutex = PTHREAD_MUTEX_INITIALIZER; + errcode_t retval; + errcode_t rc; + dgrp_t average_group; + int i; + io_manager manager = unix_io_manager; +#else + int num_threads = 1; +#endif + + if (num_threads <= 1 || (fs->flags & EXT2_FLAG_IMAGE_FILE)) + return read_bitmaps_range(fs, do_inode, do_block, 0, fs->group_desc_count - 1); + +#ifdef CONFIG_PFSCK + retval = pthread_attr_init(&attr); + if (retval) + return retval; + + thread_ids = calloc(sizeof(pthread_t), num_threads); + if (!thread_ids) + return -ENOMEM; + + thread_infos = calloc(sizeof(struct read_bitmaps_thread_info), + num_threads); + if (!thread_infos) + goto out; + + average_group = ext2fs_get_avg_group(fs); + retval = read_bitmaps_range_prepare(fs, do_inode, do_block); + if (retval) + goto out; + + fprintf(stdout, "Multiple threads triggered to read bitmaps\n"); + for (i = 0; i < num_threads; i++) { + thread_infos[i].rbt_fs = fs; + thread_infos[i].rbt_do_inode = do_inode; + thread_infos[i].rbt_do_block = do_block; + thread_infos[i].rbt_mutex = &rbt_mutex; + if (i == 0) + thread_infos[i].rbt_grp_start = 0; + else + thread_infos[i].rbt_grp_start = average_group * i + 1; + + if (i == num_threads - 1) + thread_infos[i].rbt_grp_end = fs->group_desc_count - 1; + else + thread_infos[i].rbt_grp_end = average_group * (i + 1); + retval = manager->open(fs->device_name, IO_FLAG_RW, + &thread_infos[i].rbt_io); + if (retval) + break; + io_channel_set_blksize(thread_infos[i].rbt_io, fs->io->block_size); + retval = pthread_create(&thread_ids[i], &attr, + &read_bitmaps_thread, &thread_infos[i]); + if (retval) { + io_channel_close(thread_infos[i].rbt_io); + break; + } + } + for (i = 0; i < num_threads; i++) { + if (!thread_ids[i]) + break; + rc = pthread_join(thread_ids[i], NULL); + if (rc && !retval) + retval = rc; + rc = thread_infos[i].rbt_retval; + if (rc && !retval) + retval = rc; + io_channel_close(thread_infos[i].rbt_io); + } +out: + rc = pthread_attr_destroy(&attr); + if (rc && !retval) + retval = rc; + free(thread_infos); + free(thread_ids); + + if (!retval) + retval = read_bitmaps_range_end(fs, do_inode, do_block); + + if (!retval) { + if (do_inode) + fs->flags &= ~EXT2_FLAG_IBITMAP_TAIL_PROBLEM; + if (do_block) + fs->flags &= ~EXT2_FLAG_BBITMAP_TAIL_PROBLEM; + } + + return retval; +#endif } errcode_t ext2fs_read_inode_bitmap(ext2_filsys fs) From patchwork Wed Nov 18 15:39:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402353 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=FHWYN3OW; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9c5wQWz9sTc for ; Thu, 19 Nov 2020 02:41:56 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727605AbgKRPl4 (ORCPT ); Wed, 18 Nov 2020 10:41:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727412AbgKRPly (ORCPT ); Wed, 18 Nov 2020 10:41:54 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBDC1C0613D4 for ; Wed, 18 Nov 2020 07:41:54 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id i6so1514951pgg.10 for ; Wed, 18 Nov 2020 07:41:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=GjJeR3HmrSAYgbKKIohYNzytTHCLPFhyiKxa4ICxu0I=; b=FHWYN3OWgUJiszB8swLc52iCR910jAwz6e23BtVOWyZdVt9Qe++6yMOlUkzDepEz7X TJ9GaUm6pkaKLgozHBJhlhVhJ3mmxIcSUfoKrq/LC7f3AWT9rcxOoelaJ3Yfn+jyDrO3 1Jh7UrOev9o6HRr5/N8kst/I5TCHywlDq8CmHANJMTf5egkN3sShG6BIlnChqr/EkoNJ C8GB7eEq/Pt8/cMO/gYGqXt7lpOFtFY3tDfUdJSIDOuOf9ITHmQ1ZRteav/CGl46Y1oA 1pHpyrD+6oliOsWbixPKe+/uj2g7lctdskwGGPLKlyjPWJeoh5UrfldGkWzSpS2fhRDu s65A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=GjJeR3HmrSAYgbKKIohYNzytTHCLPFhyiKxa4ICxu0I=; b=VnkURGYIB0dp6I3avHfhrk/Isd0anIC+4kZFmqqqNVNY8WQqLfjvfDgm7LLCDx/chA s1dN0fzWTNpM5LNGgo6VnzuE81979HRmtXwZ5/cL7YQutvEX5vtyobEdU1CcQgwPqPYs BtvjyK0WCYIdOdOTDV1DZ5zwnVC3aKpfIFzw1T3mLsu7jeIuu2JfMyauVITqc7cgiDHJ 9g8rBv6onKqiWRgQ56i5ECv3ktcA8LN9vP5z5q3AKyM+8hsA27IkQCWbcaVTzS5OpGlP 5FdRfKbAHP9q9IunfppJL7NJoX6er4XEYax/yETpYSmoS06jo9xeAZR8scG/tT5GGF8x CA7Q== X-Gm-Message-State: AOAM533V6lvYQpS7ZvE5CPSCHfYXa/DrfaPbAeaTr7A2pdfnOI9N50ts XAKQLY1fYjiDpTdOgaWfPnemqpomAYKxUhLEBxSkQ7hkfvPebN6ZIVjNhbo6S81IyNN5L8ycxCR TgKjrUl5WgAMvo7cKf1aj3/zlbmeFsijOE+YbNmy+yvA2TCRY9Cp1ZyqjAS4P5udpdz2xNCwOFW yq9lmda7U= X-Google-Smtp-Source: ABdhPJyKMCxL4ddnwWmP9DD5L5/iA35fwH0BsmhJbkqLXKnUW6+CBox2u3+LpOLRFtNZ5SO995DwtTpZzS8tDj62YHA= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:90b:e04:: with SMTP id ge4mr46486pjb.0.1605714113740; Wed, 18 Nov 2020 07:41:53 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:33 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-48-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 47/61] e2fsck: update mmp block in one thread From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong For multiple threads, different threads will try to update mmp block at the same time, only allow one thread to update it. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 1 + e2fsck/pass1.c | 36 ++++++++++++++++++++++++++++++++++-- 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index a66772c1..1469c3e1 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -451,6 +451,7 @@ struct e2fsck_struct { char *undo_file; #ifdef CONFIG_PFSCK __u32 fs_num_threads; + __u32 mmp_update_thread; int fs_need_locking; /* serialize fix operation for multiple threads */ pthread_rwlock_t fs_fix_rwlock; diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 60f70111..e98cda9f 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1497,7 +1497,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) dgrp_t ra_group = 0; struct ea_quota ea_ibody_quota; struct process_inode_block *inodes_to_process; - int process_inode_count; + int process_inode_count, check_mmp; init_resource_track(&rtrack, ctx->fs->io); clear_problem_context(&pctx); @@ -1646,8 +1646,33 @@ void e2fsck_pass1_run(e2fsck_t ctx) #endif while (1) { + check_mmp = 0; e2fsck_pass1_check_lock(ctx); - if (ino % (fs->super->s_inodes_per_group * 4) == 1) { +#ifdef CONFIG_PFSCK + if (!ctx->mmp_update_thread) { + e2fsck_pass1_block_map_w_lock(ctx); + if (!ctx->mmp_update_thread) { + if (ctx->global_ctx) + ctx->mmp_update_thread = + ctx->thread_info.et_thread_index + 1; + else + ctx->mmp_update_thread = 1; + check_mmp = 1; + } + e2fsck_pass1_block_map_w_unlock(ctx); + } + + /* only one active thread could update mmp block. */ + e2fsck_pass1_block_map_r_lock(ctx); + if (!ctx->global_ctx || ctx->mmp_update_thread == + (ctx->thread_info.et_thread_index + 1)) + check_mmp = 1; + e2fsck_pass1_block_map_r_unlock(ctx); +#else + check_mmp = 1; +#endif + + if (check_mmp && (ino % (fs->super->s_inodes_per_group * 4) == 1)) { if (e2fsck_mmp_update(fs)) fatal_error(ctx, 0); } @@ -2365,6 +2390,13 @@ endit: print_resource_track(ctx, _("Pass 1"), &rtrack, ctx->fs->io); else ctx->invalid_bitmaps++; +#ifdef CONFIG_PFSCK + /* reset update_thread after this thread exit */ + e2fsck_pass1_block_map_w_lock(ctx); + if (ctx->mmp_update_thread) + ctx->mmp_update_thread = 0; + e2fsck_pass1_block_map_w_unlock(ctx); +#endif } #ifdef CONFIG_PFSCK From patchwork Wed Nov 18 15:39:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402354 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=BDECphOE; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9g3Q4Gz9sTc for ; Thu, 19 Nov 2020 02:41:59 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727611AbgKRPl7 (ORCPT ); Wed, 18 Nov 2020 10:41:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727412AbgKRPl4 (ORCPT ); Wed, 18 Nov 2020 10:41:56 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B36D3C0613D4 for ; Wed, 18 Nov 2020 07:41:56 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id k7so2894169ybm.13 for ; Wed, 18 Nov 2020 07:41:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=UTLg+CGayr6X9HdLF8+fpEo3MODBqTn3f897ddxuEjs=; b=BDECphOEIpEXkOKBYjk/D/nv8Pm1Fa/e9bN6pkg+LtHTdcpbv328LDo3TQObT9M3J5 lV6STdBylxmP6QE/Dp49NkjQ3qKtHV03o7jAApprSamBexGFzluZiZ5YbdRGXLBnl/nQ fl6jfxBwUsNbhEZ45EuYCz2/43vP8MON3ze1SdGetmoYLjRhNP7JLFxKXv9ymfOEPCWa CVIwbA4ii+apOdX0zCJtIq2NsrSN7d7dkSknEzyTfJ4ZTNGqiO6Q5b6bO2CuEE+fMG4s mCey+PZOJM91Cn9G2kH1rAEQWBJQomzU1A4N//o/uwjWL12qsBbGE0RAiLw2gEyqNDV1 bLuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=UTLg+CGayr6X9HdLF8+fpEo3MODBqTn3f897ddxuEjs=; b=oWS5af2FBaWhW+hPhYNJwXqelGdmbgTphGbVH6HQnkaCfbggilrj6lCaZ0p8rtXG1e 8iHVIH6Cw/A8wsFTyFJgeHyfpbeqLzwpgbmNF7f5wjRoJ9wFjHEPJcm7gjLwXnqU/Lcf eeLerqsBIXm2wNjNArRQGBor6/rIwpi2omkCJg1H0bvaLsFjRYj+Q/yTJ8EuVpJ6KOxv L0AIN6xiDW1/uRuqBRPVb4l81z1lwJGRj24fFHw0M6tteQfWu7JxGd6gmewqYBDmktYE 29L8MT7Qc+uF//23IjM4/p9YI3YAHwfg/5BQgllD3kN1GGDaTjyrbjt3k8447UsypOeL RJ0g== X-Gm-Message-State: AOAM532XfhoxXqF64o1A3xNkfO4z3mQz2W4T2I5BYNianB7fXb3zmmAz OnA4niKl3FOutzcwcMSOTFhPzE5MtXRl5Ra7Zd/I3TimHSG5Lmu2gci6RCtzmrQqK6Bo9/ob4XE ZbcTlBBpd4bnJ9NMxPBWMaMiEuk6DgsYCC1oOjGymh8eMH50pWouXWU4pWeuIx0RX3guePiWrCV xLX5KOYTw= X-Google-Smtp-Source: ABdhPJw+EPF+egPC2g7jXAP3AWlf4o/flFAvAS4PD+0wCsL8S9NuQM8/TenVfkQvZgquwE8h7ZrUiA8CSpe3KRUW50o= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:874e:: with SMTP id e14mr7372803ybn.112.1605714115826; Wed, 18 Nov 2020 07:41:55 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:34 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-49-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 48/61] e2fsck: reset @inodes_to_rebuild if restart From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Verify multiple thread on a corrupted images hit following bug: pass1.c:2902: e2fsck_pass1_thread_prepare: Assertion `global_ctx->inodes_to_rebuild == NULL' failed. Signal (6) SIGABRT si_code=SI_TKILL ./e2fsck/e2fsck[0x43829e] /lib64/libpthread.so.0(+0x14b20)[0x7f3b45135b20] /lib64/libc.so.6(gsignal+0x145)[0x7f3b44f2c625] /lib64/libc.so.6(abort+0x12b)[0x7f3b44f158d9] /lib64/libc.so.6(+0x257a9)[0x7f3b44f157a9] /lib64/libc.so.6(+0x34a66)[0x7f3b44f24a66] ./e2fsck/e2fsck(e2fsck_pass1+0x1662)[0x423572] ./e2fsck/e2fsck(e2fsck_run+0x5a)[0x41611a] ./e2fsck/e2fsck(main+0x1608)[0x4121b8] /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3b44f171a3] ./e2fsck/e2fsck(_start+0x2e)[0x413dde] @inodes_to_rebuild could be not NULL after we restart pass1 Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/e2fsck/e2fsck.c b/e2fsck/e2fsck.c index 1fd57504..a03550c0 100644 --- a/e2fsck/e2fsck.c +++ b/e2fsck/e2fsck.c @@ -75,6 +75,10 @@ errcode_t e2fsck_reset_context(e2fsck_t ctx) ext2fs_free_block_bitmap(ctx->block_found_map); ctx->block_found_map = 0; } + if (ctx->inodes_to_rebuild) { + ext2fs_free_inode_bitmap(ctx->inodes_to_rebuild); + ctx->inodes_to_rebuild = 0; + } if (ctx->inode_link_info) { ext2fs_free_icount(ctx->inode_link_info); ctx->inode_link_info = 0; From patchwork Wed Nov 18 15:39:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402355 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=pLOuQcrA; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9h05J9z9sVC for ; Thu, 19 Nov 2020 02:42:00 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727614AbgKRPl7 (ORCPT ); Wed, 18 Nov 2020 10:41:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727608AbgKRPl6 (ORCPT ); Wed, 18 Nov 2020 10:41:58 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70E2EC0613D6 for ; Wed, 18 Nov 2020 07:41:58 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id f5so1403153pfa.18 for ; Wed, 18 Nov 2020 07:41:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=JxnBQe88GuM4rIndOAjxTgYOiO3eiDXcKskI+rG/+KI=; b=pLOuQcrASmKWhw907+WAg4vizEODBbT5/xBrU5Qj6k+QZTgsYannD7PloWovrvTJL9 Xs0U5XSw6D9z0A44heOuqZKxCdCexP7kfLD2imZIYbd2N9nKxIwfrTBQc/gCM/2cGgXA zDsnMEuBRPz8QAEpT2Qyw4Apjo9ZqJC1QBz9T8LxdlmHyNmIJet+2WUMnE1W4Nz3WNtA LiyDRx9sJuAgJacovVT21l6A+xVlpGj2mvlk6vJVNXx9XCs0kUzyWLQp/q6tleAB/KVL b4e2TJqCHxOQbhfapbDcKtBH3O7OoEonhOvF51IEDVKTPo+q9bQd23qZdbIqK0EMMfYK mX5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=JxnBQe88GuM4rIndOAjxTgYOiO3eiDXcKskI+rG/+KI=; b=ALPdrEZK9gZW7bSePBJNQkSm6BWb8zjOlvk7UCVgbZ2iwzLvsAgkVnrrf3acx8lRAN 4tO360GWNiwDX6n37G2Uy0ovaOcfdRKR1wJdnU7IYoi34QnrotlAidNAoh0WapaJAw3Q UGVCVPabdSI1eYJeswmF+inRzDXNY9bOHF8mjiVAVolH/ox9Lvv/3hMrJKE6C/muhtJW m+yqafn4byfiuBKlHOa6VJ7fUVK6CQMQQ4PXKy3BY41Z8xLJV6XZHMeLSZB2TS31USmA vkgqKawdvf+41vIcBz4NYSHz0bLQDPvVeDCeXTomMlBhajYOy7wTMvko1qoj6KavqptG XeVg== X-Gm-Message-State: AOAM530AZebbr9xCCmKJGl2FfH2sVZcKmVCbTnFVUOoYVqmAsF4sPuKF pjhMdX5G5udfYNdvF4UxJBjA7u9tp7JnTCS0moFmgqiyabSyTVA1+SIib0pQTNotbuoH9ZoPfNe 8q6L7c24azVdsN4U+AAKfZ9Tf0M2fdRmSdGpoG7oBZL0NpPm6x95HQHbX92+0SRCVel7uTRODDV 2xTaZpgjw= X-Google-Smtp-Source: ABdhPJw5M/pssc0+RyTdq9zuIB9H2G4AgUOhKSAk8p01NbynalVUMbedC52CStU3pU1kjHSPHeYvzo2b2XzNz6ZWaFw= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:902:c154:b029:d6:efa5:4ce7 with SMTP id 20-20020a170902c154b02900d6efa54ce7mr4893452plj.73.1605714117781; Wed, 18 Nov 2020 07:41:57 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:35 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-50-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 49/61] e2fsck: fix build for make rpm From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong link e2fsck with -lpthread properly to make rpm pass. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- MCONFIG.in | 1 + e2fsck/Makefile.in | 9 +++++---- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/MCONFIG.in b/MCONFIG.in index 0598f21b..a85e24f1 100644 --- a/MCONFIG.in +++ b/MCONFIG.in @@ -146,6 +146,7 @@ DEPLIBUUID = @DEPLIBUUID@ DEPLIBSUPPORT = $(LIB)/libsupport@STATIC_LIB_EXT@ DEPLIBBLKID = @DEPLIBBLKID@ @PRIVATE_LIBS_CMT@ $(DEPLIBUUID) TESTENV = LD_LIBRARY_PATH="$(LIB):$${LD_LIBRARY_PATH}" DYLD_LIBRARY_PATH="$(LIB):$${DYLD_LIBRARY_PATH}" +SEM_INIT_LIB = @SEM_INIT_LIB@ STATIC_LIBSS = $(LIB)/libss@STATIC_LIB_EXT@ @DLOPEN_LIB@ STATIC_LIBCOM_ERR = $(LIB)/libcom_err@STATIC_LIB_EXT@ @SEM_INIT_LIB@ diff --git a/e2fsck/Makefile.in b/e2fsck/Makefile.in index 6e25d27d..0ec11952 100644 --- a/e2fsck/Makefile.in +++ b/e2fsck/Makefile.in @@ -16,20 +16,21 @@ MANPAGES= e2fsck.8 FMANPAGES= e2fsck.conf.5 LIBS= $(LIBSUPPORT) $(LIBEXT2FS) $(LIBCOM_ERR) $(LIBBLKID) $(LIBUUID) \ - $(LIBINTL) $(LIBE2P) $(LIBMAGIC) $(SYSLIBS) + $(LIBINTL) $(LIBE2P) $(LIBMAGIC) $(SYSLIBS) $(SEM_INIT_LIB) DEPLIBS= $(DEPLIBSUPPORT) $(LIBEXT2FS) $(DEPLIBCOM_ERR) $(DEPLIBBLKID) \ $(DEPLIBUUID) $(DEPLIBE2P) STATIC_LIBS= $(STATIC_LIBSUPPORT) $(STATIC_LIBEXT2FS) $(STATIC_LIBCOM_ERR) \ $(STATIC_LIBBLKID) $(STATIC_LIBUUID) $(LIBINTL) $(STATIC_LIBE2P) \ - $(LIBMAGIC) $(SYSLIBS) + $(LIBMAGIC) $(SYSLIBS) $(SEM_INIT_LIB) STATIC_DEPLIBS= $(DEPSTATIC_LIBSUPPORT) $(STATIC_LIBEXT2FS) \ $(DEPSTATIC_LIBCOM_ERR) $(DEPSTATIC_LIBBLKID) \ $(DEPSTATIC_LIBUUID) $(DEPSTATIC_LIBE2P) PROFILED_LIBS= $(PROFILED_LIBSUPPORT) $(PROFILED_LIBEXT2FS) \ $(PROFILED_LIBCOM_ERR) $(PROFILED_LIBBLKID) $(PROFILED_LIBUUID) \ - $(PROFILED_LIBE2P) $(LIBINTL) $(LIBMAGIC) $(SYSLIBS) + $(PROFILED_LIBE2P) $(LIBINTL) $(LIBMAGIC) $(SYSLIBS) \ + $(SEM_INIT_LIB) PROFILED_DEPLIBS= $(DEPPROFILED_LIBSUPPORT) $(PROFILED_LIBEXT2FS) \ $(DEPPROFILED_LIBCOM_ERR) $(DEPPROFILED_LIBBLKID) \ $(DEPPROFILED_LIBUUID) $(DEPPROFILED_LIBE2P) @@ -115,7 +116,7 @@ all-static:: e2fsck.static e2fsck: $(OBJS) $(DEPLIBS) $(E) " LD $@" - $(Q) $(LD) $(ALL_LDFLAGS) $(RDYNAMIC) -o e2fsck $(OBJS) $(LIBS) + $(Q) $(LD) $(ALL_LDFLAGS) $(RDYNAMIC) -o e2fsck $(OBJS) $(LIBS) e2fsck.static: $(OBJS) $(STATIC_DEPLIBS) $(E) " LD $@" From patchwork Wed Nov 18 15:39:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402356 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=SZ3nQjcF; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9j2qxDz9sTc for ; Thu, 19 Nov 2020 02:42:01 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727621AbgKRPmB (ORCPT ); Wed, 18 Nov 2020 10:42:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727608AbgKRPmA (ORCPT ); Wed, 18 Nov 2020 10:42:00 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 820EFC0613D4 for ; Wed, 18 Nov 2020 07:42:00 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id u37so2943100ybi.15 for ; Wed, 18 Nov 2020 07:42:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=uaDW3e45uHJQl6bXyNwdqfI0txtMN4R0imzv8FDVqaE=; b=SZ3nQjcFUO1GhfYRA8oGh7U5oXDMKJLyQLT+8/KawAuY0VhJ0vkfBiN8U5ZxsM/6nP 3WgkzPmSVmpOQ2LhD9HeOracNyKrCGgzE/JxswH8DTUv1ahuPt/pUEtN7IOsx3PeyCR3 KAkDIdG1tOHNZ4WFfpEtV7bIC2hfkB0B/u+ITtCmKIQVAdbgwlRvenKZy/sjVDCMwzss 82AfWyqdNnOm3tT8oZo1tKyJlStL06MbdslxV8h0/JKMOPyN28QPsnz5slTWs4JxkTUx EpFNDbdFNN4hcd/eaU7okWaUeyV2t+IxjTZ4bDW+3TRWydnUo9deEQMN2ZUuCyojs+BK pESw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=uaDW3e45uHJQl6bXyNwdqfI0txtMN4R0imzv8FDVqaE=; b=UDZuOKh3wwEcASxp8Zl8AiIYJe5nfY4HE+FqK8CPYG1PM1hycgdgq4CKUlIXH5XBrY /5Z7NcrpIal688aR2wRgLbtPluwYh+fbm7YmzpwzqzCJuaJd9Y5jbuCVEeGfyAdpo+Ei pqh4puraQvIlKaeCMhTmM8lL+nyY8uMtIwTvM1EquCYgzBUq6ZO7ss864df+QelQH2h1 7GHJ3tDOChrgBdi5JVrDKCcWu2lDR9KSz/hiEHFyloMc6MRdKObOhgH/h2D0MmaDsTxe PvLf9C4ry6ek4kJMXbVK3ab26V/AQbYjqyjZEYW6xIa3qng58aUKgyljHlP66xINWw9H 42qw== X-Gm-Message-State: AOAM533Y0v1lP5LS17IzVH0BDR6a1ph7Igfq8ed4q0R+jTp7Ge/Oe1Ds 1J7q6CJsD9iaZm16q3fl8Z3erB1hE8kKuxrcWZWy1rPBl4VV9Walj6NhNkbt6zcbguTgYX7rOBm ZzsfwEo1Pm/gkat4Qq+2qsSkZ23LrWoxYk9V1OgGOm344GcqUCF2Yae/m25DM3yW22F6/+htvUO AB0BYi+II= X-Google-Smtp-Source: ABdhPJwtPYtoiaKg8sXW5PpaR8nXR5vaHcQYN5kD044RhlguA8A9T/g+7WCnmO/PkYb+ZqgECsnjB65E/bbQG7Pr11I= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:2e0d:: with SMTP id u13mr7449724ybu.247.1605714119662; Wed, 18 Nov 2020 07:41:59 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:36 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-51-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 50/61] e2fsck: move ext2fs_get_avg_group to rw_bitmaps.c From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , jenkins , Maloo , Andreas Dilger , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong A bounch of ext2fs_get_avg_group() unused warning messages are annoying, move it to rw_bitmaps.c to make gcc happy. Signed-off-by: Wang Shilong Change-Id: Ia7d372b7b5aadcbf5d94916f6f67363a2a9f0bfa Reviewed-on: https://review.whamcloud.com/40060 Tested-by: jenkins Tested-by: Maloo Reviewed-by: Andreas Dilger Signed-off-by: Saranya Muruganandam --- lib/ext2fs/ext2fs.h | 29 +---------------------------- lib/ext2fs/rw_bitmaps.c | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+), 28 deletions(-) diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 83f2af07..616c9412 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -2122,34 +2122,7 @@ ext2fs_const_inode(const struct ext2_inode_large * large_inode) return (const struct ext2_inode *) large_inode; } -static dgrp_t ext2fs_get_avg_group(ext2_filsys fs) -{ -#ifdef CONFIG_PFSCK - dgrp_t average_group; - unsigned flexbg_size; - - if (fs->fs_num_threads <= 1) - return fs->group_desc_count; - - average_group = fs->group_desc_count / fs->fs_num_threads; - if (average_group <= 1) - return 1; - - if (ext2fs_has_feature_flex_bg(fs->super)) { - int times = 1; - - flexbg_size = 1 << fs->super->s_log_groups_per_flex; - if (average_group % flexbg_size) { - times = average_group / flexbg_size; - average_group = times * flexbg_size; - } - } - - return average_group; -#else - return fs->group_desc_count; -#endif -} +dgrp_t ext2fs_get_avg_group(ext2_filsys fs); #undef _INLINE_ #endif diff --git a/lib/ext2fs/rw_bitmaps.c b/lib/ext2fs/rw_bitmaps.c index 960a6913..95de9b1c 100644 --- a/lib/ext2fs/rw_bitmaps.c +++ b/lib/ext2fs/rw_bitmaps.c @@ -515,6 +515,31 @@ struct read_bitmaps_thread_info { io_channel rbt_io; }; +dgrp_t ext2fs_get_avg_group(ext2_filsys fs) +{ + dgrp_t average_group; + unsigned flexbg_size; + + if (fs->fs_num_threads <= 1) + return fs->group_desc_count; + + average_group = fs->group_desc_count / fs->fs_num_threads; + if (average_group <= 1) + return 1; + + if (ext2fs_has_feature_flex_bg(fs->super)) { + int times = 1; + + flexbg_size = 1 << fs->super->s_log_groups_per_flex; + if (average_group % flexbg_size) { + times = average_group / flexbg_size; + average_group = times * flexbg_size; + } + } + + return average_group; +} + static void* read_bitmaps_thread(void *data) { struct read_bitmaps_thread_info *rbt = data; From patchwork Wed Nov 18 15:39:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402357 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=S6/og6DT; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9l24Tcz9sTc for ; Thu, 19 Nov 2020 02:42:03 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727617AbgKRPmD (ORCPT ); Wed, 18 Nov 2020 10:42:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726089AbgKRPmC (ORCPT ); Wed, 18 Nov 2020 10:42:02 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 818D4C0613D4 for ; Wed, 18 Nov 2020 07:42:02 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id v2so1524932pgv.2 for ; Wed, 18 Nov 2020 07:42:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=8oyeSDoHSskTjE71/BgsMrVoYjV8qLw3qKErTKdSEIU=; b=S6/og6DTzs5jAHDoWBG2uZ+aTMDKscwnLKuZKHRIg+9jEFNYL1l4mTB8SOb28fwIuq +LKmu1ChgD66My1+LTumj995zRYcNObmBzFS2MIVOSj7s9CnkR4Po11ebUuGVX8cJ5+T iFSG3G6pYayUwz1j2cO7zobsKCAPhoytgpdWtqI6sgbSSz2iXIGV3rzB5x/gMDFLKGgh PJ+tKrYh4ktO58kqT695gc3a/XXj2ZsPpW9g9lbgtmwifmxQshB1Fp/NDzytMis3dtfC du4BGWlDLeAbQxv3OITotpwC89tJ/Qb2BEh+vBDj157vJz2q0kGY04sCl1KC0DhqZ1Gu 4kXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8oyeSDoHSskTjE71/BgsMrVoYjV8qLw3qKErTKdSEIU=; b=bsxVU3DKaTQQtlZKo5oMTSloMkOrZGtpQaZSiOacE+asn9hlUESM9xdqPUNbvaN+RM fwuvTpejh0btXiZQfngKi8HHzRlJ9/9YVIDPgL/8dXrllLCJfCcFqg+3wPXWHSMGS6do UKgSlmGXgP7evccv9m5n96UJukJR+QsjJcCFMG9B5mywEuvOxvRHk3nABqV1rP5nmtek ZWzUfeGIxHxpHsW7BTzba0lus94yNTd1GFtljGa7OBp/hPtig3A/K0bBwDDW4EypXTDg fghwR5zREe6NhwrV/DWa9J7TvfyqqTLeFxo1GHbXhB/graOIbyfhYNAsNeWOaiHEcSN/ eNVQ== X-Gm-Message-State: AOAM532DCWc960/s6td8SPDEhlH9aotKHpFCJEre7Y+OP0XFQNmg9rVH E+fyl3OYQkNaTZxEK7b/VvCU0IdmdnhBoWqxSPkTrvRxP86KSZJXGkl7/WhmJUnwi7xVBPsP1qc 6UuWIsosn3dFWDA3IeLiOjt9E72Bponle7yY9OqHYvoHsWBjJ62D9LFk2I41VM20BGi2Xk+PIWB cdgze31Ww= X-Google-Smtp-Source: ABdhPJzHJ/upZ9qzE6UOssmsqoxgyzRozQW4y5GpcKM8GJTM9eRzStv+mD8Qx1it7PVHqwaZdYqeFGj4kGwOre01UZM= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:90b:293:: with SMTP id az19mr46500pjb.1.1605714121628; Wed, 18 Nov 2020 07:42:01 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:37 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-52-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 51/61] configure: enable pfsck by default From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Since most of work has been done, compile e2fsprogs with pfsck enabled by default. So it could testing widely now. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- configure | 13 +++++++++++-- configure.ac | 13 ++++++++++--- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/configure b/configure index 1bb7a325..511de4a1 100755 --- a/configure +++ b/configure @@ -1589,7 +1589,7 @@ Optional Features: --disable-tdb disable tdb support --disable-bmap-stats disable collection of bitmap stats. --enable-bmap-stats-ops enable collection of additional bitmap stats - --enable-pfsck enable parallel e2fsck + --disable-pfsck disable parallel e2fsck --disable-nls do not use Native Language Support --enable-threads={posix|solaris|pth|windows} specify multithreading API @@ -6169,8 +6169,17 @@ $as_echo "Enabling parallel e2fsck" >&6; } fi else - { $as_echo "$as_me:${as_lineno-$LINENO}: result: Disabling parallel e2fsck" >&5 + if test -z "PTHREAD_LIB" +then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: Disabling parallel e2fsck" >&5 $as_echo "Disabling parallel e2fsck" >&6; } +else + +$as_echo "#define CONFIG_PFSCK 1" >>confdefs.h + + { $as_echo "$as_me:${as_lineno-$LINENO}: result: Enabling parallel e2fsck by default" >&5 +$as_echo "Enabling parallel e2fsck by default" >&6; } +fi fi diff --git a/configure.ac b/configure.ac index e73dbf50..2dacd6c8 100644 --- a/configure.ac +++ b/configure.ac @@ -877,13 +877,13 @@ fi AC_MSG_RESULT([Disabling additional bitmap statistics by default]) ) dnl -dnl handle --enable-pfsck +dnl handle --disable-pfsck dnl PTHREAD_LIB='' AC_CHECK_LIB(pthread,pthread_join,PTHREAD_LIB=-pthread) AC_SUBST(PTHREAD_LIB) AC_ARG_ENABLE([pfsck], -[ --enable-pfsck enable parallel e2fsck], +[ --disable-pfsck disable parallel e2fsck], if test "$enableval" = "no" || test -z "PTHREAD_LIB" then AC_MSG_RESULT([Disabling parallel e2fsck]) @@ -893,7 +893,14 @@ else AC_MSG_RESULT([Enabling parallel e2fsck]) fi , -AC_MSG_RESULT([Disabling parallel e2fsck]) +if test -z "PTHREAD_LIB" +then + AC_MSG_RESULT([Disabling parallel e2fsck]) +else + AC_DEFINE(CONFIG_PFSCK, 1, + [Define to 1 if parallel e2fsck is enabled]) + AC_MSG_RESULT([Enabling parallel e2fsck by default]) +fi ) dnl dnl From patchwork Wed Nov 18 15:39:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402359 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=ieWkSbTJ; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9z4TCHz9sPB for ; Thu, 19 Nov 2020 02:42:15 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727635AbgKRPmO (ORCPT ); Wed, 18 Nov 2020 10:42:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727435AbgKRPmG (ORCPT ); Wed, 18 Nov 2020 10:42:06 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6C7AC0613D6 for ; Wed, 18 Nov 2020 07:42:06 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id 190so1410372pfz.16 for ; Wed, 18 Nov 2020 07:42:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=U8cT2QAxIu96iJauReA/Bomb7qYfmerVWdtdZDUXkJc=; b=ieWkSbTJT2eRJ0ussxH4h+s7Q9RzySxvlo/HcKSWQ41JRkrxV8Fzi7GK+3y/fT5Ii0 C3A82WjR/130nzSX501okoxhDg4pOVPz7pJlXKrrgsVR+WWpD0BDlMDQSYyJrEVizVuC xlElv+Gd+v95iuJJbOCadJd4R0D6KrtKx277mMYCq2T2Vm0ARt+F/IOKura+nAdmQjiG +NW/HIuj0mfGGtO4KkfkEVZNuvpk5+IDBYijri7HjbKbj7icdj8pXcWa499/d9GBgCeq eN+Zgk3isFkZrgLgD+uHza9mHVHRA876+jl96QQUW6x9yH23kuNdnzeBHpr+b7FC0AZi G5OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=U8cT2QAxIu96iJauReA/Bomb7qYfmerVWdtdZDUXkJc=; b=oPKDRUFwKbWMdCs6q9pKCgGbP5Anrg+Da1NTcSvtG9aJc8aZr8W91cHq+y7yZsIC/7 pct3TCOm3m7mJNRtlYSUKCy64lveGfx+qBSAz/C4Z493WaHAqLMO8lEmkoMXC0621hQy WqU1zKi3jsa+QLAXUkPY7d32qqAlJR3SHCSC/9KtYzMLckIEFF6ccp0Q/4kGiFz/yMlt 3YzhFiZNtgRsly5IfqPO/ZYo5w1AV22WRtpc48ZHRHaMHzDHK9+qF8HTUdmKW89aj+zu S52NtK1q3LL1how7LxNfSXI5Ne1aWAsAb/UAa7MoSIIAkCIc77mKHjviaYQHp0jgDEna yRTA== X-Gm-Message-State: AOAM5325HFflxzGgIpjVfwmSRge6rtwOrH+z5bM8w4iLczT7WMzM6Ain L0jt97ZQaq+Fkl9qui1KinPxJN8eP8MR2iYZO042aiiliRa00NPF6e7Aee4H09lycXkuGQatyRe vyX81dmnH0+nDXLG+T55LqN1iwA18Ty3sEL8fbd+ATncmTlBJMvXze5JVGJUBjMuh5KO7ZmuWy1 RfAlArwrI= X-Google-Smtp-Source: ABdhPJxJCO87gq/s83j2s7BtGFQSCdze9wgLoP2bZ9yP6jW2UUdL4zCoH0SrX/m7gPblcQxn2ofjIAsth3kneWI1ZSQ= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:902:8c8a:b029:d6:d1e7:e78e with SMTP id t10-20020a1709028c8ab02900d6d1e7e78emr4744264plo.39.1605714126081; Wed, 18 Nov 2020 07:42:06 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:39 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-54-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 53/61] e2fsck: fix f_multithread_ok test From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Andreas Dilger , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Andreas Dilger Don't use $OUT for both the input amd output of a pipeline, as the output file is truncated befor the input is read. Fix the handling in the failure case to generate the *.failed file, and print the actual $test_name instead of "test_name". Signed-off-by: Andreas Dilger Signed-off-by: Saranya Muruganandam --- tests/f_multithread_ok/script | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/f_multithread_ok/script b/tests/f_multithread_ok/script index 4010881b..c62c93ce 100644 --- a/tests/f_multithread_ok/script +++ b/tests/f_multithread_ok/script @@ -5,15 +5,15 @@ SKIP_CLEANUP="true" . $cmd_dir/run_e2fsck -cat $OUT1 | grep -v Thread > $OUT1 -rm -f $test_name.ok $test_name.failed -cmp -s $OUT1 $EXP1 +grep -v Thread $OUT1 > $OUT1.tmp +cmp -s $OUT1.tmp $EXP1 status1=$? if [ "$status1" -eq 0 ]; then echo "$test_name: $test_description: ok" touch $test_name.ok else - echo "test_name: $test_description: failed" + echo "$test_name: $test_description: failed" + cmp $OUT1.tmp $EXP1 > $test_name.failed fi unset IMAGE FSCK_OPT SECOND_FSCK_OPT OUT1 OUT2 EXP1 EXP2 From patchwork Wed Nov 18 15:39:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402358 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=ThZjbXk5; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4Cbn9y5DLHz9sPB for ; Thu, 19 Nov 2020 02:42:14 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727622AbgKRPmO (ORCPT ); Wed, 18 Nov 2020 10:42:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727460AbgKRPmJ (ORCPT ); Wed, 18 Nov 2020 10:42:09 -0500 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA231C061A48 for ; Wed, 18 Nov 2020 07:42:08 -0800 (PST) Received: by mail-qk1-x749.google.com with SMTP id b191so1759924qkc.10 for ; Wed, 18 Nov 2020 07:42:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=aTfARycfPe5TMTx5lZg7EVRgT2qvCg6B5B2Q6/C3RLw=; b=ThZjbXk50/HkJw0Dg8BJQRrRhAjwDm8m7vW/Bl9MifeXkHxoGoPZb6v48Ldpfo2lL6 tr9DImth9kDbSYpEaTPgAnQ4HmUdHYMneefd0DUlsd/3FJ75wiaLSSRYqeh0df2TtOY4 15x4VqmFSAWj3wtDKcnzvUVrA/ksH/b+xGafQuK9maq3DCuS2nydh1abQl4/l4e5MUZ3 q5M4PJH50cvISlxvI/3gQaO93O3X/tzPogwunG7j9AyIUJsfNLWTmZk8b0nlDd0M1+3B sPTnTP2qAfQIpIf/T1lIBvbJNZOrGsM4N56NDGkIkG6JBYdyqJZ6a2vgwwmKAehYFphB n3Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=aTfARycfPe5TMTx5lZg7EVRgT2qvCg6B5B2Q6/C3RLw=; b=BIpcH77oLrRxiXJvSGHrO4mIZJ+lgqFzyxJWEDA1pExmIuhNxEd0O9+oAOJSewHa9M w7t0DV4ateQRX6Zujxqd+8kKyMT/sfjvYWs6FceuHrTHxNRqRrvmQrpVPQK0xdu1ZhMG kY5rMvQn51ceELHri5WyU7UEw3VolgL/nlb/TwWueA+uRYxzDM2rnTfw0GZAJOOaAvEQ cfAax+zmXev/iaXXURxHKM4ccVSXLYxWAaeaxotfpznD3vKM7ef0vJ7gZfFYUlpuSmYN d1C5ppAfrVhG32kDSfOBEX0XJB1HJdvAwOgj+ZBrOgLQ2uXIQkv/cU56/nkblt1N4xg2 E+ew== X-Gm-Message-State: AOAM533NnRlQNy2UrFuEoWn9+0hUYsUd+yPkffez8h0o0u8NSSFmH34a B0ixr+DEpVo6RmQAYmiL/9jYzG6q5F/sKAuXHPN/u/ZZDuRurX5+uzeroOmRJn3O+eANaR1cuc1 YCB0VkD18eJtgmsx3PgzUwXBGl62KJg26Q1ApzuFXsdL2eZsj4kWbPB7x9XB+1NXQfPoFu3yKxY 8iUhJpcSo= X-Google-Smtp-Source: ABdhPJy2DCkQug/vrHg4ddX/6ngLNGTdc38EEjCNGwDHXhZUEyIggF3FeIG8PGDvMEkslL195dUqGsNmHF47ZZc3EF0= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:f9c8:: with SMTP id j8mr5220496qvo.17.1605714127986; Wed, 18 Nov 2020 07:42:07 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:40 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-55-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 54/61] e2fsck: fix race in ext2fs_read_bitmaps() From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong During corruption testing hiting following segfault: Multiple threads triggered to read bitmaps Signal (11) SIGSEGV si_code=SEGV_MAPERR fault addr=0x200 ./e2fsck[0x4382ae] /lib64/libpthread.so.0(+0x14b20)[0x7f5854d2fb20] ./e2fsck(ext2fs_rb_insert_color+0xc)[0x46ac0c] ./e2fsck[0x467bb4] ./e2fsck[0x467e6d] ./e2fsck[0x45ba95] ./e2fsck[0x45c124] /lib64/libpthread.so.0(+0x94e2)[0x7f5854d244e2] /lib64/libc.so.6(clone+0x43)[0x7f5854beb6c3] Problem is @block_map might be set NULL if one of thread exit, move such kind of cleanup operation to main thread after all threads exit. Another potential problem is e2fsck_read_bitmap() could be called during pass1, this need be serialized, serialize it in the pass1. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 2 +- lib/ext2fs/rw_bitmaps.c | 25 ++++++++----------------- 2 files changed, 9 insertions(+), 18 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index e98cda9f..3899d710 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -4183,9 +4183,9 @@ report_problem: } continue; } + e2fsck_pass1_fix_lock(ctx); e2fsck_read_bitmaps(ctx); pb->inode_modified = 1; - e2fsck_pass1_fix_lock(ctx); pctx->errcode = ext2fs_extent_delete(ehandle, 0); e2fsck_pass1_fix_unlock(ctx); diff --git a/lib/ext2fs/rw_bitmaps.c b/lib/ext2fs/rw_bitmaps.c index 95de9b1c..eb791202 100644 --- a/lib/ext2fs/rw_bitmaps.c +++ b/lib/ext2fs/rw_bitmaps.c @@ -445,14 +445,6 @@ success_cleanup: return 0; cleanup: - if (do_block) { - ext2fs_free_block_bitmap(fs->block_map); - fs->block_map = 0; - } - if (do_inode) { - ext2fs_free_inode_bitmap(fs->inode_map); - fs->inode_map = 0; - } if (inode_bitmap) ext2fs_free_mem(&inode_bitmap); if (block_bitmap) @@ -463,9 +455,12 @@ cleanup: } -static errcode_t read_bitmaps_range_end(ext2_filsys fs, int do_inode, int do_block) +static errcode_t read_bitmaps_range_end(ext2_filsys fs, int do_inode, int do_block, + errcode_t retval) { - errcode_t retval = 0; + + if (retval) + goto cleanup; /* Mark group blocks for any BLOCK_UNINIT groups */ if (do_block) { @@ -474,7 +469,7 @@ static errcode_t read_bitmaps_range_end(ext2_filsys fs, int do_inode, int do_blo goto cleanup; } - return retval; + return 0; cleanup: if (do_block) { ext2fs_free_block_bitmap(fs->block_map); @@ -497,10 +492,8 @@ static errcode_t read_bitmaps_range(ext2_filsys fs, int do_inode, int do_block, return retval; retval = read_bitmaps_range_start(fs, do_inode, do_block, start, end, NULL, NULL); - if (retval) - return retval; - return read_bitmaps_range_end(fs, do_inode, do_block); + return read_bitmaps_range_end(fs, do_inode, do_block, retval); } #ifdef CONFIG_PFSCK @@ -636,9 +629,7 @@ out: free(thread_infos); free(thread_ids); - if (!retval) - retval = read_bitmaps_range_end(fs, do_inode, do_block); - + retval = read_bitmaps_range_end(fs, do_inode, do_block, retval); if (!retval) { if (do_inode) fs->flags &= ~EXT2_FLAG_IBITMAP_TAIL_PROBLEM; From patchwork Wed Nov 18 15:39:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402360 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=DYfUx3kp; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CbnB00VdKz9sTc for ; Thu, 19 Nov 2020 02:42:16 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727634AbgKRPmO (ORCPT ); Wed, 18 Nov 2020 10:42:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727626AbgKRPmK (ORCPT ); Wed, 18 Nov 2020 10:42:10 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F06CC061A4D for ; Wed, 18 Nov 2020 07:42:10 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id h9so2932148ybj.10 for ; Wed, 18 Nov 2020 07:42:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=6IeNjD+JbZWYlVnUDIDclFUs22c3oVDKirYNwy4hr2M=; b=DYfUx3kpNmLUUozvKg+jllUuffkVzXz6e3G6nLUHL+uB/3tCfo23jtvrPgq2WUXzi4 yRkSTCqZ6zrMkvlINLISX6ue6XgKz6gfzVf3ch0tBBC+fv+eOvJ7rArT1DOqtpH0g5U8 bFsII9Lwa7ZUMGdF0CBTeNz9zPTtxPMQekyjuyewFH/D8f1hlxpQRd/IMuShSTXosrzA Kv1zUmyl1FtkY7t7+HfzWrqTwGv2OqipSy+6HrPx7I9XuTZQCFfJj1Gpe1gdDx2fs+2Z HB5BXHE/i7hqDkmpPdMtSNUu9+d4M4ag+36mSL35MoVJLxeW7/Og3Rhm78oW+Cm4oUO2 nYJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=6IeNjD+JbZWYlVnUDIDclFUs22c3oVDKirYNwy4hr2M=; b=tmTZ36M4l7MEqWTsoG+ArqJ5fDaiTqjYa8DoO7C1lFDZjCW25VRK2BSpwlhNegUKa9 qRHRuwHUSTm/jpjWDk9pye4lOkjOluKNLeIUZOptDzhhMhXk/68RqJ+0bHRu4JO1seab kusbgTDe+D7M2eWasY11PIx/k3z3KL3rRRoHaah8ZwaapDf/5ORHb9X+cW3dccgl5vES D9g7XLQuKnIHh8lQQyHLAQ4KS2WSR5A0MIk2vBHdWBM6iyap0AW5zTEwmKdOp8NcCMEJ u/RSV6mZbKnjXHbXxh49mqWmll41qCRGWgMKXMH/tYuvzcYHjW1OUnPnwai3z8rBxquK iZqQ== X-Gm-Message-State: AOAM533B8vf8c+UPwjFttedcW/Cg0T0vwOnbMaGZGHKboK25nGWSvFB/ sKoHjG/Sk5i28UoFUTwhnx63z6yR2a6tyjZXwGgXiciBOiKkwmrAHPk0KLxFwllfY6XlLjvoC6x CML9b7nw8gRJwtIMJ7k9F45wy22CHqj6xam0CpuOF1mRFo7SMvp8fh9IaEyPkfNLLj45rYryMWq ldlE04Ago= X-Google-Smtp-Source: ABdhPJwaV+fw9msfxNnip3r/s7hmtYOcPB9NZ5AZr3kW28rV5jmlpr01GboLsvLOBK+wiRlUxE3fb4s0M6IAOSjFnAQ= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a5b:b45:: with SMTP id b5mr7588636ybr.355.1605714129778; Wed, 18 Nov 2020 07:42:09 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:41 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-56-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 55/61] e2fsck: fix readahead for pass1 without pfsck From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong If admin try fsck without -m option, codes try old behavior, thread information won't be inited either. @et_group_end is 0 thus readahead for pass1 will be totally disabled. With the patch applied, we could get same performance number without pfsck as before. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 3899d710..70826866 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1100,7 +1100,8 @@ static void pass1_readahead(e2fsck_t ctx, dgrp_t *group, ext2_ino_t *next_ino) errcode_t err = EXT2_ET_INVALID_ARGUMENT; #ifdef CONFIG_PFSCK - grp_end = ctx->thread_info.et_group_end; + if (ctx->fs->fs_num_threads > 1) + grp_end = ctx->thread_info.et_group_end; #endif if (ctx->readahead_kb == 0) goto out; From patchwork Wed Nov 18 15:39:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402361 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=u56RXdlu; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CbnB0524jz9sPB for ; Thu, 19 Nov 2020 02:42:16 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727641AbgKRPmP (ORCPT ); Wed, 18 Nov 2020 10:42:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727629AbgKRPmM (ORCPT ); Wed, 18 Nov 2020 10:42:12 -0500 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62154C061A4F for ; Wed, 18 Nov 2020 07:42:12 -0800 (PST) Received: by mail-qk1-x74a.google.com with SMTP id s9so1792982qks.2 for ; Wed, 18 Nov 2020 07:42:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=xqJECZo2t3fJB3YHNQYOAUld5JMQjuX171FUJYJ+DMo=; b=u56RXdluMCYxxTxIm0SWim18/dHxvpzr7+owNoUNaFbQX0EWwaZ6f3LcNIIUl2R9hh +woFwOQ/1AEgZ8kJAxW7L7Q5XgCxjQarqgRYl+Q0yAkL62gb4OHYiMEXYckkUKs5Al/7 fqySKEJQpYu0gXHUrmzPzMDSnYLcacGkmI+54Q/jhqTV1Gj1Pcomzk+b3GcYWSSCJtib dt88jzeblcom2ySI3GQa92Lp1TMnL+CAk+G9FKT+e8BtWWXiVSM1HoIvEfZ5j+0B5itB YiNugygV+wCfE0VBLZBc/4/6TjardKxcno3JiyRgl1OyL5RAN8uZved1J0rWdgmuGILv B5LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xqJECZo2t3fJB3YHNQYOAUld5JMQjuX171FUJYJ+DMo=; b=kj/Hnkj92FPEF3SR45dOcfhC6IgVkj/BjTa3/GHLIfAgv8yWTusLMsvd7n+VWnn3d8 FTDPdzklCG9SmHeO1Ow+75IWTf2dDHlbyxAN/qs79rww/nfm8HSePVH1suGSUVmAzymY 3wCUy4scnLoEbVSvGSZ4UOidemPfkr9ZNZLEs3JS8U2cIDn0qiSK20rNqpZb3TWm8jib AALQsBHJYxx+8SfNshCBAGLPgaWm/DoSilb7ffcyaTxUbn2keVsabofI37nKZUS4tTgE AS4DidmWRf5M7l/AKyCta5/UXX6/uhOAr0SmOkjToAq6P3WcFdoy/WbpzEIjUokBF707 an0A== X-Gm-Message-State: AOAM533pfgw27otdwCbcgLdkUTRz4JdC3FM6EfVKYLvcveyTieVNgYln yyQS2RdcbrGdvLA5un+sKvETkALg/Pp5ms6gWuCpqBVO1ogJU8B17ZENI2s9AZjqrTotQByje7T r9vf+zx4SWo5D7GhikRkydJ/yMRxhUyPV7lZTdCoCJh4Cg5wCs9AySxvCcIJVL8IbBBqJs0uMMu LWWi1ueD8= X-Google-Smtp-Source: ABdhPJxGcIc9VsDStZ3RNk6PEb8E4NFurxuLRb5CrpreRC9ugsE7U4OMD9Qc+XyTJv8otqUuljTDcCpJxkPM5q0Nc3w= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a0c:b351:: with SMTP id a17mr5406992qvf.7.1605714131404; Wed, 18 Nov 2020 07:42:11 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:42 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-57-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 56/61] e2fsck: fix memory leaks with pfsck enabled From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong valgrind detected few memory leaks: 1) quota context is not released after merging. 2) three block bufs are not freed in read_bitmaps_range_start() 3) @refcount_orig should be released Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.c | 4 ++++ e2fsck/pass1.c | 1 + lib/ext2fs/rw_bitmaps.c | 14 ++++---------- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/e2fsck/e2fsck.c b/e2fsck/e2fsck.c index a03550c0..e406f6dd 100644 --- a/e2fsck/e2fsck.c +++ b/e2fsck/e2fsck.c @@ -102,6 +102,10 @@ errcode_t e2fsck_reset_context(e2fsck_t ctx) ea_refcount_free(ctx->refcount_extra); ctx->refcount_extra = 0; } + if (ctx->refcount_orig) { + ea_refcount_free(ctx->refcount_orig); + ctx->refcount_orig = 0; + } if (ctx->ea_block_quota_blocks) { ea_refcount_free(ctx->ea_block_quota_blocks); ctx->ea_block_quota_blocks = 0; diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 70826866..7768119b 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -3113,6 +3113,7 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) fclose(thread_ctx->problem_logf); } + quota_release_context(&thread_ctx->qctx); /* * @block_metadata_map and @block_dup_map are * shared, so we don't free them. diff --git a/lib/ext2fs/rw_bitmaps.c b/lib/ext2fs/rw_bitmaps.c index eb791202..5fde2632 100644 --- a/lib/ext2fs/rw_bitmaps.c +++ b/lib/ext2fs/rw_bitmaps.c @@ -269,7 +269,7 @@ static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_b dgrp_t i; char *block_bitmap = 0, *inode_bitmap = 0; char *buf; - errcode_t retval; + errcode_t retval = 0; int block_nbytes = EXT2_CLUSTERS_PER_GROUP(fs->super) / 8; int inode_nbytes = EXT2_INODES_PER_GROUP(fs->super) / 8; int tail_flags = 0; @@ -432,18 +432,12 @@ static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_b success_cleanup: if (start == 0 && end == fs->group_desc_count - 1) { - if (inode_bitmap) { - ext2fs_free_mem(&inode_bitmap); + if (inode_bitmap) fs->flags &= ~EXT2_FLAG_IBITMAP_TAIL_PROBLEM; - } - if (block_bitmap) { - ext2fs_free_mem(&block_bitmap); + if (block_bitmap) fs->flags &= ~EXT2_FLAG_BBITMAP_TAIL_PROBLEM; - } } fs->flags |= tail_flags; - return 0; - cleanup: if (inode_bitmap) ext2fs_free_mem(&inode_bitmap); @@ -451,8 +445,8 @@ cleanup: ext2fs_free_mem(&block_bitmap); if (buf) ext2fs_free_mem(&buf); - return retval; + return retval; } static errcode_t read_bitmaps_range_end(ext2_filsys fs, int do_inode, int do_block, From patchwork Wed Nov 18 15:39:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402362 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=HK3BJvrA; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CbnB119T3z9sTc for ; Thu, 19 Nov 2020 02:42:17 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727646AbgKRPmQ (ORCPT ); Wed, 18 Nov 2020 10:42:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727633AbgKRPmO (ORCPT ); Wed, 18 Nov 2020 10:42:14 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9F24C0613D4 for ; Wed, 18 Nov 2020 07:42:13 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id u3so1390454pfm.22 for ; Wed, 18 Nov 2020 07:42:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=W6rzihCPF7UlsyHHaLuAFfmf9xopI8dxJp/W6QNsfwk=; b=HK3BJvrAKA0WvPBDJPe3DM3KlGJ+EbfEsirJHno6aHRYClkh5nEmhAXNDYZSxoi70x KrNAzJfnhii44CeFnPDru+hobjnLUVTyOisZkplo9SijzNSV7O5AcCNGyVCGsSEzxSTL mUjr5fgBQ7CYfPNDJcCW8josxZd61DGImuphj3KXMyWXVVpqDMqntsTYnb1IJQ27yVk5 dih9Pnuf5ZoQUqznYTtnn5/ofJ3EAP3yDHkkKlcz4MlPK3IZpVKBoS8QhqRVqpEW9CGE 4QH5IBRIj977+GYU/xTaWvXoxsCLpq83OlO1gd75a9h7pyQaw9WMZKNNBOggHFXWw1/N 3qPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=W6rzihCPF7UlsyHHaLuAFfmf9xopI8dxJp/W6QNsfwk=; b=i8jGIQ+jyg565aJAZLlwVURbV0L9AZRNjUVUBsIySZMdcA21aC1QQ3T89UlRHDJnRn 6Mt59wFMZeDkJKMVJtmUJ2013jOqZncW+M3qp3tmjl44wtMDBXZW/nPuRiUPSRu8JChQ sTKQVpcMoHgzAlRlHSJZrAfgiZiZ7z29XeiUeT9B2l6xwr0/jfDIMmBRd1FvRVkwVxfA rnvKYAdRSN7spF7jbA/LYN1DE+VENClo73NhTcVkMp6tLuryIukOti0jW+Q/IrtJQMEM zSbMkph8zgkgtCFHSV6iIR7VqDSM75/sp+ejF0sIERrkHpxDN/HZfJEn5dd1XhxGC9LO jb9w== X-Gm-Message-State: AOAM532WNr1Lz+xFImgTCL11Efxi+QPph+JxsVr0RnciXpmi2fiprnX3 9z6iEGVznxRtDCazzMkmzO8zhvJvKxZ7Mn7JZOtsphHq3YPGijHtcUn9PVXtbLcN0pSmtAAwt65 iM87ACAATxc2aTXrsP5DjZ4UcD0PPXw7ROtnvetnqyJaUIoxuZ4NAJzCIw4gDNrxsBekKaTZyv6 NDPeCgEqY= X-Google-Smtp-Source: ABdhPJyKDuVjAvFf3NyFopb/zzYEFjlSS9ZQ1ktp2J6VOZHp2A0QyppnrzhaQDHLtW3S1gs3DmEHikaGKvGg6tELDFI= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a62:1896:0:b029:197:491c:be38 with SMTP id 144-20020a6218960000b0290197491cbe38mr5020930pfy.15.1605714133245; Wed, 18 Nov 2020 07:42:13 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:43 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-58-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 57/61] ext2fs: fix to set tail flags with pfsck enabled From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong If any of block/inode bitmap block checksum error happen, tail flag should be set properly. However, we firstly set tail flags in each thread, after threads finish we clear those tail problem wrongly. This will make fsck miss bitmap checksum erors later, patch try to fix the problem by move all this kind of logic in read_bitmaps_range_end() Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- lib/ext2fs/rw_bitmaps.c | 46 +++++++++++++++++------------------------ 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/lib/ext2fs/rw_bitmaps.c b/lib/ext2fs/rw_bitmaps.c index 5fde2632..b66ba773 100644 --- a/lib/ext2fs/rw_bitmaps.c +++ b/lib/ext2fs/rw_bitmaps.c @@ -264,7 +264,7 @@ cleanup: static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_block, dgrp_t start, dgrp_t end, pthread_mutex_t *mutex, - io_channel io) + io_channel io, int *tail_flags) { dgrp_t i; char *block_bitmap = 0, *inode_bitmap = 0; @@ -272,7 +272,6 @@ static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_b errcode_t retval = 0; int block_nbytes = EXT2_CLUSTERS_PER_GROUP(fs->super) / 8; int inode_nbytes = EXT2_INODES_PER_GROUP(fs->super) / 8; - int tail_flags = 0; int csum_flag; unsigned int cnt; blk64_t blk; @@ -343,7 +342,7 @@ static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_b blk_itr += cnt; blk_cnt -= cnt; } - goto success_cleanup; + goto cleanup; } blk_itr += ((blk64_t)start * (block_nbytes << 3)); @@ -374,7 +373,7 @@ static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_b } if (!bitmap_tail_verify((unsigned char *) block_bitmap, block_nbytes, fs->blocksize - 1)) - tail_flags |= EXT2_FLAG_BBITMAP_TAIL_PROBLEM; + *tail_flags |= EXT2_FLAG_BBITMAP_TAIL_PROBLEM; } else memset(block_bitmap, 0, block_nbytes); cnt = block_nbytes << 3; @@ -414,7 +413,7 @@ static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_b } if (!bitmap_tail_verify((unsigned char *) inode_bitmap, inode_nbytes, fs->blocksize - 1)) - tail_flags |= EXT2_FLAG_IBITMAP_TAIL_PROBLEM; + *tail_flags |= EXT2_FLAG_IBITMAP_TAIL_PROBLEM; } else memset(inode_bitmap, 0, inode_nbytes); cnt = inode_nbytes << 3; @@ -430,14 +429,6 @@ static errcode_t read_bitmaps_range_start(ext2_filsys fs, int do_inode, int do_b } } -success_cleanup: - if (start == 0 && end == fs->group_desc_count - 1) { - if (inode_bitmap) - fs->flags &= ~EXT2_FLAG_IBITMAP_TAIL_PROBLEM; - if (block_bitmap) - fs->flags &= ~EXT2_FLAG_BBITMAP_TAIL_PROBLEM; - } - fs->flags |= tail_flags; cleanup: if (inode_bitmap) ext2fs_free_mem(&inode_bitmap); @@ -450,7 +441,7 @@ cleanup: } static errcode_t read_bitmaps_range_end(ext2_filsys fs, int do_inode, int do_block, - errcode_t retval) + errcode_t retval, int tail_flags) { if (retval) @@ -461,7 +452,11 @@ static errcode_t read_bitmaps_range_end(ext2_filsys fs, int do_inode, int do_blo retval = mark_uninit_bg_group_blocks(fs); if (retval) goto cleanup; + fs->flags &= ~EXT2_FLAG_BBITMAP_TAIL_PROBLEM; } + if (do_inode) + fs->flags &= ~EXT2_FLAG_IBITMAP_TAIL_PROBLEM; + fs->flags |= tail_flags; return 0; cleanup: @@ -480,14 +475,16 @@ static errcode_t read_bitmaps_range(ext2_filsys fs, int do_inode, int do_block, dgrp_t start, dgrp_t end) { errcode_t retval; + int tail_flags = 0; retval = read_bitmaps_range_prepare(fs, do_inode, do_block); if (retval) return retval; - retval = read_bitmaps_range_start(fs, do_inode, do_block, start, end, NULL, NULL); + retval = read_bitmaps_range_start(fs, do_inode, do_block, start, end, NULL, + NULL, &tail_flags); - return read_bitmaps_range_end(fs, do_inode, do_block, retval); + return read_bitmaps_range_end(fs, do_inode, do_block, retval, tail_flags); } #ifdef CONFIG_PFSCK @@ -499,6 +496,7 @@ struct read_bitmaps_thread_info { dgrp_t rbt_grp_end; errcode_t rbt_retval; pthread_mutex_t *rbt_mutex; + int rbt_tail_flags; io_channel rbt_io; }; @@ -534,7 +532,7 @@ static void* read_bitmaps_thread(void *data) rbt->rbt_retval = read_bitmaps_range_start(rbt->rbt_fs, rbt->rbt_do_inode, rbt->rbt_do_block, rbt->rbt_grp_start, rbt->rbt_grp_end, - rbt->rbt_mutex, rbt->rbt_io); + rbt->rbt_mutex, rbt->rbt_io, &rbt->rbt_tail_flags); return NULL; } #endif @@ -550,7 +548,7 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) errcode_t retval; errcode_t rc; dgrp_t average_group; - int i; + int i, tail_flags = 0; io_manager manager = unix_io_manager; #else int num_threads = 1; @@ -584,6 +582,7 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) thread_infos[i].rbt_do_inode = do_inode; thread_infos[i].rbt_do_block = do_block; thread_infos[i].rbt_mutex = &rbt_mutex; + thread_infos[i].rbt_tail_flags = 0; if (i == 0) thread_infos[i].rbt_grp_start = 0; else @@ -614,6 +613,7 @@ static errcode_t read_bitmaps(ext2_filsys fs, int do_inode, int do_block) rc = thread_infos[i].rbt_retval; if (rc && !retval) retval = rc; + tail_flags |= thread_infos[i].rbt_tail_flags; io_channel_close(thread_infos[i].rbt_io); } out: @@ -623,15 +623,7 @@ out: free(thread_infos); free(thread_ids); - retval = read_bitmaps_range_end(fs, do_inode, do_block, retval); - if (!retval) { - if (do_inode) - fs->flags &= ~EXT2_FLAG_IBITMAP_TAIL_PROBLEM; - if (do_block) - fs->flags &= ~EXT2_FLAG_BBITMAP_TAIL_PROBLEM; - } - - return retval; + return read_bitmaps_range_end(fs, do_inode, do_block, retval, tail_flags); #endif } From patchwork Wed Nov 18 15:39:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402363 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=iy4Qkr6E; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CbnB14LTjz9sPB for ; Thu, 19 Nov 2020 02:42:17 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727647AbgKRPmR (ORCPT ); Wed, 18 Nov 2020 10:42:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50398 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727629AbgKRPmQ (ORCPT ); Wed, 18 Nov 2020 10:42:16 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EECF3C0613D4 for ; Wed, 18 Nov 2020 07:42:15 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id x3so1394461plr.23 for ; Wed, 18 Nov 2020 07:42:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=AMvkoRCCFsGkzkWRSSMZqjAs0Ax8qqFNLPWjD1f+sNw=; b=iy4Qkr6EoAiNyRhnNHcSQ8doFB7RbZkzRIPB8Q8eMYW0H0JvRBY33yM4YQi1OI7JFI JW2DSL9r3ApKhdi3miEts3JlxH8IK2H0hkALlusotTGYAVKzvWS92URa4FzlWrj8UilW LzaH9zUkhbhkeyijUrT8oUmMliGwSjL/wU0JkW2roi/xlidnuZtCk/G6xqU2EwM4AXjh zku3a/8PRXlsARQKyj+2h5U4v0jXyx+8DVHI/dhvg1zpGU0ZMLOL1t4ATLPt23vvGNXC 4d/RXTL9saE8KxasXojlvrLjOfpkb87RANlqNhAWBwIYMGgapLitr45PM/FwGD22TpJa Kfpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=AMvkoRCCFsGkzkWRSSMZqjAs0Ax8qqFNLPWjD1f+sNw=; b=uVyxJxG8rErsLzhp2FbozhnNo8Vqjrv54+cspPB0WrZXUxJbTzlM1Iq8aFk1dC9RkT E8BhSkwTARt4eiJ1PGsl71ly31sHEW3NXqZ2HxkhYJq89/iALtMOzJ+OjBDSOthcHMUB q9xCvkLbUBs3O21BzztVxe5hWwJXqHgifFKz3vbKFa8b21rFkdnHaacl4rwhhVrVdjrw uRryzBQ/SMEK4XaA0WFnk8bpxzdtR1fq8+JMJl89FBqwU51Fg8vsYg5HwDEhnqMlrR7B 3BLbkA1SyLjq082p52WHr+tzbxANGTvgzzyrJ/AIUbDVhRvWK6J7cWpYuFDF7TIb5zgL GxLw== X-Gm-Message-State: AOAM532eRYpukw52cRXxVbFeP0KAz1AvPtEjJw/1CzjFeUxrvPBuaxPR ZtsrpihTnHAGHKVtq/Sexb+zIgHmjLcW+ZHeq8v9QECn2DXAAJqxRKLjrqpkMbtdecyryUFuQr6 cj7jrtJ9S0NiPAmYVUurT1ja5SFFt0SVSsouRtAfE0cx2PZTctGRjwmcWDdhQx2yLKsq8HsWfO0 i7gCU+jrM= X-Google-Smtp-Source: ABdhPJyVh9uMShOsIHfe1dTVvxanWom7kNQC+sl1JieTodHNDva4ORcld6NONW6b/8sCJnuMH43dyLxb/FeJOUs8YSk= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:902:b613:b029:d9:56e:d93c with SMTP id b19-20020a170902b613b02900d9056ed93cmr4972664pls.14.1605714135181; Wed, 18 Nov 2020 07:42:15 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:44 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-59-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 58/61] e2fsck: misc cleanups for pfsck From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Andreas Dilger , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Andreas Dilger Add -m option description to e2fsck.8 man page. Rename e2fsck_struct fs_num_threads to pfs_num_threads to avoid confusion with the ext2_filsys fs_num_threads field, and move thread_info to be together with the other CONFIG_PFSCK fields. Move ext2_filsys fs_num_threads to fit into the __u16 "pad" field to avoid consuming one of the few remaining __u32 reserved fields. Fix a few print format warnings. Signed-off-by: Andreas Dilger Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.8.in | 8 +++++++- e2fsck/e2fsck.h | 13 +++++-------- e2fsck/pass1.c | 29 +++++++++++++++-------------- e2fsck/unix.c | 4 ++-- lib/ext2fs/ext2fs.h | 5 ++--- 5 files changed, 31 insertions(+), 28 deletions(-) diff --git a/e2fsck/e2fsck.8.in b/e2fsck/e2fsck.8.in index 4e3890b2..404d07fe 100644 --- a/e2fsck/e2fsck.8.in +++ b/e2fsck/e2fsck.8.in @@ -8,7 +8,7 @@ e2fsck \- check a Linux ext2/ext3/ext4 file system .SH SYNOPSIS .B e2fsck [ -.B \-pacnyrdfkvtDFV +.B \-pacnyrdfkmvtDFV ] [ .B \-b @@ -329,6 +329,12 @@ Set the bad blocks list to be the list of blocks specified by option, except the bad blocks list is cleared before the blocks listed in the file are added to the bad blocks list.) .TP +.B \-m " threads" +Run e2fsck with up to the specified number of +.IR threads . +The actual number of threads may be lower, if the filesystem does not +have enough block groups to effectively parallelize the workload. +.TP .B \-n Open the filesystem read-only, and assume an answer of `no' to all questions. Allows diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 1469c3e1..362e128c 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -243,8 +243,8 @@ struct e2fsck_thread { dgrp_t et_group_next; /* Scanned inode number */ ext2_ino_t et_inode_number; - char et_log_buf[2048]; char et_log_length; + char et_log_buf[2048]; }; #endif @@ -333,11 +333,6 @@ struct e2fsck_struct { ext2_ino_t stashed_ino; struct ext2_inode *stashed_inode; - /* if @global_ctx is null, this field is unused */ -#ifdef CONFIG_PFSCK - struct e2fsck_thread thread_info; -#endif - /* * Location of the lost and found directory */ @@ -450,7 +445,9 @@ struct e2fsck_struct { /* Undo file */ char *undo_file; #ifdef CONFIG_PFSCK - __u32 fs_num_threads; + /* if @global_ctx is null, this field is unused */ + struct e2fsck_thread thread_info; + __u32 pfs_num_threads; __u32 mmp_update_thread; int fs_need_locking; /* serialize fix operation for multiple threads */ @@ -689,7 +686,7 @@ int check_backup_super_block(e2fsck_t ctx); void check_resize_inode(e2fsck_t ctx); /* util.c */ -#define E2FSCK_MAX_THREADS (65536) +#define E2FSCK_MAX_THREADS (65535) extern void *e2fsck_allocate_memory(e2fsck_t ctx, unsigned long size, const char *description); extern int ask(e2fsck_t ctx, const char * string, int def); diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 7768119b..8d4e2675 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1260,7 +1260,7 @@ static void e2fsck_pass1_set_thread_num(e2fsck_t ctx) { unsigned flexbg_size = 1; ext2_filsys fs = ctx->fs; - int num_threads = ctx->fs_num_threads; + int num_threads = ctx->pfs_num_threads; int max_threads; if (num_threads < 1) { @@ -1280,6 +1280,8 @@ static void e2fsck_pass1_set_thread_num(e2fsck_t ctx) max_threads = fs->group_desc_count / flexbg_size; if (max_threads == 0) max_threads = 1; + if (max_threads > E2FSCK_MAX_THREADS) + max_threads = E2FSCK_MAX_THREADS; if (num_threads > max_threads) { fprintf(stderr, "Use max possible thread num: %d instead\n", @@ -1287,7 +1289,7 @@ static void e2fsck_pass1_set_thread_num(e2fsck_t ctx) num_threads = max_threads; } out: - ctx->fs_num_threads = num_threads; + ctx->pfs_num_threads = num_threads; ctx->fs->fs_num_threads = num_threads; } #endif @@ -1315,7 +1317,7 @@ static errcode_t e2fsck_pass1_prepare(e2fsck_t ctx) #ifdef CONFIG_PFSCK /* don't use more than 1/10 of memory for threads checking */ - readahead_kb = get_memory_size() / (10 * ctx->fs_num_threads); + readahead_kb = get_memory_size() / (10 * ctx->pfs_num_threads); /* maybe better disable RA if this is too small? */ if (ctx->readahead_kb > readahead_kb) ctx->readahead_kb = readahead_kb; @@ -1373,7 +1375,7 @@ static errcode_t e2fsck_pass1_prepare(e2fsck_t ctx) #ifdef CONFIG_PFSCK pthread_rwlock_init(&ctx->fs_fix_rwlock, NULL); pthread_rwlock_init(&ctx->fs_block_map_rwlock, NULL); - if (ctx->fs_num_threads > 1) + if (ctx->pfs_num_threads > 1) ctx->fs_need_locking = 1; #endif @@ -1633,7 +1635,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) if (ctx->global_ctx) { if (ctx->options & E2F_OPT_DEBUG && ctx->options & E2F_OPT_MULTITHREAD) - fprintf(stderr, "thread %d jumping to group %d\n", + fprintf(stderr, "thread %d jumping to group %u\n", ctx->thread_info.et_thread_index, ctx->thread_info.et_group_start); pctx.errcode = ext2fs_inode_scan_goto_blockgroup(scan, @@ -3129,11 +3131,11 @@ static int e2fsck_pass1_thread_join(e2fsck_t global_ctx, e2fsck_t thread_ctx) static int e2fsck_pass1_threads_join(struct e2fsck_thread_info *infos, e2fsck_t global_ctx) { - errcode_t rc; - errcode_t ret = 0; - int i; - struct e2fsck_thread_info *pinfo; - int num_threads = global_ctx->fs_num_threads; + errcode_t rc; + errcode_t ret = 0; + struct e2fsck_thread_info *pinfo; + int num_threads = global_ctx->pfs_num_threads; + int i; /* merge invalid bitmaps will recalculate it */ global_ctx->invalid_bitmaps = 0; @@ -3199,7 +3201,7 @@ static void *e2fsck_pass1_thread(void *arg) out: if (thread_ctx->options & E2F_OPT_MULTITHREAD) log_out(thread_ctx, - _("Scanned group range [%lu, %lu), inodes %lu\n"), + _("Scanned group range [%u, %u), inodes %u\n"), thread_ctx->thread_info.et_group_start, thread_ctx->thread_info.et_group_end, thread_ctx->thread_info.et_inode_number); @@ -3225,7 +3227,7 @@ static int e2fsck_pass1_threads_start(struct e2fsck_thread_info **pinfo, int i; e2fsck_t thread_ctx; dgrp_t average_group; - int num_threads = global_ctx->fs_num_threads; + int num_threads = global_ctx->pfs_num_threads; #ifdef DEBUG_THREADS struct e2fsck_thread_debug thread_debug = {PTHREAD_MUTEX_INITIALIZER, PTHREAD_COND_INITIALIZER, 0}; @@ -3329,8 +3331,7 @@ void e2fsck_pass1(e2fsck_t ctx) if (retval) return; #ifdef CONFIG_PFSCK - if (ctx->fs_num_threads > 1 || - ctx->options & E2F_OPT_MULTITHREAD) { + if (ctx->pfs_num_threads > 1 || ctx->options & E2F_OPT_MULTITHREAD) { need_single = 0; e2fsck_pass1_multithread(ctx); } diff --git a/e2fsck/unix.c b/e2fsck/unix.c index cd31bcd5..bebc19ed 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -908,12 +908,12 @@ static errcode_t PRS(int argc, char *argv[], e2fsck_t *ret_ctx) _("Invalid multiple thread num.\n")); if (thread_num > E2FSCK_MAX_THREADS) { fprintf(stderr, - _("threads %lu too large (max %lu)\n"), + _("threads %lu too large (max %u)\n"), thread_num, E2FSCK_MAX_THREADS); fatal_error(ctx, 0); } ctx->options |= E2F_OPT_MULTITHREAD; - ctx->fs_num_threads = thread_num; + ctx->pfs_num_threads = thread_num; break; #endif case 'n': diff --git a/lib/ext2fs/ext2fs.h b/lib/ext2fs/ext2fs.h index 616c9412..f74303f4 100644 --- a/lib/ext2fs/ext2fs.h +++ b/lib/ext2fs/ext2fs.h @@ -254,12 +254,11 @@ struct struct_ext2_filsys { time_t now; int cluster_ratio_bits; __u16 default_bitmap_type; - __u16 pad; - __u32 fs_num_threads; + __u16 fs_num_threads; /* * Reserved for future expansion */ - __u32 reserved[4]; + __u32 reserved[5]; /* * Reserved for the use of the calling application. From patchwork Wed Nov 18 15:39:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402364 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=gJ2IzK5c; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CbnB36kcsz9sPB for ; Thu, 19 Nov 2020 02:42:19 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727649AbgKRPmT (ORCPT ); Wed, 18 Nov 2020 10:42:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727629AbgKRPmT (ORCPT ); Wed, 18 Nov 2020 10:42:19 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B14CAC0613D4 for ; Wed, 18 Nov 2020 07:42:17 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id 7so1327030pjm.3 for ; Wed, 18 Nov 2020 07:42:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=1P7rffJ7thv4qkZFeY0lM2jm0Zu/jaf+KpsPdqhTum0=; b=gJ2IzK5clZe7HMWQUhCGfcogPLtSwZLHJsrOM02Q+Gw6ruTBCe7RT+fbjyJnEzUBNx dpJAEhMzUb51oJceXvsqxV3VXONmqf6EMOiwpAWXNA1Qt9FUW6GqOO1/885mO5lLPWwN 1eJXBdPaH5OoRA0VGPQkE1TBHsssYm+0KET3kVJeR/UBJ7GiDkWoJYcTHJbJdqMq0j+U r4zSvjAMRIvyquVoVYVIR4ODtlT8NBe+O6eYjB2VfKLt815SQmvNU00pK4UWpDfNPhfj 4kHd+BI/nIGOOOiSIHR9aPORWz1WxsJXTt9bdOsiFVtTHwnGEoB/nnDqSnTyFcC/M3tt uLJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1P7rffJ7thv4qkZFeY0lM2jm0Zu/jaf+KpsPdqhTum0=; b=E6RhyhGE/69aDu2Qkvd303sWGomw8xrzYAbwlLM9vP5+qovqqM5oGW19Iyfjgy0k1F Jt1s/HdtEYWQFEItc7HXDaqniySnhmT+fbMSs/MA6Ch8lHhk1usxY2fGyGnSQ8RU3uh8 +ckc/VPitUp6T6aiR28+1r0u5j4J33YL6IXq9AZjbxhuXqgs0njHacsDkbpo+zMStY5C MgcwK/lKkZl8XWNiuPawn7SgtOhrqfnPbfFxlZse0DeSEYGC7KoN10uJKeEGgpUEFlO1 VEAtXKz8HmEOg6hJk99p/YkuwxwTaKh5bWdb9pegAhUzCObAnGD5ltrtfeBuzCvW3a26 2TCQ== X-Gm-Message-State: AOAM5316b8nc5hhs5/5kywCXkfVEsRyERbfiNq8MCrPJ8SGM7IA55uHD 8xl5XN0xZYFWBzl0JHIvLdf2oN+nW5MqAYXe02o0833dEK0wxbc3dKRRIl4XX/1QY9v9TmDWExn CAEwXQN70iK5XcrfGDHu4ppLZ4LJ/J7oTkN5VF8LXYHz/zVQkZn/ohJN2e4WOVkXziLoJ9KBmAl XNb/dS51U= X-Google-Smtp-Source: ABdhPJzGs0m+TpJGTDHeg9e/hYdqtLqX7DbqHbVe9DDsewMMxerDMup+M4w0g+aR9555onHXatU5p8Yv0hM9tB+H2HI= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a63:c00b:: with SMTP id h11mr8488387pgg.7.1605714137104; Wed, 18 Nov 2020 07:42:17 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:45 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-60-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 59/61] e2fsck: update mmp block race From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Wang Shilong , Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org From: Wang Shilong Update mmp block is only expected in one thread, @mmp_update_thread is used to get/set active thread number, however this should be set globally shared with different threads rather than be private per thread. Signed-off-by: Wang Shilong Signed-off-by: Saranya Muruganandam --- e2fsck/pass1.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/e2fsck/pass1.c b/e2fsck/pass1.c index 8d4e2675..a51fe20f 100644 --- a/e2fsck/pass1.c +++ b/e2fsck/pass1.c @@ -1501,6 +1501,7 @@ void e2fsck_pass1_run(e2fsck_t ctx) struct ea_quota ea_ibody_quota; struct process_inode_block *inodes_to_process; int process_inode_count, check_mmp; + e2fsck_t global_ctx = ctx->global_ctx ? ctx->global_ctx : ctx; init_resource_track(&rtrack, ctx->fs->io); clear_problem_context(&pctx); @@ -1652,14 +1653,11 @@ void e2fsck_pass1_run(e2fsck_t ctx) check_mmp = 0; e2fsck_pass1_check_lock(ctx); #ifdef CONFIG_PFSCK - if (!ctx->mmp_update_thread) { + if (!global_ctx->mmp_update_thread) { e2fsck_pass1_block_map_w_lock(ctx); - if (!ctx->mmp_update_thread) { - if (ctx->global_ctx) - ctx->mmp_update_thread = - ctx->thread_info.et_thread_index + 1; - else - ctx->mmp_update_thread = 1; + if (!global_ctx->mmp_update_thread) { + global_ctx->mmp_update_thread = + ctx->thread_info.et_thread_index + 1; check_mmp = 1; } e2fsck_pass1_block_map_w_unlock(ctx); @@ -1667,8 +1665,8 @@ void e2fsck_pass1_run(e2fsck_t ctx) /* only one active thread could update mmp block. */ e2fsck_pass1_block_map_r_lock(ctx); - if (!ctx->global_ctx || ctx->mmp_update_thread == - (ctx->thread_info.et_thread_index + 1)) + if (global_ctx->mmp_update_thread == + ctx->thread_info.et_thread_index + 1) check_mmp = 1; e2fsck_pass1_block_map_r_unlock(ctx); #else @@ -2396,8 +2394,8 @@ endit: #ifdef CONFIG_PFSCK /* reset update_thread after this thread exit */ e2fsck_pass1_block_map_w_lock(ctx); - if (ctx->mmp_update_thread) - ctx->mmp_update_thread = 0; + if (check_mmp) + global_ctx->mmp_update_thread = 0; e2fsck_pass1_block_map_w_unlock(ctx); #endif } From patchwork Wed Nov 18 15:39:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402365 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=Xl/oPQbh; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CbnB51sW5z9sPB for ; Thu, 19 Nov 2020 02:42:21 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727651AbgKRPmU (ORCPT ); Wed, 18 Nov 2020 10:42:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727398AbgKRPmU (ORCPT ); Wed, 18 Nov 2020 10:42:20 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1818FC0613D4 for ; Wed, 18 Nov 2020 07:42:20 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id r6so1425187pfg.4 for ; Wed, 18 Nov 2020 07:42:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=TgxfBleuwBbiJ7V0ZDGA7yS/l34SxlrP250dNmU9++8=; b=Xl/oPQbhi6gbyqrHnE17F2l4yTLlEIK2hoomXsbGA3scog68er50CI5e57yewMtZyn P2QSePYFFfQkGI4g/ER+ph5u1B2O5uVnrvLCp5WREbUT91KbVF1/n9sFNQN7ItVApMD4 JRj4zpFhxgpSLVXrYXvoDhesnUpboihORzeQ9WH22E3PNkj7g7cQ8LZQcrx5cMinI2lf flFuTdrM2nHPMvAJmZy1fkwjOqgeXkH77fL/nyJCdrlV66DxO07rojiMBOWW9nTCqsK5 eFgazpnUrXaj4Gm2tTIL8TCf+Ji+vV52pH1WpTNbsN7PsG/fyMqYnDzenpphd6zKBu1D 1Dgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=TgxfBleuwBbiJ7V0ZDGA7yS/l34SxlrP250dNmU9++8=; b=Vf1GBdUb+oBWG+iDcJOY+sMmQd222oepBsQzil8+ztw2LSvUoRr76qphmVFFfqc2MW X9qR+F8T9FOwawvyAmW90vKIJBsatcKFvT19d6RyolRlnpQTKpXMD68XgfFTsWL70C6Q YUUvQbXca+N0egw0Yogvo3y+IGeR9o6xvQKfIYMQbDkisFEV707I4fcXPrWJLubxmAje TjGx2gkdVG8b+pIZM7V65mYGOt4VGeRG7tff/Br0i7EGKBCbPzK/PfnDiV3sTCVy50EV w8N9Lfy5g+6wDeJKJBl2ARARvW15/G2KvJEyrJMDweE63Y87TXI/4QoqUpZS7BZ6EWxc Usgw== X-Gm-Message-State: AOAM531VmOYTBYlaaVkvGITTH7+qjl4GXi2XGwVhI28MsV3RK2dSUjS9 +sDhpPhqoA7H/6rLhaDsEzTULsnL+N2bVWpF4BqNgo6Y2tRuPa56vZQpLVbj8KEPJO6NqV9Y+IU MDSvqdhQeCLx0kBUgefLW3USl7BmQUlc+5a1b8WYfiJN+OPaUq1bZ/N2jZqWWXRve4q2raY1pkK A//+lTm9A= X-Google-Smtp-Source: ABdhPJz/klBme1zJoxTP83PgdK8RZQ9N0XsSoQWy07736wuknI77Bj7gk1cgO+sRZ87P1TZI5TxxelBJzOqn8YYzgRk= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a17:90b:e04:: with SMTP id ge4mr46609pjb.0.1605714139152; Wed, 18 Nov 2020 07:42:19 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:46 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-61-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 60/61] e2fsck: propagate number of threads From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Sometimes, such as in orphan_inode case, e2fsck_pass1 is called after reading the block bitmaps. This results in reading the block bitmap sequentially and multithreading only gets kicked in later. Fix the thread count earlier while setting up the file system. Signed-off-by: Saranya Muruganandam --- e2fsck/unix.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/e2fsck/unix.c b/e2fsck/unix.c index bebc19ed..a2c6a178 100644 --- a/e2fsck/unix.c +++ b/e2fsck/unix.c @@ -1729,6 +1729,9 @@ failure: ctx->fs = fs; fs->now = ctx->now; +#ifdef CONFIG_PFSCK + fs->fs_num_threads = ctx->pfs_num_threads; +#endif sb = fs->super; if (sb->s_rev_level > E2FSCK_CURRENT_REV) { From patchwork Wed Nov 18 15:39:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saranya Muruganandam X-Patchwork-Id: 1402366 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=linux-ext4-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=KkbP+bO7; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 4CbnB82snDz9sTc for ; Thu, 19 Nov 2020 02:42:24 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727457AbgKRPmY (ORCPT ); Wed, 18 Nov 2020 10:42:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726352AbgKRPmX (ORCPT ); Wed, 18 Nov 2020 10:42:23 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00B64C0613D4 for ; Wed, 18 Nov 2020 07:42:21 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id r4so2042384ybs.1 for ; Wed, 18 Nov 2020 07:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=/y900TGUU79g0E/kX5hTm3DULqidupZjxJK3msemDmE=; b=KkbP+bO7kFH0Jrjmpv41zNcxXUEqgOxCSgPYkH4cJlaMIBxhthy0wnrKWqGUQbZalV 4CQ+6uP5RBnaP2u6yBzNvrvNB7PtUzvEE/vkzlAkIs11F9TqSikwlcy1isVJaRyJvyla VJS3h6o0Rc19NRVHUueCHntoabuzjOQDkuktQHrAVHZO650uTPvNiyYwupK8mHflnJNE PSdTkGk3wGXrDBcsgPV80W8ngOa05EyoLk9jsy19OjLdB47XXkwCFiC+njW2SXfCUJHL T086b4bL1mI+haVAz0V838l9w42IPXISbObYpgivMyTS8BGLkMbNfR/mfL52AU/YFYq5 z9zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/y900TGUU79g0E/kX5hTm3DULqidupZjxJK3msemDmE=; b=XJFNTvjAXs1DoM7ABBAU+opj4eQGOvQ8jUKBUzJ3p5C2U7echav/pPdf7DZp1+pilh TQzU+Hz8o/4fzjPFdaKO0qn4rsmZ4IUJLPa7CsQGWjNQOtrsAb3/GV8ce5cquwRK5QhW kUpqLjkev7d63mDLMaNsT5dLZGkp4b+Ume5a2L2MP5bgZLpVFaZtNoFJON7i6Jg9IgJ7 bhtn8q/TKlPEBnEu6QcrfO3+AVOj2sHxAekZjchDwn/TDbDCK0xX5KC3V/HTzVMZkSg1 BddYFxD0/WxNomY8czexIb3XVxaBQ8TWgbPCGbny3bcyY6Ku4Hs4cE0vQXiqwm98dBEV 6CHA== X-Gm-Message-State: AOAM531q1LsS8BXdsl+Hgn17cyrPMGyoSXQicpHFm6D+gbrWxoELbfI3 Nke6V0UbkCvYGnBOdNjSX9BAYpiC0rOmnx3qmWAxFGVeLuUhLZbilchsPd2X1lIn618Q2OFAx9w E3UGUOCIXwYLCBXCli40c3nLWfVjmBlF8ioNfVgTm2WeB/VyAUxMXxZRIAKjzJZ4B2+vtOzCL10 mHz8ORFhE= X-Google-Smtp-Source: ABdhPJwDHI+E6Qy/OtDhWpbEKt1d1Z8T9cYgOSP//oQQK2YCWncOTj89y32s4BxbT0n6eQTh/LhtpDsQc46SPV1sNT8= Sender: "saranyamohan via sendgmr" X-Received: from saranyamohan.svl.corp.google.com ([100.116.76.178]) (user=saranyamohan job=sendgmr) by 2002:a25:4249:: with SMTP id p70mr7364470yba.259.1605714141172; Wed, 18 Nov 2020 07:42:21 -0800 (PST) Date: Wed, 18 Nov 2020 07:39:47 -0800 In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com> Message-Id: <20201118153947.3394530-62-saranyamohan@google.com> Mime-Version: 1.0 References: <20201118153947.3394530-1-saranyamohan@google.com> X-Mailer: git-send-email 2.29.2.299.gdc1121823c-goog Subject: [RFC PATCH v3 61/61] e2fsck: Annotating fields in e2fsck_struct From: Saranya Muruganandam To: linux-ext4@vger.kernel.org, tytso@mit.edu Cc: adilger.kernel@dilger.ca, Saranya Muruganandam Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org Adding information on fields in e2fsck_struct on how they are used when running parallel fsck. Signed-off-by: Saranya Muruganandam --- e2fsck/e2fsck.h | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/e2fsck/e2fsck.h b/e2fsck/e2fsck.h index 362e128c..f15c383d 100644 --- a/e2fsck/e2fsck.h +++ b/e2fsck/e2fsck.h @@ -251,7 +251,7 @@ struct e2fsck_thread { struct e2fsck_struct { /* Global context to get the cancel flag */ e2fsck_t global_ctx; - ext2_filsys fs; + ext2_filsys fs; /* [fs_fix_rwlock] */ const char *program_name; char *filesystem_name; char *device_name; @@ -260,7 +260,9 @@ struct e2fsck_struct { char *log_fn; FILE *problem_logf; char *problem_log_fn; - int flags; /* E2fsck internal flags */ + /* E2fsck internal flags. + * shared by different threads for pass1 [fs_fix_rwlock] */ + int flags; int options; unsigned blocksize; /* blocksize */ blk64_t use_superblock; /* sb requested by user */ @@ -281,6 +283,7 @@ struct e2fsck_struct { int (*progress)(e2fsck_t ctx, int pass, unsigned long cur, unsigned long max); + /* The following inode bitmaps are separately used in thread_ctx Pass1*/ ext2fs_inode_bitmap inode_used_map; /* Inodes which are in use */ ext2fs_inode_bitmap inode_bad_map; /* Inodes which are bad somehow */ ext2fs_inode_bitmap inode_dir_map; /* Inodes which are directories */ @@ -288,12 +291,14 @@ struct e2fsck_struct { ext2fs_inode_bitmap inode_imagic_map; /* AFS inodes */ ext2fs_inode_bitmap inode_reg_map; /* Inodes which are regular files*/ + /* Following 3 protected by [fs_block_map_rwlock] */ ext2fs_block_bitmap block_found_map; /* Blocks which are in use */ ext2fs_block_bitmap block_dup_map; /* Blks referenced more than once */ ext2fs_block_bitmap block_ea_map; /* Blocks which are used by EA's */ /* - * Inode count arrays + * Inode count arrays. + * Separately used in thread_ctx, pass1 */ ext2_icount_t inode_count; ext2_icount_t inode_link_info; @@ -315,7 +320,8 @@ struct e2fsck_struct { /* * Array of flags indicating whether an inode bitmap, block - * bitmap, or inode table is invalid + * bitmap, or inode table is invalid. + * Separately used in thread_ctx, pass1 */ int *invalid_inode_bitmap_flag; int *invalid_block_bitmap_flag; @@ -328,7 +334,8 @@ struct e2fsck_struct { char *block_buf; /* - * For pass1_check_directory and pass1_get_blocks + * For pass1_check_directory and pass1_get_blocks. + * Separately used in thread_ctx in pass1 */ ext2_ino_t stashed_ino; struct ext2_inode *stashed_inode; @@ -387,6 +394,7 @@ struct e2fsck_struct { /* * How we display the progress update (for unix) + * shared by different threads for pass1 [fs_fix_rwlock] */ int progress_fd; int progress_pos; @@ -395,7 +403,7 @@ struct e2fsck_struct { int interactive; /* Are we connected directly to a tty? */ char start_meta[2], stop_meta[2]; - /* File counts */ + /* File counts. Separately used in thread_ctx, pass1 */ __u32 fs_directory_count; __u32 fs_regular_count; __u32 fs_blockdev_count;