From patchwork Mon Jul 15 22:57:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Connor Kuehl X-Patchwork-Id: 1132335 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45nf7N6ZvGz9sNf; Tue, 16 Jul 2019 08:57:36 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1hn9uW-0006Z3-Od; Mon, 15 Jul 2019 22:57:32 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.86_2) (envelope-from ) id 1hn9uU-0006YZ-6y for kernel-team@lists.ubuntu.com; Mon, 15 Jul 2019 22:57:30 +0000 Received: from mail-pl1-f197.google.com ([209.85.214.197]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1hn9uT-00077f-Lr for kernel-team@lists.ubuntu.com; Mon, 15 Jul 2019 22:57:29 +0000 Received: by mail-pl1-f197.google.com with SMTP id d6so9017281pls.17 for ; Mon, 15 Jul 2019 15:57:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=njdvTWtQUZOiY2nfYt/qtOkKH1wRf1qRZej6rtkukK0=; b=cfqyONY0hNv8ngWqw+Nt9Al9+CNPKkLL5s51H/BDhiVw0bW6LPM0OPFlPsEyA36ki6 BWEYhG1M8NimAsjLkIiEdTdfg1+w75dR6jljuwc5+/TE9Gc4zMHak7WlWoBumfbNHn8y Uv3IcK9ZaLYflHkTQMK5+FLhw32H0u51nPUMCF1MeulRnk8paUanurs11PdQ6qm2rUuy NXgh+hpbYPDG8pWfp+bZUm+3wdS+kSMoySx/bHoMCeDE+NYQGFRWYFc2vgbjx1RPHBr2 d/T2Frk1UYpyRQQI5swdm9h/P93nuNimSnyQ1tlb2HM53j8l7H+tib7u7yMPnfIB/9qc an0A== X-Gm-Message-State: APjAAAXK2ZZqDrrSPFZkvFiBBHh7GFbhJXqDHd1zFnYDMUp6vfFmt/PM Qzqc9Rq3+p24YAPMmSm++rISn7NR2IkndpFQuLhR7d20hdXQG9FMLIMo8tASiBcweNfAWxpAM0E h2oH0EQ4nmiHHO/ff/miL0SkrJK5rlgtvNW9YEuxakw== X-Received: by 2002:a17:902:2ae8:: with SMTP id j95mr28575632plb.276.1563231448062; Mon, 15 Jul 2019 15:57:28 -0700 (PDT) X-Google-Smtp-Source: APXvYqyew5jf8Wr8YlH+jty/d5KIHeWGb196GTsw6vi/0IstIrU9sk6qsCpYiI9DmcqNXmWGXXk+hQ== X-Received: by 2002:a17:902:2ae8:: with SMTP id j95mr28575619plb.276.1563231447843; Mon, 15 Jul 2019 15:57:27 -0700 (PDT) Received: from localhost.localdomain (c-71-63-131-226.hsd1.or.comcast.net. [71.63.131.226]) by smtp.gmail.com with ESMTPSA id f12sm15676407pgq.52.2019.07.15.15.57.26 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 15 Jul 2019 15:57:26 -0700 (PDT) From: Connor Kuehl To: kernel-team@lists.ubuntu.com Subject: [SRU][Bionic][PATCH 1/1] md: fix raid10 hang issue caused by barrier Date: Mon, 15 Jul 2019 15:57:24 -0700 Message-Id: <20190715225724.21890-2-connor.kuehl@canonical.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190715225724.21890-1-connor.kuehl@canonical.com> References: <20190715225724.21890-1-connor.kuehl@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Guoqing Jiang BugLink: https://bugs.launchpad.net/bugs/1767992 When both regular IO and resync IO happen at the same time, and if we also need to split regular. Then we can see tasks hang due to barrier. 1. resync thread [ 1463.757205] INFO: task md1_resync:5215 blocked for more than 480 seconds. [ 1463.757207] Not tainted 4.19.5-1-default #1 [ 1463.757209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1463.757212] md1_resync D 0 5215 2 0x80000000 [ 1463.757216] Call Trace: [ 1463.757223] ? __schedule+0x29a/0x880 [ 1463.757231] ? raise_barrier+0x8d/0x140 [raid10] [ 1463.757236] schedule+0x78/0x110 [ 1463.757243] raise_barrier+0x8d/0x140 [raid10] [ 1463.757248] ? wait_woken+0x80/0x80 [ 1463.757257] raid10_sync_request+0x1f6/0x1e30 [raid10] [ 1463.757265] ? _raw_spin_unlock_irq+0x22/0x40 [ 1463.757284] ? is_mddev_idle+0x125/0x137 [md_mod] [ 1463.757302] md_do_sync.cold.78+0x404/0x969 [md_mod] [ 1463.757311] ? wait_woken+0x80/0x80 [ 1463.757336] ? md_rdev_init+0xb0/0xb0 [md_mod] [ 1463.757351] md_thread+0xe9/0x140 [md_mod] [ 1463.757358] ? _raw_spin_unlock_irqrestore+0x2e/0x60 [ 1463.757364] ? __kthread_parkme+0x4c/0x70 [ 1463.757369] kthread+0x112/0x130 [ 1463.757374] ? kthread_create_worker_on_cpu+0x40/0x40 [ 1463.757380] ret_from_fork+0x3a/0x50 2. regular IO [ 1463.760679] INFO: task kworker/0:8:5367 blocked for more than 480 seconds. [ 1463.760683] Not tainted 4.19.5-1-default #1 [ 1463.760684] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1463.760687] kworker/0:8 D 0 5367 2 0x80000000 [ 1463.760718] Workqueue: md submit_flushes [md_mod] [ 1463.760721] Call Trace: [ 1463.760731] ? __schedule+0x29a/0x880 [ 1463.760741] ? wait_barrier+0xdd/0x170 [raid10] [ 1463.760746] schedule+0x78/0x110 [ 1463.760753] wait_barrier+0xdd/0x170 [raid10] [ 1463.760761] ? wait_woken+0x80/0x80 [ 1463.760768] raid10_write_request+0xf2/0x900 [raid10] [ 1463.760774] ? wait_woken+0x80/0x80 [ 1463.760778] ? mempool_alloc+0x55/0x160 [ 1463.760795] ? md_write_start+0xa9/0x270 [md_mod] [ 1463.760801] ? try_to_wake_up+0x44/0x470 [ 1463.760810] raid10_make_request+0xc1/0x120 [raid10] [ 1463.760816] ? wait_woken+0x80/0x80 [ 1463.760831] md_handle_request+0x121/0x190 [md_mod] [ 1463.760851] md_make_request+0x78/0x190 [md_mod] [ 1463.760860] generic_make_request+0x1c6/0x470 [ 1463.760870] raid10_write_request+0x77a/0x900 [raid10] [ 1463.760875] ? wait_woken+0x80/0x80 [ 1463.760879] ? mempool_alloc+0x55/0x160 [ 1463.760895] ? md_write_start+0xa9/0x270 [md_mod] [ 1463.760904] raid10_make_request+0xc1/0x120 [raid10] [ 1463.760910] ? wait_woken+0x80/0x80 [ 1463.760926] md_handle_request+0x121/0x190 [md_mod] [ 1463.760931] ? _raw_spin_unlock_irq+0x22/0x40 [ 1463.760936] ? finish_task_switch+0x74/0x260 [ 1463.760954] submit_flushes+0x21/0x40 [md_mod] So resync io is waiting for regular write io to complete to decrease nr_pending (conf->barrier++ is called before waiting). The regular write io splits another bio after call wait_barrier which call nr_pending++, then the splitted bio would continue with raid10_write_request -> wait_barrier, so the splitted bio has to wait for barrier to be zero, then deadlock happens as follows. resync io regular io raise_barrier wait_barrier generic_make_request wait_barrier To resolve the issue, we need to call allow_barrier to decrease nr_pending before generic_make_request since regular IO is not issued to underlying devices, and wait_barrier is called again to ensure no internal IO happening. Fixes: fc9977dd069e ("md/raid10: simplify the splitting of requests.") Reported-and-tested-by: SiniĊĦa Bandin Signed-off-by: Guoqing Jiang Signed-off-by: Shaohua Li (cherry picked from commit e820d55cb99dd93ac2dc949cf486bb187e5cd70d) Signed-off-by: Connor Kuehl --- drivers/md/raid10.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index f617c91e915a..150bc623af46 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -1194,7 +1194,9 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, struct bio *split = bio_split(bio, max_sectors, gfp, conf->bio_split); bio_chain(split, bio); + allow_barrier(conf); generic_make_request(bio); + wait_barrier(conf); bio = split; r10_bio->master_bio = bio; r10_bio->sectors = max_sectors; @@ -1499,7 +1501,9 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, struct bio *split = bio_split(bio, r10_bio->sectors, GFP_NOIO, conf->bio_split); bio_chain(split, bio); + allow_barrier(conf); generic_make_request(bio); + wait_barrier(conf); bio = split; r10_bio->master_bio = bio; }