From patchwork Tue Jun 6 10:28:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gerald Yang X-Patchwork-Id: 1791051 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=FQXNOJCg; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Qb6C930GTz20X0 for ; Tue, 6 Jun 2023 20:28:53 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1q6Tvh-0005sn-TF; Tue, 06 Jun 2023 10:28:45 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1q6Tvf-0005s1-Lc for kernel-team@lists.ubuntu.com; Tue, 06 Jun 2023 10:28:43 +0000 Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 73A923F137 for ; Tue, 6 Jun 2023 10:28:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1686047323; bh=GQakCYPPVd/VCcY0IKFtGEJyInnRoZ3ihL5b5J0BiNg=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FQXNOJCgrWku97G+zF+eIQ/V+JnD9j5BIXqutx9IPLWiixzEY3IJpiGou8kMHnZ0M CXMckS4jjLzJfaLpNE0iI3lweKnog7qbaKHz5qQKQc3Dn1ThdfNGQ9bEDIln3QVLJ7 Br/u2q6kQOluvMGNsSshkvAjbnQd1V1Zy6tSrBNySeWQ2iijS9Y1CJ8Agfh95xgy5/ GuWGuGpUQyShr2XOokcBk7Oc9qSYg+9Dvw7lO1WBgW+E3Iwsn3Z2PH4Qxa41yLpUaj idbSjInJAZKbRg29MwG8elgyG2IO8b6jkWmI0vmTSUvtTOzUzF5s/TKLi3spSDjqsy R0iL92VcafSFg== Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-256563a2097so2000522a91.0 for ; Tue, 06 Jun 2023 03:28:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686047321; x=1688639321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GQakCYPPVd/VCcY0IKFtGEJyInnRoZ3ihL5b5J0BiNg=; b=KO3/zfya2Ek0PV/V+fga44R+Bc9ZM2qsmY1imh3TRsRi5Tw+ypjecQNPG4cJ6Bh86N 138iYBhFVtBaKKXBDUTlQDbuDMc3rBpGvzzF60J090uX3kGIoH1/oxkQ6vG5hg2Ty2DF IOetpWL5NRjIlPBBSdmArQ62K/8O79geNP3YqyIjk5LLiOTS07pzY56SMf+znLqvN7dW AebfMbgqAfBCcxvbqvHC/SD8qqWTy53RFfM+pCQvvcD2bo0DT5ZQOfzOwGkEWF87Wk/f d1MsxF6AhHif90spRQ2sIfpGplsI09BDGLnfc6RLDYDtFE/LRYeJM4kiJMdmlIHGYdGA FhiQ== X-Gm-Message-State: AC+VfDzjX5/jLsJPB9GAmoVYSFpX815l7nGEjyO9lDO2wl3ooVj7wAPY zcp1uXvBPossuVYgT5uoZDOmSe7lQVtKFuBHVdlsIjydJZZ8uJoRxlhw6qbhwme8CeqMwOng2UF at/vagxQ6/eUawWUSoEHNtpiz+TR13icdhQE3HwmQmdeR4ZE0Vw== X-Received: by 2002:a17:90a:51:b0:259:3e26:b5d9 with SMTP id 17-20020a17090a005100b002593e26b5d9mr347610pjb.2.1686047321045; Tue, 06 Jun 2023 03:28:41 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4eos7XSgkynHlejNzA0YTHvhHMASC/upigf2z6ybga7NgYOuWHvQmnDj1m/Ddwbg5BFMsd3A== X-Received: by 2002:a17:90a:51:b0:259:3e26:b5d9 with SMTP id 17-20020a17090a005100b002593e26b5d9mr347608pjb.2.1686047320624; Tue, 06 Jun 2023 03:28:40 -0700 (PDT) Received: from localhost.localdomain (220-135-31-21.hinet-ip.hinet.net. [220.135.31.21]) by smtp.gmail.com with ESMTPSA id 8-20020a17090a0c0800b00256a6ec8507sm9777629pjs.19.2023.06.06.03.28.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jun 2023 03:28:40 -0700 (PDT) From: Gerald Yang To: kernel-team@lists.ubuntu.com Subject: [SRU][K][PATCH 1/6] sbitmap: fix possible io hung due to lost wakeup Date: Tue, 6 Jun 2023 18:28:23 +0800 Message-Id: <20230606102828.218014-2-gerald.yang@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230606102828.218014-1-gerald.yang@canonical.com> References: <20230606102828.218014-1-gerald.yang@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Yu Kuai BugLink: https://bugs.launchpad.net/bugs/2022318 There are two problems can lead to lost wakeup: 1) invalid wakeup on the wrong waitqueue: For example, 2 * wake_batch tags are put, while only wake_batch threads are woken: __sbq_wake_up atomic_cmpxchg -> reset wait_cnt __sbq_wake_up -> decrease wait_cnt ... __sbq_wake_up -> wait_cnt is decreased to 0 again atomic_cmpxchg sbq_index_atomic_inc -> increase wake_index wake_up_nr -> wake up and waitqueue might be empty sbq_index_atomic_inc -> increase again, one waitqueue is skipped wake_up_nr -> invalid wake up because old wakequeue might be empty To fix the problem, increasing 'wake_index' before resetting 'wait_cnt'. 2) 'wait_cnt' can be decreased while waitqueue is empty As pointed out by Jan Kara, following race is possible: CPU1 CPU2 __sbq_wake_up __sbq_wake_up sbq_wake_ptr() sbq_wake_ptr() -> the same wait_cnt = atomic_dec_return() /* decreased to 0 */ sbq_index_atomic_inc() /* move to next waitqueue */ atomic_set() /* reset wait_cnt */ wake_up_nr() /* wake up on the old waitqueue */ wait_cnt = atomic_dec_return() /* * decrease wait_cnt in the old * waitqueue, while it can be * empty. */ Fix the problem by waking up before updating 'wake_index' and 'wait_cnt'. With this patch, noted that 'wait_cnt' is still decreased in the old empty waitqueue, however, the wakeup is redirected to a active waitqueue, and the extra decrement on the old empty waitqueue is not handled. Fixes: 88459642cba4 ("blk-mq: abstract tag allocation out into sbitmap library") Signed-off-by: Yu Kuai Reviewed-by: Jan Kara Link: https://lore.kernel.org/r/20220803121504.212071-1-yukuai1@huaweicloud.com Signed-off-by: Jens Axboe (cherry picked from commit 040b83fcecfb86f3225d3a5de7fd9b3fbccf83b4) Signed-off-by: Gerald Yang --- lib/sbitmap.c | 55 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 22 deletions(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index 29eb0484215a..1f31147872e6 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -611,32 +611,43 @@ static bool __sbq_wake_up(struct sbitmap_queue *sbq) return false; wait_cnt = atomic_dec_return(&ws->wait_cnt); - if (wait_cnt <= 0) { - int ret; + /* + * For concurrent callers of this, callers should call this function + * again to wakeup a new batch on a different 'ws'. + */ + if (wait_cnt < 0 || !waitqueue_active(&ws->wait)) + return true; - wake_batch = READ_ONCE(sbq->wake_batch); + if (wait_cnt > 0) + return false; - /* - * Pairs with the memory barrier in sbitmap_queue_resize() to - * ensure that we see the batch size update before the wait - * count is reset. - */ - smp_mb__before_atomic(); + wake_batch = READ_ONCE(sbq->wake_batch); - /* - * For concurrent callers of this, the one that failed the - * atomic_cmpxhcg() race should call this function again - * to wakeup a new batch on a different 'ws'. - */ - ret = atomic_cmpxchg(&ws->wait_cnt, wait_cnt, wake_batch); - if (ret == wait_cnt) { - sbq_index_atomic_inc(&sbq->wake_index); - wake_up_nr(&ws->wait, wake_batch); - return false; - } + /* + * Wake up first in case that concurrent callers decrease wait_cnt + * while waitqueue is empty. + */ + wake_up_nr(&ws->wait, wake_batch); - return true; - } + /* + * Pairs with the memory barrier in sbitmap_queue_resize() to + * ensure that we see the batch size update before the wait + * count is reset. + * + * Also pairs with the implicit barrier between decrementing wait_cnt + * and checking for waitqueue_active() to make sure waitqueue_active() + * sees result of the wakeup if atomic_dec_return() has seen the result + * of atomic_set(). + */ + smp_mb__before_atomic(); + + /* + * Increase wake_index before updating wait_cnt, otherwise concurrent + * callers can see valid wait_cnt in old waitqueue, which can cause + * invalid wakeup on the old waitqueue. + */ + sbq_index_atomic_inc(&sbq->wake_index); + atomic_set(&ws->wait_cnt, wake_batch); return false; } From patchwork Tue Jun 6 10:28:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gerald Yang X-Patchwork-Id: 1791053 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=Aj90e3pX; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Qb6CD2r83z20Ty for ; Tue, 6 Jun 2023 20:28:56 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1q6Tvk-0005vF-8a; Tue, 06 Jun 2023 10:28:48 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1q6Tvf-0005rt-EF for kernel-team@lists.ubuntu.com; Tue, 06 Jun 2023 10:28:43 +0000 Received: from mail-yb1-f197.google.com (mail-yb1-f197.google.com [209.85.219.197]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 336573F0CB for ; Tue, 6 Jun 2023 10:28:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1686047323; bh=v8itPFhmoWkgEqdEOrOo6zxzQ+nwLw766Wi+/FHI7Bs=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Aj90e3pXwfBs+ljjD4iHKkaEvG/8Dv0XkxEgbpVVImXpzKgWmu3/lOD1Qtj/c5ib0 8hIrJ8bpz55d726lREp9DUa76winYrGwBo5V9I7Y/SpHqjN50UKmNtyJaMu96DJJow jXVvxKp+lVTvhJen96IAkYPsmnLJZ4zEnAbBvnuO76wlmozfoh1JkuwB0gLm8cQRB+ V/L1jCCbo9jyKQRwXPiSvLqVqJU0Lx+hoIeV+/hcumE9UQg7INBHwR0Du89LDC1tUC eYcrlnRoFlULTO2/clHGfk5Fs4GobaE+XrTcwKTXlL4d3H/vvaA2YW6e7aaxQTYAsA yVlCubi2AKYNQ== Received: by mail-yb1-f197.google.com with SMTP id 3f1490d57ef6-bb2202e0108so7397200276.1 for ; Tue, 06 Jun 2023 03:28:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686047322; x=1688639322; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v8itPFhmoWkgEqdEOrOo6zxzQ+nwLw766Wi+/FHI7Bs=; b=MyCejCC2fGeW/Ehc/z09Dfod2QaelSbwXGhq5HXRhkm3WQq4hdRyqdJS12BLwm6SZ9 ch/oX8o9pdt+PXsIEO84w5u6dCbFJ5vxDBU0HpIqELtjMnGkAS/r3gtPW52uKWSd4iI/ +cGx5RfPaLZ3vxgihRy5JsATl0NKsNk0uOcYO7qPACRJNj/L8bKkpgDL/nKpp9Od8EPo xlCUMJP34kI+F/n9yHBNQpEBCIjfITQFrYf+jzvzlpP1Ppiq4dbHQ6uZM0ip4JcgEHb+ yUUUgxLAoPCpqRBYZvjNSzbLlmyUvLFZY89/JcAqJ8V+WRzc0E2med71mVxMYGVVALeD EUMA== X-Gm-Message-State: AC+VfDyGyKwmFtbMrldFM3GRMp2qwADxxVi11ky74Y40PlW1SsxnRX32 yydU8ivzWMj/V+uywEIGg5uW+QxKA50sfaVJCCfJObCABTZusc7ejsXUoBbuIuB5QdL6CJ7Yi84 i/W8iCY5z4v9TVOygxThs3NzihgjJNLL0UulYof7nkT2l7fA34g== X-Received: by 2002:a25:5341:0:b0:bb1:5766:1b80 with SMTP id h62-20020a255341000000b00bb157661b80mr1423776ybb.8.1686047321839; Tue, 06 Jun 2023 03:28:41 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ75AtSFtnr665bLMdU11llkg7FgrxRVCbs327n7pkowR1KM8WkUELWUqV0+mRfSSkwXKJ5+LA== X-Received: by 2002:a25:5341:0:b0:bb1:5766:1b80 with SMTP id h62-20020a255341000000b00bb157661b80mr1423761ybb.8.1686047321535; Tue, 06 Jun 2023 03:28:41 -0700 (PDT) Received: from localhost.localdomain (220-135-31-21.hinet-ip.hinet.net. [220.135.31.21]) by smtp.gmail.com with ESMTPSA id 8-20020a17090a0c0800b00256a6ec8507sm9777629pjs.19.2023.06.06.03.28.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jun 2023 03:28:41 -0700 (PDT) From: Gerald Yang To: kernel-team@lists.ubuntu.com Subject: [SRU][K][PATCH 2/6] sbitmap: remove unnecessary code in __sbitmap_queue_get_batch Date: Tue, 6 Jun 2023 18:28:24 +0800 Message-Id: <20230606102828.218014-3-gerald.yang@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230606102828.218014-1-gerald.yang@canonical.com> References: <20230606102828.218014-1-gerald.yang@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Liu Song BugLink: https://bugs.launchpad.net/bugs/2022318 If "nr + nr_tags <= map_depth", then the value of nr_tags will not be greater than map_depth, so no additional comparison is required. Signed-off-by: Liu Song Link: https://lore.kernel.org/r/1661483653-27326-1-git-send-email-liusong@linux.alibaba.com Signed-off-by: Jens Axboe (cherry picked from commit ddbfc34fcf5d0bc33b006b90c580c56edeb31068) Signed-off-by: Gerald Yang --- lib/sbitmap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index 1f31147872e6..a39b1a877366 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -533,10 +533,9 @@ unsigned long __sbitmap_queue_get_batch(struct sbitmap_queue *sbq, int nr_tags, nr = find_first_zero_bit(&map->word, map_depth); if (nr + nr_tags <= map_depth) { atomic_long_t *ptr = (atomic_long_t *) &map->word; - int map_tags = min_t(int, nr_tags, map_depth); unsigned long val, ret; - get_mask = ((1UL << map_tags) - 1) << nr; + get_mask = ((1UL << nr_tags) - 1) << nr; do { val = READ_ONCE(map->word); if ((val & ~get_mask) != val) @@ -547,7 +546,7 @@ unsigned long __sbitmap_queue_get_batch(struct sbitmap_queue *sbq, int nr_tags, if (get_mask) { *offset = nr + (index << sb->shift); update_alloc_hint_after_get(sb, depth, hint, - *offset + map_tags - 1); + *offset + nr_tags - 1); return get_mask; } } From patchwork Tue Jun 6 10:28:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gerald Yang X-Patchwork-Id: 1791054 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=T8Xte2dX; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Qb6CF4dWFz20Wd for ; Tue, 6 Jun 2023 20:28:57 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1q6Tvl-0005xA-VT; Tue, 06 Jun 2023 10:28:50 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1q6Tvg-0005s8-LU for kernel-team@lists.ubuntu.com; Tue, 06 Jun 2023 10:28:44 +0000 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 6D7853F0CB for ; Tue, 6 Jun 2023 10:28:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1686047324; bh=vmJtpKhYUdAYezGaQzm5jogkHKsopiQMTe/e2rQ97s8=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T8Xte2dXkCuTgAHSOoQitlx1Fy/9Mb+jd7EkG+ODA2PVFpMzhPhoPNIBaOEaOrDjl yUOoMVKw+3264c1YLM2K7+f4heawc+l+ePdhfJjvNmBjE1Po/mMngSMEmCYKmkST2l tVbmPnarV5t0Hd31PPOv0lOOW09ZpjEejSZfCW7Jn1OxlvXLajbYMe3+zd7sZ7ZSrg 6CTxAt5X+gidqFbEkRegkhW5n5qe9hgrtQP8u0xL8hTG+fLjcHTRol6PpDedK5e68w T5ZgHEz41tBDItfqQT7aO9tbmrf9muPHkoTi9c82QMyXVjDVqHcb2KKGuzUFn2p/To BKVMdKaB2Qbdg== Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-2566f2acd88so5141178a91.1 for ; Tue, 06 Jun 2023 03:28:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686047323; x=1688639323; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vmJtpKhYUdAYezGaQzm5jogkHKsopiQMTe/e2rQ97s8=; b=Es0Ruwh3OZKmkqScWYYU3KZ21Wmx7oodH46LiiNUnK5X1bqWl/5SRAu/Mr6gFnyOtj jvCN9i3UPeanAH4ogGVeOaBTuNjHKrkKnqiWrOuvgLjMFpbK1Nwg0O1FHjeSaPhRNE7z uZH2KxvJWW0F23ljapAoLGawPZs9zIy4eg9ojVZyq7+dTpfA3T8FVJGiITG3aXNMxuVX /NlUeaAyKsW/CZn8l21VqnfoK4afxUjgY8bt3iRynx2T47Yl1nb43Q0jAaUbwaUe9nrK agzglf1YJ3uJu1qR6dfFv44bQNHVcO2Fc/8wHhju1oMTe8P9p0nfVuSefRVuwMZhz0S0 VFwg== X-Gm-Message-State: AC+VfDwYXQEdZZHlIQyx2J8ZePodgPwOy9R3vvagBtjT1Eky1HbKzvLR 5i6lw+N1fF5P4QDxP8zTgNJqgq2BrPQm2PpGoMp3y9gTzA4NaZB2kWcKL8sUlFRYVkXttvQiwkH c0xVwgAt4KtqV9YZViIgitjvXEFIftL6F6Agm/80dtrpk1NNBmg== X-Received: by 2002:a17:90a:f2cf:b0:259:45c2:7339 with SMTP id gt15-20020a17090af2cf00b0025945c27339mr1446499pjb.23.1686047322822; Tue, 06 Jun 2023 03:28:42 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5pH0fybCIkVum5CfAJVWOGzjDPTfbNqm+jUJFUI5iNFo2Lr7kc7G/SqIa9iMJ10E/qF+/lfQ== X-Received: by 2002:a17:90a:f2cf:b0:259:45c2:7339 with SMTP id gt15-20020a17090af2cf00b0025945c27339mr1446487pjb.23.1686047322494; Tue, 06 Jun 2023 03:28:42 -0700 (PDT) Received: from localhost.localdomain (220-135-31-21.hinet-ip.hinet.net. [220.135.31.21]) by smtp.gmail.com with ESMTPSA id 8-20020a17090a0c0800b00256a6ec8507sm9777629pjs.19.2023.06.06.03.28.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jun 2023 03:28:42 -0700 (PDT) From: Gerald Yang To: kernel-team@lists.ubuntu.com Subject: [SRU][K][PATCH 3/6] sbitmap: Avoid leaving waitqueue in invalid state in __sbq_wake_up() Date: Tue, 6 Jun 2023 18:28:25 +0800 Message-Id: <20230606102828.218014-4-gerald.yang@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230606102828.218014-1-gerald.yang@canonical.com> References: <20230606102828.218014-1-gerald.yang@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Jan Kara BugLink: https://bugs.launchpad.net/bugs/2022318 When __sbq_wake_up() decrements wait_cnt to 0 but races with someone else waking the waiter on the waitqueue (so the waitqueue becomes empty), it exits without reseting wait_cnt to wake_batch number. Once wait_cnt is 0, nobody will ever reset the wait_cnt or wake the new waiters resulting in possible deadlocks or busyloops. Fix the problem by making sure we reset wait_cnt even if we didn't wake up anybody in the end. Fixes: 040b83fcecfb ("sbitmap: fix possible io hung due to lost wakeup") Reported-by: Keith Busch Signed-off-by: Jan Kara Link: https://lore.kernel.org/r/20220908130937.2795-1-jack@suse.cz Signed-off-by: Jens Axboe (cherry picked from commit 48c033314f372478548203c583529f53080fd078) Signed-off-by: Gerald Yang --- lib/sbitmap.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index a39b1a877366..47cd8fb894ba 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -604,6 +604,7 @@ static bool __sbq_wake_up(struct sbitmap_queue *sbq) struct sbq_wait_state *ws; unsigned int wake_batch; int wait_cnt; + bool ret; ws = sbq_wake_ptr(sbq); if (!ws) @@ -614,12 +615,23 @@ static bool __sbq_wake_up(struct sbitmap_queue *sbq) * For concurrent callers of this, callers should call this function * again to wakeup a new batch on a different 'ws'. */ - if (wait_cnt < 0 || !waitqueue_active(&ws->wait)) + if (wait_cnt < 0) return true; + /* + * If we decremented queue without waiters, retry to avoid lost + * wakeups. + */ if (wait_cnt > 0) - return false; + return !waitqueue_active(&ws->wait); + /* + * When wait_cnt == 0, we have to be particularly careful as we are + * responsible to reset wait_cnt regardless whether we've actually + * woken up anybody. But in case we didn't wakeup anybody, we still + * need to retry. + */ + ret = !waitqueue_active(&ws->wait); wake_batch = READ_ONCE(sbq->wake_batch); /* @@ -648,7 +660,7 @@ static bool __sbq_wake_up(struct sbitmap_queue *sbq) sbq_index_atomic_inc(&sbq->wake_index); atomic_set(&ws->wait_cnt, wake_batch); - return false; + return ret; } void sbitmap_queue_wake_up(struct sbitmap_queue *sbq) From patchwork Tue Jun 6 10:28:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gerald Yang X-Patchwork-Id: 1791052 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=gn9lRCSa; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Qb6CB2mBnz20Ty for ; Tue, 6 Jun 2023 20:28:54 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1q6Tvj-0005uR-HI; Tue, 06 Jun 2023 10:28:47 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1q6Tvh-0005sW-BV for kernel-team@lists.ubuntu.com; Tue, 06 Jun 2023 10:28:45 +0000 Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 29FEA3F0CB for ; Tue, 6 Jun 2023 10:28:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1686047325; bh=VJ8QsgM/p+NQinqeHGdtKTJtpsBq8gLMWxMvIAZzaXk=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gn9lRCSaNMtubw6G2hgV8b+jiUsVU3t98lmTHlgK8PQjO8wSUDxPcnDP2e9qlC6dk Vq4+91NZQ8DB3YgaqDjKt2QjbMPJvAwbmC4zX9kAUVp6ZLmbvFnnBX4dMy89MRROIc 58jUMW2c1UgnfJnCToWLmUOSJM3ZbJMIoc5zx9rBetkwMbsgbHfFvnC5WkY8WL2c5A Jsvhpb7DVnu1edRwIkGShV+/kn4nojlGNpJEkn/QpgxXd7IpfekF2GEMnveB/jUop6 x03FZ1Ct8kIIF8H+9LPHyYuyWBsCF6O02cc109wvJL5DtsOoBMlDuS8DhzzQfOagKV MqH0LijVXb9Cw== Received: by mail-oo1-f72.google.com with SMTP id 006d021491bc7-5559caee9d3so3981671eaf.2 for ; Tue, 06 Jun 2023 03:28:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686047324; x=1688639324; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VJ8QsgM/p+NQinqeHGdtKTJtpsBq8gLMWxMvIAZzaXk=; b=BbJbL0OaaeFQsV2oIgACF10b9O8RDOMgR9CFVmbJAcoa1YAB9SThf+tCSA5u+E0T6N badrcHiKxmbeb6yK0fCL5WO/iWr2BJhr/LljxohPKPMv6B5YBbqU2RAO0wkvjauGDftm VgrCblIqjI8FXBhEhaoNPTJYTiubw8BVDbOGNxf5N2f9Vr/SzGI9Hz5PATG2bhXdwKBQ +IeLpEqiGXud5xpLM55DEwCfUQQg2Xlr0xw6IfzvMb1ytazmK76f+FG37KqpfWiP+3NR 51ShS9rx2TS4urs0ZqQbBkZQB5z4CpB4h19FZf5vewRNlfbYo3cq6Yzbiq1+IpshIPKn hVog== X-Gm-Message-State: AC+VfDy0EVhNC1qAYa8M4JST8Wdpx0fnYUZOEzJxv3Uh8ysW4FkvK440 E9pcnuhSGKrf2oWTOvXYN1mbyyoo82Na9HWEJfTfSzidPO09Om5pOJooQSQ1m2s3J/Eic3ACy/a ZOXiFuvzlpZS6jTn5oO4IEm4QbCO+hM3AdZJq9LnM72UT8oeLrw== X-Received: by 2002:aca:210d:0:b0:398:6008:f460 with SMTP id 13-20020aca210d000000b003986008f460mr1885656oiz.28.1686047323882; Tue, 06 Jun 2023 03:28:43 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5ePxbrk12LV64GbsKRWRd8W5vm8VWu2aCxsIFfNV03TRfQRDANlNGCs/k0GzJMkPbMQ4qkTA== X-Received: by 2002:aca:210d:0:b0:398:6008:f460 with SMTP id 13-20020aca210d000000b003986008f460mr1885637oiz.28.1686047323564; Tue, 06 Jun 2023 03:28:43 -0700 (PDT) Received: from localhost.localdomain (220-135-31-21.hinet-ip.hinet.net. [220.135.31.21]) by smtp.gmail.com with ESMTPSA id 8-20020a17090a0c0800b00256a6ec8507sm9777629pjs.19.2023.06.06.03.28.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jun 2023 03:28:43 -0700 (PDT) From: Gerald Yang To: kernel-team@lists.ubuntu.com Subject: [SRU][K][PATCH 4/6] sbitmap: Use atomic_long_try_cmpxchg in __sbitmap_queue_get_batch Date: Tue, 6 Jun 2023 18:28:26 +0800 Message-Id: <20230606102828.218014-5-gerald.yang@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230606102828.218014-1-gerald.yang@canonical.com> References: <20230606102828.218014-1-gerald.yang@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Uros Bizjak BugLink: https://bugs.launchpad.net/bugs/2022318 Use atomic_long_try_cmpxchg instead of atomic_long_cmpxchg (*ptr, old, new) == old in __sbitmap_queue_get_batch. x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg). Also, atomic_long_cmpxchg implicitly assigns old *ptr value to "old" when cmpxchg fails, enabling further code simplifications, e.g. an extra memory read can be avoided in the loop. No functional change intended. Cc: Jens Axboe Signed-off-by: Uros Bizjak Link: https://lore.kernel.org/r/20220908151200.9993-1-ubizjak@gmail.com Signed-off-by: Jens Axboe (cherry picked from commit c35227d4e8cbc70a6622cc7cc5f8c3bff513f1fa) Signed-off-by: Gerald Yang --- lib/sbitmap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index 47cd8fb894ba..cbfd2e677d87 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -533,16 +533,16 @@ unsigned long __sbitmap_queue_get_batch(struct sbitmap_queue *sbq, int nr_tags, nr = find_first_zero_bit(&map->word, map_depth); if (nr + nr_tags <= map_depth) { atomic_long_t *ptr = (atomic_long_t *) &map->word; - unsigned long val, ret; + unsigned long val; get_mask = ((1UL << nr_tags) - 1) << nr; + val = READ_ONCE(map->word); do { - val = READ_ONCE(map->word); if ((val & ~get_mask) != val) goto next; - ret = atomic_long_cmpxchg(ptr, val, get_mask | val); - } while (ret != val); - get_mask = (get_mask & ~ret) >> nr; + } while (!atomic_long_try_cmpxchg(ptr, &val, + get_mask | val)); + get_mask = (get_mask & ~val) >> nr; if (get_mask) { *offset = nr + (index << sb->shift); update_alloc_hint_after_get(sb, depth, hint, From patchwork Tue Jun 6 10:28:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gerald Yang X-Patchwork-Id: 1791055 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=Hl4jjUuf; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Qb6CH36X8z20Ty for ; Tue, 6 Jun 2023 20:28:59 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1q6Tvo-00060a-8o; Tue, 06 Jun 2023 10:28:52 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1q6Tvh-0005sm-RJ for kernel-team@lists.ubuntu.com; Tue, 06 Jun 2023 10:28:45 +0000 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 94C3B3F0CB for ; Tue, 6 Jun 2023 10:28:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1686047325; bh=dtF0Nn2Mg4hlepgMWSTgw0mDqiPZ+L4XQJBCc2rOIuM=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Hl4jjUufdUozwjTy9NFVkYsTz33zPiQnv48vzYXfw8fjA2wsR37Ai/CzEP9LNKcug 6/Eee4bPldbwq7TriJc4s4a66zpdMY+PIYYKj/ugWYaXBmlbd/PHxjOZ+kG8tL02Df HpDZHw+TIuv1UM4/wPa6ct5Rn9iyv+zVBgxuoh3+vHmcQdwj2LbrtYp1MuZmylQ1KE godOqKBGeeVTBTg9KpQ+AlMpUcSoXbNfhYt5eF2W337lEbkxeE1HfpnK3khaLaNklS aSirsAQak2aCaFzen4hzKxt4jUqo2+v4+3ZZ17/DXqoMgJMRW2Gw4VI3fHCscXjg7B 9NUOnhCimhb+w== Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-2566f2acd88so5141225a91.1 for ; Tue, 06 Jun 2023 03:28:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686047325; x=1688639325; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dtF0Nn2Mg4hlepgMWSTgw0mDqiPZ+L4XQJBCc2rOIuM=; b=Oo95YKat0grM1H0eonMi3ns2c55NvYU/3BZfx/UThU5zEQc/GEKScUwQiffdBBHuNQ n3wWcXxYn/ok+We+gbUrJPoNWikob6dha2HZ8udEUTP4C2Ko+DCQlrQBzZ7lIkL3tC9q ikx05RQIuejGB6q0TNXCHZ7W+RKR5Xojp8ieH8zOw5bCHE8OrOkfl/SSMWQsqgSqtgba mXl3n35Hpm6sE+gwYp5C6AXNPYxZuxmu0JN2j1k/c9AaxPdnXP3PtLTwkNfHeXOuzDJo K15in0Cx5XshGqU0WlTCoN3UVgDrEKrRnV5noc4BBi9KrtDAAJLJnE+VNQYE5Jbs2ETz S13A== X-Gm-Message-State: AC+VfDxKPx644I9YZhAoBJQi8f0NbuxH0UBEobLEBzvmv5EaJgthXoAM 2gNFLGf/7DDyHDQWx4HkXgpqti43C2u5mvI2LEVM16mgy3Fisi3iVJnVINPrAgi2OE/N/LXNVPd oD098tTLO/H6zxiHwrN5jbxFvTuXx6BJM7RjAxreW9x6OjVdbEg== X-Received: by 2002:a17:90a:354:b0:258:9174:20ca with SMTP id 20-20020a17090a035400b00258917420camr1530586pjf.15.1686047324887; Tue, 06 Jun 2023 03:28:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5N1f5yTyrYSzJMtjDmDVYXXlnar9jzJlibxC2Ahz3dwwi3hrUDpQ2pEv3HCps04hw1qw7xew== X-Received: by 2002:a17:90a:354:b0:258:9174:20ca with SMTP id 20-20020a17090a035400b00258917420camr1530579pjf.15.1686047324552; Tue, 06 Jun 2023 03:28:44 -0700 (PDT) Received: from localhost.localdomain (220-135-31-21.hinet-ip.hinet.net. [220.135.31.21]) by smtp.gmail.com with ESMTPSA id 8-20020a17090a0c0800b00256a6ec8507sm9777629pjs.19.2023.06.06.03.28.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jun 2023 03:28:44 -0700 (PDT) From: Gerald Yang To: kernel-team@lists.ubuntu.com Subject: [SRU][K][PATCH 5/6] sbitmap: fix batched wait_cnt accounting Date: Tue, 6 Jun 2023 18:28:27 +0800 Message-Id: <20230606102828.218014-6-gerald.yang@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230606102828.218014-1-gerald.yang@canonical.com> References: <20230606102828.218014-1-gerald.yang@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Keith Busch BugLink: https://bugs.launchpad.net/bugs/2022318 Batched completions can clear multiple bits, but we're only decrementing the wait_cnt by one each time. This can cause waiters to never be woken, stalling IO. Use the batched count instead. Link: https://bugzilla.kernel.org/show_bug.cgi?id=215679 Signed-off-by: Keith Busch Link: https://lore.kernel.org/r/20220909184022.1709476-1-kbusch@fb.com Signed-off-by: Jens Axboe (cherry picked from commit 4acb83417cadfdcbe64215f9d0ddcf3132af808e) Signed-off-by: Gerald Yang --- block/blk-mq-tag.c | 2 +- include/linux/sbitmap.h | 3 ++- lib/sbitmap.c | 37 +++++++++++++++++++++++-------------- 3 files changed, 26 insertions(+), 16 deletions(-) diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 2dcd738c6952..7aea93047caf 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -200,7 +200,7 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) * other allocations on previous queue won't be starved. */ if (bt != bt_prev) - sbitmap_queue_wake_up(bt_prev); + sbitmap_queue_wake_up(bt_prev, 1); ws = bt_wait_ptr(bt, data->hctx); } while (1); diff --git a/include/linux/sbitmap.h b/include/linux/sbitmap.h index 8f5a86e210b9..4d2d5205ab58 100644 --- a/include/linux/sbitmap.h +++ b/include/linux/sbitmap.h @@ -575,8 +575,9 @@ void sbitmap_queue_wake_all(struct sbitmap_queue *sbq); * sbitmap_queue_wake_up() - Wake up some of waiters in one waitqueue * on a &struct sbitmap_queue. * @sbq: Bitmap queue to wake up. + * @nr: Number of bits cleared. */ -void sbitmap_queue_wake_up(struct sbitmap_queue *sbq); +void sbitmap_queue_wake_up(struct sbitmap_queue *sbq, int nr); /** * sbitmap_queue_show() - Dump &struct sbitmap_queue information to a &struct diff --git a/lib/sbitmap.c b/lib/sbitmap.c index cbfd2e677d87..624fa7f118d1 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -599,24 +599,31 @@ static struct sbq_wait_state *sbq_wake_ptr(struct sbitmap_queue *sbq) return NULL; } -static bool __sbq_wake_up(struct sbitmap_queue *sbq) +static bool __sbq_wake_up(struct sbitmap_queue *sbq, int *nr) { struct sbq_wait_state *ws; unsigned int wake_batch; - int wait_cnt; + int wait_cnt, cur, sub; bool ret; + if (*nr <= 0) + return false; + ws = sbq_wake_ptr(sbq); if (!ws) return false; - wait_cnt = atomic_dec_return(&ws->wait_cnt); - /* - * For concurrent callers of this, callers should call this function - * again to wakeup a new batch on a different 'ws'. - */ - if (wait_cnt < 0) - return true; + cur = atomic_read(&ws->wait_cnt); + do { + /* + * For concurrent callers of this, callers should call this + * function again to wakeup a new batch on a different 'ws'. + */ + if (cur == 0) + return true; + sub = min(*nr, cur); + wait_cnt = cur - sub; + } while (!atomic_try_cmpxchg(&ws->wait_cnt, &cur, wait_cnt)); /* * If we decremented queue without waiters, retry to avoid lost @@ -625,6 +632,8 @@ static bool __sbq_wake_up(struct sbitmap_queue *sbq) if (wait_cnt > 0) return !waitqueue_active(&ws->wait); + *nr -= sub; + /* * When wait_cnt == 0, we have to be particularly careful as we are * responsible to reset wait_cnt regardless whether we've actually @@ -660,12 +669,12 @@ static bool __sbq_wake_up(struct sbitmap_queue *sbq) sbq_index_atomic_inc(&sbq->wake_index); atomic_set(&ws->wait_cnt, wake_batch); - return ret; + return ret || *nr; } -void sbitmap_queue_wake_up(struct sbitmap_queue *sbq) +void sbitmap_queue_wake_up(struct sbitmap_queue *sbq, int nr) { - while (__sbq_wake_up(sbq)) + while (__sbq_wake_up(sbq, &nr)) ; } EXPORT_SYMBOL_GPL(sbitmap_queue_wake_up); @@ -705,7 +714,7 @@ void sbitmap_queue_clear_batch(struct sbitmap_queue *sbq, int offset, atomic_long_andnot(mask, (atomic_long_t *) addr); smp_mb__after_atomic(); - sbitmap_queue_wake_up(sbq); + sbitmap_queue_wake_up(sbq, nr_tags); sbitmap_update_cpu_hint(&sbq->sb, raw_smp_processor_id(), tags[nr_tags - 1] - offset); } @@ -733,7 +742,7 @@ void sbitmap_queue_clear(struct sbitmap_queue *sbq, unsigned int nr, * waiter. See the comment on waitqueue_active(). */ smp_mb__after_atomic(); - sbitmap_queue_wake_up(sbq); + sbitmap_queue_wake_up(sbq, 1); sbitmap_update_cpu_hint(&sbq->sb, cpu, nr); } EXPORT_SYMBOL_GPL(sbitmap_queue_clear); From patchwork Tue Jun 6 10:28:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gerald Yang X-Patchwork-Id: 1791056 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=n4RsLrAN; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Qb6CJ1ssYz20Wd for ; Tue, 6 Jun 2023 20:29:00 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1q6Tvp-00062q-Dp; Tue, 06 Jun 2023 10:28:53 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1q6Tvj-0005ue-M9 for kernel-team@lists.ubuntu.com; Tue, 06 Jun 2023 10:28:47 +0000 Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 69EB23F117 for ; Tue, 6 Jun 2023 10:28:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1686047327; bh=RTzBHpf7pNxeVKUNSCI3X7DgPVjuRy939w+zZ4Zr6Fg=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=n4RsLrANk+s51Hp/VN+bMfCRp2cuX2kgCAl+43vrlOCipNR+A0duCxF5MCuUMl4yd ciZeR51l77/NZf4SMPf+opq6YA9YIP1YOmDWGLD9LoHTBEeZtt7AtudBZk1RIi+Qdr IUOHGrqF3tdV3hs7EsyaBHbc+crzwoBe6Qfs40+iY+q1O50+jZxsSWw5WxjWg9ArVn p3O1aykRwvuCbv41MKQ6/TQbIxDXI9B88mqtUL4jqM78fAgsLc063X2q7RjlRRLGcs iBjBduWy5QphdkP0NxffDcnb5fekw2hwnOtR0Nj3kocmdDL61HjbTj7owIBNB+opqh 1ur/tb29hFhTQ== Received: by mail-oi1-f198.google.com with SMTP id 5614622812f47-395fd55e523so3060709b6e.2 for ; Tue, 06 Jun 2023 03:28:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686047326; x=1688639326; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RTzBHpf7pNxeVKUNSCI3X7DgPVjuRy939w+zZ4Zr6Fg=; b=AqNAcW1OXcCcHuDb1HJOCefPnpgM0tetVX4RvBTP/E/IkAndfPlJfwJJbf/V1KLJUS 4vx5y8JoNqSRH/2fBYsFiIb5LAVPWsyxZVYg13z/eHMk4FCNiIGX1hX85ukVxHlEF2mS hiTmYO8uLmCf19wNMznQfFh5vq8hRoxYNH5aldnLEb5LTyWmQCr699s5lIVGVHWIwO48 Ivm4NiarTUuSdFj1GRYsmdebbesG6d5EqHdzdIdZYsmEf4OaKwW007hkWRDDDXSCL9MW SchrsE9jKX5zNA0eDPe8c6d5B3uX1AiYKhw86cRv2aN0mdOnk8FznMyTEzkxaws8VgAE fznQ== X-Gm-Message-State: AC+VfDyNuQQoXkxFw4o2f68/JOW+BlHgE7FywEW300BckyojGjxWlf99 o63+nezDnRAZyg0NsOfSjp7IBLUBZK6+vGStdEvx8fq0Z/i8tMjziYYRB5X+BudK3U+dZO6ghQG M2HhqNkQ41gThYCXCGzidNMWAHQk1z8wknqEUKYXZvDO1fROwQw== X-Received: by 2002:a05:6359:20:b0:129:c8e2:e7eb with SMTP id en32-20020a056359002000b00129c8e2e7ebmr1431723rwb.27.1686047325965; Tue, 06 Jun 2023 03:28:45 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ67h38yUbo9EFLH10OasTdEk0P6VeUez7VpqgAgEfJYNPinuDAidFqO5tQkTRj3YGkXfS8svw== X-Received: by 2002:a05:6359:20:b0:129:c8e2:e7eb with SMTP id en32-20020a056359002000b00129c8e2e7ebmr1431720rwb.27.1686047325580; Tue, 06 Jun 2023 03:28:45 -0700 (PDT) Received: from localhost.localdomain (220-135-31-21.hinet-ip.hinet.net. [220.135.31.21]) by smtp.gmail.com with ESMTPSA id 8-20020a17090a0c0800b00256a6ec8507sm9777629pjs.19.2023.06.06.03.28.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jun 2023 03:28:45 -0700 (PDT) From: Gerald Yang To: kernel-team@lists.ubuntu.com Subject: [SRU][K][PATCH 6/6] sbitmap: fix lockup while swapping Date: Tue, 6 Jun 2023 18:28:28 +0800 Message-Id: <20230606102828.218014-7-gerald.yang@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230606102828.218014-1-gerald.yang@canonical.com> References: <20230606102828.218014-1-gerald.yang@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Hugh Dickins BugLink: https://bugs.launchpad.net/bugs/2022318 Commit 4acb83417cad ("sbitmap: fix batched wait_cnt accounting") is a big improvement: without it, I had to revert to before commit 040b83fcecfb ("sbitmap: fix possible io hung due to lost wakeup") to avoid the high system time and freezes which that had introduced. Now okay on the NVME laptop, but 4acb83417cad is a disaster for heavy swapping (kernel builds in low memory) on another: soon locking up in sbitmap_queue_wake_up() (into which __sbq_wake_up() is inlined), cycling around with waitqueue_active() but wait_cnt 0 . Here is a backtrace, showing the common pattern of outer sbitmap_queue_wake_up() interrupted before setting wait_cnt 0 back to wake_batch (in some cases other CPUs are idle, in other cases they're spinning for a lock in dd_bio_merge()): sbitmap_queue_wake_up < sbitmap_queue_clear < blk_mq_put_tag < __blk_mq_free_request < blk_mq_free_request < __blk_mq_end_request < scsi_end_request < scsi_io_completion < scsi_finish_command < scsi_complete < blk_complete_reqs < blk_done_softirq < __do_softirq < __irq_exit_rcu < irq_exit_rcu < common_interrupt < asm_common_interrupt < _raw_spin_unlock_irqrestore < __wake_up_common_lock < __wake_up < sbitmap_queue_wake_up < sbitmap_queue_clear < blk_mq_put_tag < __blk_mq_free_request < blk_mq_free_request < dd_bio_merge < blk_mq_sched_bio_merge < blk_mq_attempt_bio_merge < blk_mq_submit_bio < __submit_bio < submit_bio_noacct_nocheck < submit_bio_noacct < submit_bio < __swap_writepage < swap_writepage < pageout < shrink_folio_list < evict_folios < lru_gen_shrink_lruvec < shrink_lruvec < shrink_node < do_try_to_free_pages < try_to_free_pages < __alloc_pages_slowpath < __alloc_pages < folio_alloc < vma_alloc_folio < do_anonymous_page < __handle_mm_fault < handle_mm_fault < do_user_addr_fault < exc_page_fault < asm_exc_page_fault See how the process-context sbitmap_queue_wake_up() has been interrupted, after bringing wait_cnt down to 0 (and in this example, after doing its wakeups), before advancing wake_index and refilling wake_cnt: an interrupt-context sbitmap_queue_wake_up() of the same sbq gets stuck. I have almost no grasp of all the possible sbitmap races, and their consequences: but __sbq_wake_up() can do nothing useful while wait_cnt 0, so it is better if sbq_wake_ptr() skips on to the next ws in that case: which fixes the lockup and shows no adverse consequence for me. The check for wait_cnt being 0 is obviously racy, and ultimately can lead to lost wakeups: for example, when there is only a single waitqueue with waiters. However, lost wakeups are unlikely to matter in these cases, and a proper fix requires redesign (and benchmarking) of the batched wakeup code: so let's plug the hole with this bandaid for now. Signed-off-by: Hugh Dickins Reviewed-by: Jan Kara Reviewed-by: Keith Busch Link: https://lore.kernel.org/r/9c2038a7-cdc5-5ee-854c-fbc6168bf16@google.com Signed-off-by: Jens Axboe (cherry picked from commit 30514bd2dd4e86a3ecfd6a93a3eadf7b9ea164a0) Signed-off-by: Gerald Yang --- lib/sbitmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index 624fa7f118d1..a8108a962dfd 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -587,7 +587,7 @@ static struct sbq_wait_state *sbq_wake_ptr(struct sbitmap_queue *sbq) for (i = 0; i < SBQ_WAIT_QUEUES; i++) { struct sbq_wait_state *ws = &sbq->ws[wake_index]; - if (waitqueue_active(&ws->wait)) { + if (waitqueue_active(&ws->wait) && atomic_read(&ws->wait_cnt)) { if (wake_index != atomic_read(&sbq->wake_index)) atomic_set(&sbq->wake_index, wake_index); return ws;