From patchwork Thu Mar 14 02:10:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: LIU Yulong X-Patchwork-Id: 1911954 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Tw9pG66SGz1yWn for ; Thu, 14 Mar 2024 13:10:45 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 177D0416A3; Thu, 14 Mar 2024 02:10:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id celgnSpAlZxj; Thu, 14 Mar 2024 02:10:41 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 239FF416A2 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 239FF416A2; Thu, 14 Mar 2024 02:10:41 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id EF9EBC0077; Thu, 14 Mar 2024 02:10:40 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id AE696C0037 for ; Thu, 14 Mar 2024 02:10:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 9D40840184 for ; Thu, 14 Mar 2024 02:10:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AkY2mEPHnud1 for ; Thu, 14 Mar 2024 02:10:38 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=43.155.65.254; helo=bg4.exmail.qq.com; envelope-from=i@liuyulong.me; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org E0CD7416A2 Authentication-Results: smtp2.osuosl.org; dmarc=none (p=none dis=none) header.from=liuyulong.me DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org E0CD7416A2 Received: from bg4.exmail.qq.com (bg4.exmail.qq.com [43.155.65.254]) by smtp2.osuosl.org (Postfix) with ESMTPS id E0CD7416A2 for ; Thu, 14 Mar 2024 02:10:37 +0000 (UTC) X-QQ-mid: bizesmtp87t1710382210tgnk9esp X-QQ-Originating-IP: IdBTLbjphlgeCYRFacPIsyT+Zymd/GnweO7zh4baqcg= Received: from localhost.localdomain ( [111.202.93.194]) by bizesmtp.qq.com (ESMTP) with id ; Thu, 14 Mar 2024 10:10:09 +0800 (CST) X-QQ-SSF: 01100000000000D0Z000000A0000000 X-QQ-FEAT: URB33ikl3VfFs3epuEJmQ4lyckqoLQF4lf8EU/VZJPh0rsfnaxVaBrZuFjSK+ +3n8e2E93WSTSoCnj+gHKc9+NVj+djv5ZQpkZg3lV1HP20kslIWCHMrqTHzMV9cI4+y0tmE jz7BOjHQ/HqdUtZBVyjDmQqsWrtnRm4qdOi8av8dWYXoe6fA9iM/vPmG81tqx7LrUrb7rDk Y5iZEIbH1pjOz+44a85ASkI6rMxly7GywgfNYHnoVaNNXejoJxgKlWfu9xBgrDp8ytq4biW uNr+gmdSCx7q06TNWdR1wfKrLhoXxzzzu+qVUu3JAgwnZPBp3cy1UuDpGfGEoB1mEOYvnT7 QD27IpAxLuRFQB1bEVsoiSYCag+nU7ZyTCdjr2/ X-QQ-GoodBg: 0 X-BIZMAIL-ID: 2653977833192090745 From: LIU Yulong To: ovs-dev@openvswitch.org Date: Thu, 14 Mar 2024 10:10:10 +0800 Message-Id: <20240314021010.1923729-1-i@liuyulong.me> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:liuyulong.me:qybglogicsvrgz:qybglogicsvrgz5a-3 Subject: [ovs-dev] [PATCH] ofproto-dpif-upcall: try lock for umap iteration during sweep X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" A potential race condition happened with the following 3 threads: * PMD thread replaced the old_ukey and transitioned the state to UKEY_DELETED. * RCU thread is freeing the old_ukey mutex. * While the revalidator thread is trying to lock the old_ukey mutex. Then vswitchd process aborts at the revalidator thread try_lock of ukey->mutex because of the NULL pointer. This patch adds the try_lock for the ukeys' basket umap to avoid the PMD and revalidator access to the same umap for replacing the ukey and transitioning the ukey state. More details can be found at: [1] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-March/052973.html [2] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-February/052949.html Signed-off-by: LIU Yulong --- ofproto/ofproto-dpif-upcall.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 9a5c5c29c..ef13f820a 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -2974,6 +2974,10 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) struct umap *umap = &udpif->ukeys[i]; size_t n_ops = 0; + if (ovs_mutex_trylock(&umap->mutex)) { + continue; + } + CMAP_FOR_EACH(ukey, cmap_node, &umap->cmap) { enum ukey_state ukey_state; @@ -3013,9 +3017,7 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) if (ukey_state == UKEY_EVICTED) { /* The common flow deletion case involves deletion of the flow * during the dump phase and ukey deletion here. */ - ovs_mutex_lock(&umap->mutex); ukey_delete(umap, ukey); - ovs_mutex_unlock(&umap->mutex); } if (n_ops == REVALIDATE_MAX_BATCH) { @@ -3025,6 +3027,7 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) n_ops = 0; } } + ovs_mutex_unlock(&umap->mutex); if (n_ops) { push_ukey_ops(udpif, umap, ops, n_ops);