From patchwork Fri Mar 15 07:20:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: LIU Yulong X-Patchwork-Id: 1912370 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TwwdQ68rKz1yWn for ; Fri, 15 Mar 2024 18:20:41 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C34CB820E7; Fri, 15 Mar 2024 07:20:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1gMjFL4wL0Xh; Fri, 15 Mar 2024 07:20:37 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org BBE31813AA Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id BBE31813AA; Fri, 15 Mar 2024 07:20:37 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9A7D7C0077; Fri, 15 Mar 2024 07:20:37 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6EA46C0037 for ; Fri, 15 Mar 2024 07:20:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 50821606E7 for ; Fri, 15 Mar 2024 07:20:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Wp0azAY5Z1w3 for ; Fri, 15 Mar 2024 07:20:34 +0000 (UTC) X-Greylist: delayed 105002 seconds by postgrey-1.37 at util1.osuosl.org; Fri, 15 Mar 2024 07:20:32 UTC DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org F0C11606DA Authentication-Results: smtp3.osuosl.org; dmarc=none (p=none dis=none) header.from=liuyulong.me DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org F0C11606DA Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=43.155.67.158; helo=bg4.exmail.qq.com; envelope-from=i@liuyulong.me; receiver= Received: from bg4.exmail.qq.com (bg4.exmail.qq.com [43.155.67.158]) by smtp3.osuosl.org (Postfix) with ESMTPS id F0C11606DA for ; Fri, 15 Mar 2024 07:20:31 +0000 (UTC) X-QQ-mid: bizesmtp90t1710487224tuwlk65j X-QQ-Originating-IP: jbdrBS0YfY6gGHno3SsJQPC3YJ+cEX7igkbwtubwZ4c= Received: from localhost.localdomain ( [111.202.93.194]) by bizesmtp.qq.com (ESMTP) with id ; Fri, 15 Mar 2024 15:20:23 +0800 (CST) X-QQ-SSF: 01100000000000D0Z000000A0000000 X-QQ-FEAT: 7YFKcddXagiVlvPlL8zc+ly8PLUooVBeTNsDvkj5I6Zd2iyVb4VoZ1skby0bh AgAoYmQigqzl9VA22a1x1HB3gIy2hfEa/ahDqWZeTw9labwBlkrQmyEec06G4DI7gjAZpmp 8I3vjDoftgIA1VqcTRPj8k7BfIn1IFvFFKD+7/l8aWsO2SMl9dy80boS2xs1YRxJhamTOZ8 oNnaA7x17pPEwDUxx7H0yURMvN7923SiIKhXnOGD1wuB/PM+muoY+dKZS6VEI/H6LWsXdPk xtydGeESxatBksqK51qbJ0VlTWkqIeIGgyoDhFfxw+i5dgMS2CDUbqX40cmY35n9PG/N1gj UBHRBp/uEbjkNkAmdw= X-QQ-GoodBg: 0 X-BIZMAIL-ID: 13834258763585879295 From: LIU Yulong To: ovs-dev@openvswitch.org Date: Fri, 15 Mar 2024 15:20:29 +0800 Message-Id: <20240315072029.1974541-1-i@liuyulong.me> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:liuyulong.me:qybglogicsvrgz:qybglogicsvrgz5a-3 Subject: [ovs-dev] [PATCH] ofproto-dpif-upcall: Try lock for umap iteration during sweep. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" A potential race condition happened with the following 3 threads: * PMD thread replaced the old_ukey and transitioned the state to UKEY_DELETED. * RCU thread is freeing the old_ukey mutex. * While the revalidator thread is trying to lock the old_ukey mutex. Then vswitchd process aborts at the revalidator thread try_lock of ukey->mutex because of the NULL pointer. This patch adds the try_lock for the ukeys' basket umap to avoid the PMD and revalidator access to the same umap for replacing the ukey and transitioning the ukey state. More details can be found at: [1] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-March/052973.html [2] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-February/052949.html Signed-off-by: LIU Yulong --- ofproto/ofproto-dpif-upcall.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 9a5c5c29c..ef13f820a 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -2974,6 +2974,10 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) struct umap *umap = &udpif->ukeys[i]; size_t n_ops = 0; + if (ovs_mutex_trylock(&umap->mutex)) { + continue; + } + CMAP_FOR_EACH(ukey, cmap_node, &umap->cmap) { enum ukey_state ukey_state; @@ -3013,9 +3017,7 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) if (ukey_state == UKEY_EVICTED) { /* The common flow deletion case involves deletion of the flow * during the dump phase and ukey deletion here. */ - ovs_mutex_lock(&umap->mutex); ukey_delete(umap, ukey); - ovs_mutex_unlock(&umap->mutex); } if (n_ops == REVALIDATE_MAX_BATCH) { @@ -3025,6 +3027,7 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) n_ops = 0; } } + ovs_mutex_unlock(&umap->mutex); if (n_ops) { push_ukey_ops(udpif, umap, ops, n_ops);