From patchwork Fri Sep 23 16:29:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng He X-Patchwork-Id: 1681767 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jdpHxJzs; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MZH9Y4sc0z1yqH for ; Sat, 24 Sep 2022 15:08:49 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id E2E1E41B61; Sat, 24 Sep 2022 05:08:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org E2E1E41B61 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jdpHxJzs X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fi_eVmCwa7xi; Sat, 24 Sep 2022 05:08:43 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 7F69041B4E; Sat, 24 Sep 2022 05:08:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 7F69041B4E Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7168CC007C; Sat, 24 Sep 2022 05:08:41 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 820C2C007C for ; Sat, 24 Sep 2022 05:08:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 4FD5F813F0 for ; Sat, 24 Sep 2022 05:08:40 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 4FD5F813F0 Authentication-Results: smtp1.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jdpHxJzs X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RbTPwChfbQTJ for ; Sat, 24 Sep 2022 05:08:39 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org A11F7813DE Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by smtp1.osuosl.org (Postfix) with ESMTPS id A11F7813DE for ; Sat, 24 Sep 2022 05:08:39 +0000 (UTC) Received: by mail-pf1-x431.google.com with SMTP id d10so800903pfh.6 for ; Fri, 23 Sep 2022 22:08:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=kE+JR+ryu8ZWLLha1+tEucqyeyXP1OXhjCN/2FCX3No=; b=jdpHxJzsiuevUpuU8emqElyJn1aqEsDHfhCmnG2sgsla4w4bj3bnRvS7tprcStH1g7 Ih5yf5GW6/a2ZJZ6N4DkQh25W6sBJeIU8gwHExAF52/xlvnSACcYcHCfst3RiwdsJqi3 knAXL4hKluoT5yD1nb+uB34rO5bSvW7CD/kKacVlrfxE+Rxi6SFSipqzqClzhGmB2VYM e+MAs9pufs8OB6XG4dOEhrRkoEZy6HhzeAQ40nssWQ/VRcD5tZLKELlw6UFZ7rvp/mxF Qo7RDG8FQKxTEO1dMaOiCQgnxbPnKNsDH7rnVSogtZT41iLbkPMh9t9GhX3mPiqC/w4e KBfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=kE+JR+ryu8ZWLLha1+tEucqyeyXP1OXhjCN/2FCX3No=; b=g8P1eZihYQ7qQZZUEGPrJ1I8eGwG+EDrpyht5/G2HrVdcJj00m1LarpoMyR/Cy2xk7 g5Srk3KSn3a2Aq9D4EQ5sZ/rorNTj6Ameyh9xyhsOPs2qVSsJW88s4BxlleC9Yrpel5P uNqevDtN9wJeQy3p+qNnSsVkUb/nkEHcUdb2O9PNInTsOdGi/JUxvG5RJkGEWdReDimJ Fw0lFJ7QC8/u9FWu/wmDfbD87sFyLyDAjpB4L+NqE494JxnCXEKT8wYD3EpbQH8n2IdC ftniIyQv/o30DivJD/1WMrc3rGhko8jeha7lnTWYTO8AXg2dgBkpZLo4v2TWf/UV8bcu U98g== X-Gm-Message-State: ACrzQf3e1zbmRJxCiqNlXUXEH+dpdk+wVmv8rIm83lGv0gnsjYovCl3n codFaDYW7okgxBDJhqKR87sFXBdahfv69EOC X-Google-Smtp-Source: AMsMyM4Brxyi0SuPVlU4E4QG9TdFs/hg+SaWqCDYGZKztkhu0wDBrlYyeSin/RTNQ0Glbi809sVLaw== X-Received: by 2002:a05:6a00:a82:b0:547:d660:c077 with SMTP id b2-20020a056a000a8200b00547d660c077mr12474761pfl.38.1663996118783; Fri, 23 Sep 2022 22:08:38 -0700 (PDT) Received: from localhost ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w24-20020a17090a461800b001fe39bda429sm2448211pjg.38.2022.09.23.22.08.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Sep 2022 22:08:38 -0700 (PDT) From: Peng He X-Google-Original-From: Peng He To: dev@openvswitch.org Date: Fri, 23 Sep 2022 16:29:18 +0000 Message-Id: <20220923162920.612492-2-hepeng.0320@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220923162920.612492-1-hepeng.0320@bytedance.com> References: <20220923162920.612492-1-hepeng.0320@bytedance.com> MIME-Version: 1.0 Subject: [ovs-dev] [ovs-dev v3 2/4] ofproto-dpif-upcall: fix race condition X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" There is a race condition between the revalidator threads and the handler/pmd threads. revalidator PMD threads push_dp_ops deletes a key and tries to del the dp magaflow. does the upcall, generates a new ukey, and replaces the old ukey, now the old ukey state is UKEY_DELETED dp_ops succeeds, tries to change the old ukey's state into UKEY_EVICTED, however, the old ukey's state is already UKEY_DELETED, so OVS aborts. I did not observe this in the real environment, as it takes time for PMDs to finish the upcall and replace the old ukeys. Normally, the revalidator will change ukey state into UKEY_EVICTED first. But it's better to cover this case. Signed-off-by: Peng He --- ofproto/ofproto-dpif-upcall.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 7ea2a30f5..e8bbcfeaf 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -2420,7 +2420,9 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops, size_t n_ops) if (op->dop.error) { if (op->ukey) { ovs_mutex_lock(&op->ukey->mutex); - transition_ukey(op->ukey, UKEY_EVICTED); + if (op->ukey->state < UKEY_EVICTED) { + transition_ukey(op->ukey, UKEY_EVICTED); + } ovs_mutex_unlock(&op->ukey->mutex); } /* if it's a flow_del error, 'stats' is unusable, it's ok @@ -2441,7 +2443,9 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops, size_t n_ops) if (op->ukey) { ovs_mutex_lock(&op->ukey->mutex); - transition_ukey(op->ukey, UKEY_EVICTED); + if (op->ukey->state < UKEY_EVICTED) { + transition_ukey(op->ukey, UKEY_EVICTED); + } push->used = MAX(stats->used, op->ukey->stats.used); push->tcp_flags = stats->tcp_flags | op->ukey->stats.tcp_flags; push->n_packets = stats->n_packets - op->ukey->stats.n_packets;