From patchwork Fri Sep 23 16:29:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng He X-Patchwork-Id: 1681766 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Ny2ZSSSa; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MZH9V3HWkz1yqH for ; Sat, 24 Sep 2022 15:08:44 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 07CD741B3E; Sat, 24 Sep 2022 05:08:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 07CD741B3E Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Ny2ZSSSa X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZTmUXFCzLSpm; Sat, 24 Sep 2022 05:08:40 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id CFA72408AC; Sat, 24 Sep 2022 05:08:39 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org CFA72408AC Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A088CC0033; Sat, 24 Sep 2022 05:08:39 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4DC80C002D for ; Sat, 24 Sep 2022 05:08:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2F43F40185 for ; Sat, 24 Sep 2022 05:08:38 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 2F43F40185 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Ny2ZSSSa X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lB1osVACbPmE for ; Sat, 24 Sep 2022 05:08:36 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 51A764015E Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by smtp2.osuosl.org (Postfix) with ESMTPS id 51A764015E for ; Sat, 24 Sep 2022 05:08:36 +0000 (UTC) Received: by mail-pl1-x62e.google.com with SMTP id jm5so1826518plb.13 for ; Fri, 23 Sep 2022 22:08:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date; bh=X7H2oeMvBYt5K7RT6MskWGzUuWv25VBYorg4+HmMZtU=; b=Ny2ZSSSan32cAIJTyuCfp9u8FbU1VDPnqmW8m8CBZ7NRlQeeS/6KXIJSdqcm6z5NnR 0gLu/8zEpXjhlN/787VJbD++UCxktkb1m3Rf/CEc0zmjQpQRtgUnjrCwwLZqGfQiGpRx u1Eh8MutWze7o23OfSdNvqa+kxDIYbOWxGJtgsQxlNo6jN7ItAQTMLMq4dNDD2xtbhdx lYdPAvgUrqSDdcNTNO1wUG6I7bSAinIe4lFT74AzjY1r6zGROSM9WEe4CtiF/1bgNXN+ 4ns3FpyCDZxsz686Bv49VxElijo1Q91WYPsjNPszSFUez9AfLh0OU9OHzDbEsIIzl3va 7jaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date; bh=X7H2oeMvBYt5K7RT6MskWGzUuWv25VBYorg4+HmMZtU=; b=sg+M3KScMIqi4R9Hy5KUX7lwNG8KhZTi2S/zBQ4qCEEow+AAzm0b7yvKTS/RWFEQFL 0lc8ALeqkNO+KQqMwjyQW1Nw9UQ/2Y+CPjt5C66zzN39JKGj7uYxqMM4NvO4B+EybNl7 EXISDM3E6ANBQxkpVdIvo1RnzH287h4wtgQLHs7e3JHsXU6vWfpYJ3WFS7iK+MKzq1CG kAieqVoJp4fYSUoYHKL+hcAfKFmab4wZwFmFI6KJ/BJkPU/pC3vOKfURBE/O6Sej57nz maFH8Bms7WV2IpKDRSviJpm1MvQSf2Q3SKSDELrgVk/wkZiXNrake3ihblCsBv7SsBCh AiGA== X-Gm-Message-State: ACrzQf1d8magQ01tOkWVLnwrqgjnZMLNi2EMxpJjJxVTcZ9X4hANYcWc wh7GkfCn50ZOBM4ayAFoFgTSu1UsLXhnvXY/ X-Google-Smtp-Source: AMsMyM4OlFtuZ3XmqeDotJoCFVOgciv4fnVj1kufCWHPwu5KVibA4VPmBx9dgZetILOuMBtBZdJcVg== X-Received: by 2002:a17:90b:4b04:b0:203:9c4e:7cc1 with SMTP id lx4-20020a17090b4b0400b002039c4e7cc1mr24449059pjb.16.1663996115421; Fri, 23 Sep 2022 22:08:35 -0700 (PDT) Received: from localhost ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id q17-20020a17090a2e1100b001fb1de10a4dsm2427323pjd.33.2022.09.23.22.08.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Sep 2022 22:08:34 -0700 (PDT) From: Peng He X-Google-Original-From: Peng He To: dev@openvswitch.org Date: Fri, 23 Sep 2022 16:29:17 +0000 Message-Id: <20220923162920.612492-1-hepeng.0320@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Subject: [ovs-dev] [ovs-dev v3 1/4] ofproto-dpif-upcall: fix push_dp_ops X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" push_dp_ops only handles delete ops errors but ignores the modify ops results. It's better to handle all the dp operation errors in a consistent way. We observe in the production environment that sometimes a megaflow with wrong actions keep staying in datapath. The coverage command shows revalidators have dumped several times, however the correct actions are not set. This implies that the ukey's action does not equal to the meagaflow's, i.e. revalidators think the underlying megaflow's actions are correct however they are not. We also check the megaflow using the ofproto/trace command, and the actions are not matched with the ones in the actual magaflow. By performing a revalidator/purge command, the right actions are set. This patch prevents the inconsistency by considering modify failure in revalidators. Signed-off-by: Peng He --- ofproto/ofproto-dpif-upcall.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 7ad728adf..7ea2a30f5 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -2416,23 +2416,26 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops, size_t n_ops) for (i = 0; i < n_ops; i++) { struct ukey_op *op = &ops[i]; - struct dpif_flow_stats *push, *stats, push_buf; - - stats = op->dop.flow_del.stats; - push = &push_buf; - - if (op->dop.type != DPIF_OP_FLOW_DEL) { - /* Only deleted flows need their stats pushed. */ - continue; - } if (op->dop.error) { - /* flow_del error, 'stats' is unusable. */ if (op->ukey) { ovs_mutex_lock(&op->ukey->mutex); transition_ukey(op->ukey, UKEY_EVICTED); ovs_mutex_unlock(&op->ukey->mutex); } + /* if it's a flow_del error, 'stats' is unusable, it's ok + * to discard the stats. + */ + continue; + } + + struct dpif_flow_stats *push, *stats, push_buf; + + stats = op->dop.flow_del.stats; + push = &push_buf; + + if (op->dop.type != DPIF_OP_FLOW_DEL) { + /* Only deleted flows need their stats pushed. */ continue; } From patchwork Fri Sep 23 16:29:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng He X-Patchwork-Id: 1681767 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jdpHxJzs; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MZH9Y4sc0z1yqH for ; Sat, 24 Sep 2022 15:08:49 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id E2E1E41B61; Sat, 24 Sep 2022 05:08:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org E2E1E41B61 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jdpHxJzs X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fi_eVmCwa7xi; Sat, 24 Sep 2022 05:08:43 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 7F69041B4E; Sat, 24 Sep 2022 05:08:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 7F69041B4E Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7168CC007C; Sat, 24 Sep 2022 05:08:41 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 820C2C007C for ; Sat, 24 Sep 2022 05:08:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 4FD5F813F0 for ; Sat, 24 Sep 2022 05:08:40 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 4FD5F813F0 Authentication-Results: smtp1.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jdpHxJzs X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RbTPwChfbQTJ for ; Sat, 24 Sep 2022 05:08:39 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org A11F7813DE Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by smtp1.osuosl.org (Postfix) with ESMTPS id A11F7813DE for ; Sat, 24 Sep 2022 05:08:39 +0000 (UTC) Received: by mail-pf1-x431.google.com with SMTP id d10so800903pfh.6 for ; Fri, 23 Sep 2022 22:08:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=kE+JR+ryu8ZWLLha1+tEucqyeyXP1OXhjCN/2FCX3No=; b=jdpHxJzsiuevUpuU8emqElyJn1aqEsDHfhCmnG2sgsla4w4bj3bnRvS7tprcStH1g7 Ih5yf5GW6/a2ZJZ6N4DkQh25W6sBJeIU8gwHExAF52/xlvnSACcYcHCfst3RiwdsJqi3 knAXL4hKluoT5yD1nb+uB34rO5bSvW7CD/kKacVlrfxE+Rxi6SFSipqzqClzhGmB2VYM e+MAs9pufs8OB6XG4dOEhrRkoEZy6HhzeAQ40nssWQ/VRcD5tZLKELlw6UFZ7rvp/mxF Qo7RDG8FQKxTEO1dMaOiCQgnxbPnKNsDH7rnVSogtZT41iLbkPMh9t9GhX3mPiqC/w4e KBfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=kE+JR+ryu8ZWLLha1+tEucqyeyXP1OXhjCN/2FCX3No=; b=g8P1eZihYQ7qQZZUEGPrJ1I8eGwG+EDrpyht5/G2HrVdcJj00m1LarpoMyR/Cy2xk7 g5Srk3KSn3a2Aq9D4EQ5sZ/rorNTj6Ameyh9xyhsOPs2qVSsJW88s4BxlleC9Yrpel5P uNqevDtN9wJeQy3p+qNnSsVkUb/nkEHcUdb2O9PNInTsOdGi/JUxvG5RJkGEWdReDimJ Fw0lFJ7QC8/u9FWu/wmDfbD87sFyLyDAjpB4L+NqE494JxnCXEKT8wYD3EpbQH8n2IdC ftniIyQv/o30DivJD/1WMrc3rGhko8jeha7lnTWYTO8AXg2dgBkpZLo4v2TWf/UV8bcu U98g== X-Gm-Message-State: ACrzQf3e1zbmRJxCiqNlXUXEH+dpdk+wVmv8rIm83lGv0gnsjYovCl3n codFaDYW7okgxBDJhqKR87sFXBdahfv69EOC X-Google-Smtp-Source: AMsMyM4Brxyi0SuPVlU4E4QG9TdFs/hg+SaWqCDYGZKztkhu0wDBrlYyeSin/RTNQ0Glbi809sVLaw== X-Received: by 2002:a05:6a00:a82:b0:547:d660:c077 with SMTP id b2-20020a056a000a8200b00547d660c077mr12474761pfl.38.1663996118783; Fri, 23 Sep 2022 22:08:38 -0700 (PDT) Received: from localhost ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w24-20020a17090a461800b001fe39bda429sm2448211pjg.38.2022.09.23.22.08.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Sep 2022 22:08:38 -0700 (PDT) From: Peng He X-Google-Original-From: Peng He To: dev@openvswitch.org Date: Fri, 23 Sep 2022 16:29:18 +0000 Message-Id: <20220923162920.612492-2-hepeng.0320@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220923162920.612492-1-hepeng.0320@bytedance.com> References: <20220923162920.612492-1-hepeng.0320@bytedance.com> MIME-Version: 1.0 Subject: [ovs-dev] [ovs-dev v3 2/4] ofproto-dpif-upcall: fix race condition X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" There is a race condition between the revalidator threads and the handler/pmd threads. revalidator PMD threads push_dp_ops deletes a key and tries to del the dp magaflow. does the upcall, generates a new ukey, and replaces the old ukey, now the old ukey state is UKEY_DELETED dp_ops succeeds, tries to change the old ukey's state into UKEY_EVICTED, however, the old ukey's state is already UKEY_DELETED, so OVS aborts. I did not observe this in the real environment, as it takes time for PMDs to finish the upcall and replace the old ukeys. Normally, the revalidator will change ukey state into UKEY_EVICTED first. But it's better to cover this case. Signed-off-by: Peng He --- ofproto/ofproto-dpif-upcall.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 7ea2a30f5..e8bbcfeaf 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -2420,7 +2420,9 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops, size_t n_ops) if (op->dop.error) { if (op->ukey) { ovs_mutex_lock(&op->ukey->mutex); - transition_ukey(op->ukey, UKEY_EVICTED); + if (op->ukey->state < UKEY_EVICTED) { + transition_ukey(op->ukey, UKEY_EVICTED); + } ovs_mutex_unlock(&op->ukey->mutex); } /* if it's a flow_del error, 'stats' is unusable, it's ok @@ -2441,7 +2443,9 @@ push_dp_ops(struct udpif *udpif, struct ukey_op *ops, size_t n_ops) if (op->ukey) { ovs_mutex_lock(&op->ukey->mutex); - transition_ukey(op->ukey, UKEY_EVICTED); + if (op->ukey->state < UKEY_EVICTED) { + transition_ukey(op->ukey, UKEY_EVICTED); + } push->used = MAX(stats->used, op->ukey->stats.used); push->tcp_flags = stats->tcp_flags | op->ukey->stats.tcp_flags; push->n_packets = stats->n_packets - op->ukey->stats.n_packets; From patchwork Fri Sep 23 16:29:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng He X-Patchwork-Id: 1681768 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=ZGyROvvW; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MZH9Z2XWcz1yqM for ; Sat, 24 Sep 2022 15:08:49 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id BD5DD60D95; Sat, 24 Sep 2022 05:08:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org BD5DD60D95 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=ZGyROvvW X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h0nc_KUK5-Eb; Sat, 24 Sep 2022 05:08:46 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 9D39660E07; Sat, 24 Sep 2022 05:08:45 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 9D39660E07 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 617A7C0070; Sat, 24 Sep 2022 05:08:45 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0AAC0C002D for ; Sat, 24 Sep 2022 05:08:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C0C4541B5F for ; Sat, 24 Sep 2022 05:08:43 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org C0C4541B5F Authentication-Results: smtp4.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=ZGyROvvW X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AY5xcp0eLmf1 for ; Sat, 24 Sep 2022 05:08:43 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 1A14C41B5D Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by smtp4.osuosl.org (Postfix) with ESMTPS id 1A14C41B5D for ; Sat, 24 Sep 2022 05:08:43 +0000 (UTC) Received: by mail-pj1-x1031.google.com with SMTP id x1-20020a17090ab00100b001fda21bbc90so7684508pjq.3 for ; Fri, 23 Sep 2022 22:08:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=yDTgknB+Yx8fauACd/B1W8M0qilLiOhYSyraWmujGUg=; b=ZGyROvvW1/UzuBMyTcKYet1ND8tlM6SQv67J86k8DiF3NdujIaxGQYf5WDKUUM8wSA luUpKQvLzLVQGT4yilbqB/MHH1oWLGxKcBJeQNNVSoVO73GqnBEgasdqio9UvKAtIlGB kkV7DivYa3whNmUYVTSogad3vi63u89kgRLL5Lbu9qUGxLaxmmSpYlDr5L26wlcGmcVf cY+44GHfHB3aAms3xhBHhckjStkyAZVER0i+sjSglUqltPSIzJEvOeHl7iTZbzBSd4BZ l47LF6VS31dEEDC3QloZrh6VpMK67cUph48CaLDXWVr8wWbWFEUT5BnriyE097inhk4g UcPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=yDTgknB+Yx8fauACd/B1W8M0qilLiOhYSyraWmujGUg=; b=3asBOs7DqByipl7E7t+xB+a0lB7/Gra7KsoLB6W2Y82O7BcW4CvnJU3YNgUQZ3uNde USCGuM1TNIR6WcHztKDPZGTtvv7nA+kB9Emoctzo/ILK69Yn7UQQ6YPdkspusTg2JYX7 r7i91vHKHSkd0sR8WpDYTnVDipTiCEKh4SLSsTA1WLaOn8BAVy7DZXojyV9mp596nZxz 8qtQLtSGHZcEu6kpmULiTc6XAtEK/GDLMCgvioeUU3hAIU/XXZxF2DW/5Cs25hTB8Khm SSFsBuzbfDStmT2259zvPhQ6qD7GHYDc/WmVqCR2k1ZmY4goWgFL6witGHwg48w/o4BM jzWg== X-Gm-Message-State: ACrzQf0C5fUoKBEo6BsOi6Xv0ZWT7gQDkAF3TNtuIFHILbTKLLqT/ekM aHIW2SAbJeWD/IOxF17r9IdzH21iUCeRE5ZD X-Google-Smtp-Source: AMsMyM77yR8WIiYT5n9o7gjc4UMmyBb8NmzEoEVZOti1UP0dwlkyOevkbIYJrcD5p55KDgT46540ow== X-Received: by 2002:a17:90a:7843:b0:203:5861:fc3d with SMTP id y3-20020a17090a784300b002035861fc3dmr13105562pjl.87.1663996121981; Fri, 23 Sep 2022 22:08:41 -0700 (PDT) Received: from localhost ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id w22-20020aa79556000000b0053e0d6f353esm7467986pfq.27.2022.09.23.22.08.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Sep 2022 22:08:41 -0700 (PDT) From: Peng He X-Google-Original-From: Peng He To: dev@openvswitch.org Date: Fri, 23 Sep 2022 16:29:19 +0000 Message-Id: <20220923162920.612492-3-hepeng.0320@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220923162920.612492-1-hepeng.0320@bytedance.com> References: <20220923162920.612492-1-hepeng.0320@bytedance.com> MIME-Version: 1.0 Subject: [ovs-dev] [ovs-dev v3 3/4] ofproto-dpif-upcall: new ukey needs to take the old ukey's dump seq X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The userspace datapath mananges all the magaflows by a cmap. The cmap data structrue will grow/shrink during the datapath processing and it will re-position megaflows. This might result in two revalidator threads might process a same megaflow during one dump stage. Consider a situation that, revalidator 1 processes a megaflow A, and decides to delete it from the datapath, at the mean time, this megaflow A is also queued in the process batch of revalidator 2. Normally it's ok for revalidators to process the same megaflow multiple times, as the dump_seq shows it's already dumped and the stats will not be contributed twice. Assume that right after A is deleted, a PMD thread generates again a new megaflow B which has the same match and action of A. The ukey of megaflow B will replace the one of megaflow A. Now the ukey B is new to the revalidator system and its dump seq is 0. Now since the dump seq of ukey B is 0, when processing megaflow A, the revalidator 2 will not identify this megaflow A has already been dumped by revalidator 1 and will contribute the old megaflow A's stats again, this results in an inconsistent stats between ukeys and megaflows. To fix this, the newly generated the ukey B should take the dump_seq of the replaced ukey A to avoid a same megaflow being revalidated twice in one dump stage. We observe in the production environment, the OpenFlow rules' stats sometimes are amplified compared to the actual value. I believe this is also the reason that why somtimes there is mismatch between the ukey and megaflow in stats value. The Eelco's patch [ovs-dev] [PATCH v2 09/10] revalidator: Fix datapath statistics update tried to fix it in the past. Signed-off-by: Peng He --- ofproto/ofproto-dpif-upcall.c | 1 + 1 file changed, 1 insertion(+) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index e8bbcfeaf..89fad1bdf 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -1877,6 +1877,7 @@ try_ukey_replace(struct umap *umap, struct udpif_key *old_ukey, ovs_mutex_lock(&new_ukey->mutex); cmap_replace(&umap->cmap, &old_ukey->cmap_node, &new_ukey->cmap_node, new_ukey->hash); + new_ukey->dump_seq = old_ukey->dump_seq; ovsrcu_postpone(ukey_delete__, old_ukey); transition_ukey(old_ukey, UKEY_DELETED); transition_ukey(new_ukey, UKEY_VISIBLE); From patchwork Fri Sep 23 16:29:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peng He X-Patchwork-Id: 1681769 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=V7HVjAwF; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MZH9c3kn8z1yqH for ; Sat, 24 Sep 2022 15:08:52 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 506A240FBF; Sat, 24 Sep 2022 05:08:50 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 506A240FBF Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=V7HVjAwF X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8KEev4xonNLO; Sat, 24 Sep 2022 05:08:49 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 53DDC410B9; Sat, 24 Sep 2022 05:08:48 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 53DDC410B9 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 213C9C007F; Sat, 24 Sep 2022 05:08:48 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 265ABC007C for ; Sat, 24 Sep 2022 05:08:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id D06B060D79 for ; Sat, 24 Sep 2022 05:08:46 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org D06B060D79 Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=V7HVjAwF X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H5zO7d1v-hWd for ; Sat, 24 Sep 2022 05:08:46 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 305CA60DDF Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) by smtp3.osuosl.org (Postfix) with ESMTPS id 305CA60DDF for ; Sat, 24 Sep 2022 05:08:46 +0000 (UTC) Received: by mail-pg1-x530.google.com with SMTP id 129so565204pgc.5 for ; Fri, 23 Sep 2022 22:08:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date; bh=9qzSWnywehD6turLpoex3ifk+j4rOro8W6h3KpAFec4=; b=V7HVjAwFh0ExEDPQ1lKy2rUjJrvBXSxZd21jpoaHn8HkTkbWQdyzTsJZ0vzikqR6gd lpMnyk8lwfUZvkCE43dYaUbtPizaXuCwVaeSFr8DpjbSxzuzCyDnnpgaxYKoFE+/d6uc rooLvGmbDwEhxAi800T8/527ltUF4Z/3uqiwtcNe8WUjixbeH8XCrqi5yqbU9tLZYmlA TJeE4a6ULJK+pi2iseX2iVMBNOj1HX1JLc2qkx9zfu1iTxNYhXyGmQqIXH91mVlWpRwG 31UZcCAveyX+crDSjNlpmuHAzIEKbA+UXDVgjRLbN9phktBmn7NNDHmbX11hQ8rrALv1 tB4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date; bh=9qzSWnywehD6turLpoex3ifk+j4rOro8W6h3KpAFec4=; b=F22SnSxAHEoD27LW+1abvRAMhqq9MdxY/J1uI5BRk2EwuvP4EpbvpMaD5HT4Equ16l idfC8nt6WF1QyUJ1YjKoufvYviVAWPNnn5sUc6TXKc41CowGeZYNqMkEQBRatbSkcZo5 GZ0tk8aesROGpdGihaH20VibxWSMovqVaSjewFQtrcqyBz1vZNgFewqBTvFX7O50TOHp koQ4aPQ2BDhG1PVMtasAAnatKX+a4SJd4Mcko6etp0RkbLXSNBorlPl/ueVWDKqRZA3s RjBdOIJ8EnWFxD2cCPUp0c1LMzoQVy60fLPgH6NaLU7kj+wymxtZ0bc0QJ3w0cL3kppC aV+A== X-Gm-Message-State: ACrzQf2PrqClcz+UU09oOrwiuJmNtaPVL5IO2qla6O1MBVDaPu4J0pOD 56Kk+d3/WrwKoL+zZVhx/mwppEV2MURXOjz+ X-Google-Smtp-Source: AMsMyM4P7UHgmUrI6BeVmhyZmrS4qHnzX/k43eFy1VuTuVfPIUjcYzDVsDvPOUnH0ivTt/A8rBvDfA== X-Received: by 2002:a05:6a00:2314:b0:546:ce91:89a3 with SMTP id h20-20020a056a00231400b00546ce9189a3mr12416302pfh.77.1663996125376; Fri, 23 Sep 2022 22:08:45 -0700 (PDT) Received: from localhost ([139.177.225.230]) by smtp.gmail.com with ESMTPSA id e30-20020a63545e000000b00429c5270710sm6631473pgm.1.2022.09.23.22.08.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Sep 2022 22:08:44 -0700 (PDT) From: Peng He X-Google-Original-From: Peng He To: dev@openvswitch.org Date: Fri, 23 Sep 2022 16:29:20 +0000 Message-Id: <20220923162920.612492-4-hepeng.0320@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220923162920.612492-1-hepeng.0320@bytedance.com> References: <20220923162920.612492-1-hepeng.0320@bytedance.com> MIME-Version: 1.0 Subject: [ovs-dev] [ovs-dev v3 4/4] dpif-netdev: fix inconsistent processing between ukey and megaflow X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When PMDs perform upcalls, the newly generated ukey will replace the old, however, the newly generated mageflow will be discard to reuse the old one without checking if the actions of new and old are equal. This code prevents in case someone runs dpctl/add-flow to add a dp flow with inconsistent actions with the actions of ukey, and causes more confusion. Signed-off-by: Peng He --- lib/dpif-netdev.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index a45b46014..b316e59ef 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -8304,7 +8304,22 @@ handle_packet_upcall(struct dp_netdev_pmd_thread *pmd, * to be locking revalidators out of making flow modifications. */ ovs_mutex_lock(&pmd->flow_mutex); netdev_flow = dp_netdev_pmd_lookup_flow(pmd, key, NULL); - if (OVS_LIKELY(!netdev_flow)) { + if (OVS_UNLIKELY(netdev_flow)) { + struct dp_netdev_actions *old_act = + dp_netdev_flow_get_actions(netdev_flow); + + if ((add_actions->size != old_act->size) || + memcmp(old_act->actions, add_actions->data, + add_actions->size)) { + + struct dp_netdev_actions *new_act = + dp_netdev_actions_create(add_actions->data, + add_actions->size); + + ovsrcu_set(&netdev_flow->actions, new_act); + ovsrcu_postpone(dp_netdev_actions_free, old_act); + } + } else { netdev_flow = dp_netdev_flow_add(pmd, &match, &ufid, add_actions->data, add_actions->size, orig_in_port);