From patchwork Sat Feb 4 09:00:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "wushaohua@chinatelecom.cn" X-Patchwork-Id: 1737451 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4P862G2cV5z23hn for ; Sat, 4 Feb 2023 20:01:09 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D978442A86; Sat, 4 Feb 2023 09:01:05 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org D978442A86 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I1GIhMHjyWdK; Sat, 4 Feb 2023 09:01:04 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 3145241D81; Sat, 4 Feb 2023 09:01:03 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 3145241D81 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 06489C0033; Sat, 4 Feb 2023 09:01:03 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3BFDBC002B for ; Sat, 4 Feb 2023 09:01:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 188B04038B for ; Sat, 4 Feb 2023 09:01:01 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 188B04038B X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vA0DwCttvzEg for ; Sat, 4 Feb 2023 09:00:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 09A8640286 Received: from chinatelecom.cn (prt-mail.chinatelecom.cn [42.123.76.226]) by smtp2.osuosl.org (Postfix) with ESMTP id 09A8640286 for ; Sat, 4 Feb 2023 09:00:57 +0000 (UTC) HMM_SOURCE_IP: 172.18.0.218:45490.1718971234 HMM_ATTACHE_NUM: 0000 HMM_SOURCE_TYPE: SMTP Received: from clientip-101.43.140.16 (unknown [172.18.0.218]) by chinatelecom.cn (HERMES) with SMTP id 8D734280099; Sat, 4 Feb 2023 17:00:48 +0800 (CST) X-189-SAVE-TO-SEND: wushaohua@chinatelecom.cn Received: from ([101.43.140.16]) by app0025 with ESMTP id 9cd1fab807364917919cd73a595ed237 for dev@openvswitch.org; Sat, 04 Feb 2023 17:00:52 CST X-Transaction-ID: 9cd1fab807364917919cd73a595ed237 X-Real-From: wushaohua@chinatelecom.cn X-Receive-IP: 101.43.140.16 X-MEDUSA-Status: 0 From: wushaohua@chinatelecom.cn To: dev@openvswitch.org Date: Sat, 4 Feb 2023 17:00:47 +0800 Message-Id: <20230204090047.778044-1-wushaohua@chinatelecom.cn> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Cc: wenxu@chinatelecom.cn Subject: [ovs-dev] [PATCH] dpif-netdev:The ukey is deleted during del-flows X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Shaohua Wu description: Using the command line to delete flow tables is a risky behavior. When the netdev flow table was deleted, only the dpflow was deleted. The ukey associated with the dpflow is not deleted. When the packet sending persists, the packet fails to match the flow table after the dpflow is deleted. After the upcall matches openflow, the dpflow fails to be generated because the ukey of the ufid is still in the cache.Frequent flow table matching failures are sent to the controller, and increased controller load. The forwarding capability decreases. scheme: After the dp flow table is deleted, the ukey associated with it is also deleted. Signed-off-by: Shaohua Wu --- lib/dpctl.c | 2 +- lib/dpif-netdev.c | 72 +++++++++++++++++++++++++++++++++++ lib/dpif-netlink.c | 2 + lib/dpif-provider.h | 11 ++++++ lib/dpif.c | 20 ++++++++++ lib/dpif.h | 4 ++ ofproto/ofproto-dpif-upcall.c | 29 +++++++++++++- 7 files changed, 138 insertions(+), 2 deletions(-) diff --git a/lib/dpctl.c b/lib/dpctl.c index c501a0cd7..8b5be755b 100644 --- a/lib/dpctl.c +++ b/lib/dpctl.c @@ -1579,7 +1579,7 @@ dpctl_del_flows(int argc, const char *argv[], struct dpctl_params *dpctl_p) return error; } - error = dpif_flow_flush(dpif); + error = dpif_flow_and_ukey_flush(dpif); if (error) { dpctl_error(dpctl_p, error, "deleting all flows"); } diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index c9f7179c3..0375a4bdc 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -303,6 +303,10 @@ struct dp_netdev { dp_purge_callback *dp_purge_cb; void *dp_purge_aux; + /* Callback function for notifying clean ukey */ + del_ukey_callback *del_ukey_cb; + void *del_ukey_aux; + /* Stores all 'struct dp_netdev_pmd_thread's. */ struct cmap poll_threads; /* id pool for per thread static_tx_qid. */ @@ -1859,6 +1863,7 @@ create_dp_netdev(const char *name, const struct dpif_class *class, dp_netdev_disable_upcall(dp); dp->upcall_aux = NULL; dp->upcall_cb = NULL; + dp->del_ukey_cb = NULL; dp->conntrack = conntrack_init(); @@ -3052,6 +3057,62 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, dp_netdev_offload_flow_enqueue(item); } +static void +dp_netdev_remove_ukey(struct dp_netdev_flow *flow, + struct dp_netdev_pmd_thread *pmd) +{ + struct dp_netdev *dp = pmd->dp; + + if (dp->del_ukey_cb) { + dp->del_ukey_cb(dp->del_ukey_aux,flow->ufid,pmd->core_id); + } +} + +static void +dp_netdev_pmd_remove_flow_and_ukey(struct dp_netdev_pmd_thread *pmd, + struct dp_netdev_flow *flow) + OVS_REQUIRES(pmd->flow_mutex) +{ + struct cmap_node *node = CONST_CAST(struct cmap_node *, &flow->node); + struct dpcls *cls; + odp_port_t in_port = flow->flow.in_port.odp_port; + + cls = dp_netdev_pmd_lookup_dpcls(pmd, in_port); + ovs_assert(cls != NULL); + dpcls_remove(cls, &flow->cr); + dp_netdev_simple_match_remove(pmd, flow); + cmap_remove(&pmd->flow_table, node, dp_netdev_flow_hash(&flow->ufid)); + ccmap_dec(&pmd->n_flows, odp_to_u32(in_port)); + queue_netdev_flow_del(pmd, flow); + flow->dead = true; + dp_netdev_remove_ukey(flow,pmd); + dp_netdev_flow_unref(flow); +} + +static void +dp_netdev_pmd_flow_and_ukey_flush(struct dp_netdev_pmd_thread *pmd) +{ + struct dp_netdev_flow *netdev_flow; + + ovs_mutex_lock(&pmd->flow_mutex); + CMAP_FOR_EACH (netdev_flow, node, &pmd->flow_table) { + dp_netdev_pmd_remove_flow_and_ukey(pmd, netdev_flow); + } + ovs_mutex_unlock(&pmd->flow_mutex); +} + +static int +dpif_netdev_flow_and_ukey_flush(struct dpif *dpif) +{ + struct dp_netdev *dp = get_dp_netdev(dpif); + struct dp_netdev_pmd_thread *pmd; + + CMAP_FOR_EACH (pmd, node, &dp->poll_threads) { + dp_netdev_pmd_flow_and_ukey_flush(pmd); + } + return 0; +} + static void dp_netdev_pmd_remove_flow(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow) @@ -8653,6 +8714,15 @@ dpif_netdev_register_upcall_cb(struct dpif *dpif, upcall_callback *cb, dp->upcall_cb = cb; } +static void +dpif_netdev_register_del_ukey_cb(struct dpif *dpif, del_ukey_callback *cb, + void *aux) +{ + struct dp_netdev *dp = get_dp_netdev(dpif); + dp->del_ukey_aux = aux; + dp->del_ukey_cb = cb; +} + static void dpif_netdev_xps_revalidate_pmd(const struct dp_netdev_pmd_thread *pmd, bool purge) @@ -9655,6 +9725,7 @@ const struct dpif_class dpif_netdev_class = { NULL, /* recv_purge */ dpif_netdev_register_dp_purge_cb, dpif_netdev_register_upcall_cb, + dpif_netdev_register_del_ukey_cb, dpif_netdev_enable_upcall, dpif_netdev_disable_upcall, dpif_netdev_get_datapath_version, @@ -9696,6 +9767,7 @@ const struct dpif_class dpif_netdev_class = { NULL, /* cache_get_name */ NULL, /* cache_get_size */ NULL, /* cache_set_size */ + dpif_netdev_flow_and_ukey_flush, }; static void diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index 586fb8893..b32d31a1e 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -4554,6 +4554,7 @@ const struct dpif_class dpif_netlink_class = { dpif_netlink_recv_purge, NULL, /* register_dp_purge_cb */ NULL, /* register_upcall_cb */ + NULL, /* register_del_ukey_cb */ NULL, /* enable_upcall */ NULL, /* disable_upcall */ dpif_netlink_get_datapath_version, /* get_datapath_version */ @@ -4595,6 +4596,7 @@ const struct dpif_class dpif_netlink_class = { dpif_netlink_cache_get_name, dpif_netlink_cache_get_size, dpif_netlink_cache_set_size, + NULL, }; static int diff --git a/lib/dpif-provider.h b/lib/dpif-provider.h index 12477a24f..07ec5b46d 100644 --- a/lib/dpif-provider.h +++ b/lib/dpif-provider.h @@ -431,6 +431,14 @@ struct dpif_class { * callback on invocation. */ void (*register_upcall_cb)(struct dpif *, upcall_callback *, void *aux); + /* When 'dpif' is about to delete flow,the ukey associated with the flow + * is alse deleted.(e.g. Command line: ovs-appctl dpctl/del-flows) + * + * Registers an upcall callback function with 'dpif'. This is only used + * if 'dpif' needs to notify the deltete the flows. 'aux' is passed to + * the callback on invocation. */ + void (*register_del_ukey_cb)(struct dpif *, del_ukey_callback *,void *aux); + /* Enables upcalls if 'dpif' directly executes upcall functions. */ void (*enable_upcall)(struct dpif *); @@ -662,6 +670,9 @@ struct dpif_class { /* Set cache size. */ int (*cache_set_size)(struct dpif *dpif, uint32_t level, uint32_t size); + + /* delete ukey and flow */ + int (*ukey_and_flow_flush)(struct dpif *dpif); }; extern const struct dpif_class dpif_netlink_class; diff --git a/lib/dpif.c b/lib/dpif.c index fe4db83fb..a1cb81d10 100644 --- a/lib/dpif.c +++ b/lib/dpif.c @@ -934,6 +934,18 @@ dpif_flow_flush(struct dpif *dpif) return error; } +/* deletes all flows and ukeys from 'dpif'. Return 0 if successful, +* otherwise a positive errno value */ +int +dpif_flow_and_ukey_flush(struct dpif *dpif) +{ + int error; + + error = dpif->dpif_class->ukey_and_flow_flush(dpif); + log_operation(dpif, "ukey_and_flow_flush", error); + return error; + +} /* Attempts to install 'key' into the datapath, fetches it, then deletes it. * Returns true if the datapath supported installing 'flow', false otherwise. */ @@ -1526,6 +1538,14 @@ dpif_register_dp_purge_cb(struct dpif *dpif, dp_purge_callback *cb, void *aux) } } +void +dpif_register_del_ukey_cb(struct dpif *dpif, del_ukey_callback *cb, void *aux) +{ + if (dpif->dpif_class->register_del_ukey_cb) { + dpif->dpif_class->register_del_ukey_cb(dpif, cb, aux); + } +} + void dpif_register_upcall_cb(struct dpif *dpif, upcall_callback *cb, void *aux) { diff --git a/lib/dpif.h b/lib/dpif.h index 6cb4dae6d..1bce5c530 100644 --- a/lib/dpif.h +++ b/lib/dpif.h @@ -539,6 +539,7 @@ enum dpif_flow_put_flags { bool dpif_probe_feature(struct dpif *, const char *name, const struct ofpbuf *key, const struct ofpbuf *actions, const ovs_u128 *ufid); +int dpif_flow_and_ukey_flush(struct dpif *dpif); int dpif_flow_flush(struct dpif *); int dpif_flow_put(struct dpif *, enum dpif_flow_put_flags, const struct nlattr *key, size_t key_len, @@ -882,6 +883,9 @@ typedef int upcall_callback(const struct dp_packet *packet, void dpif_register_upcall_cb(struct dpif *, upcall_callback *, void *aux); +typedef int del_ukey_callback(void *aux, ovs_u128 ufid, unsigned pmd_id); +void dpif_register_del_ukey_cb(struct dpif *, del_ukey_callback *, void *aux); + int dpif_recv_set(struct dpif *, bool enable); int dpif_handlers_set(struct dpif *, uint32_t n_handlers); bool dpif_number_handlers_required(struct dpif *, uint32_t *n_handlers); diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 31ac02d11..32d4e5a71 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -415,6 +415,7 @@ static int udpif_flow_unprogram(struct udpif *udpif, struct udpif_key *ukey, static upcall_callback upcall_cb; static dp_purge_callback dp_purge_cb; +static del_ukey_callback dp_del_ukey_cb; static atomic_bool enable_megaflows = ATOMIC_VAR_INIT(true); static atomic_bool enable_ufid = ATOMIC_VAR_INIT(true); @@ -473,6 +474,7 @@ udpif_create(struct dpif_backer *backer, struct dpif *dpif) dpif_register_upcall_cb(dpif, upcall_cb, udpif); dpif_register_dp_purge_cb(dpif, dp_purge_cb, udpif); + dpif_register_del_ukey_cb(dpif, dp_del_ukey_cb, udpif); return udpif; } @@ -2972,7 +2974,32 @@ dp_purge_cb(void *aux, unsigned pmd_id) } udpif_resume_revalidators(udpif); } - + +static int +dp_del_ukey_cb(void *aux, ovs_u128 ufid, unsigned pmd_id) + OVS_NO_THREAD_SAFETY_ANALYSIS +{ + struct udpif *udpif = aux; + size_t i; + + for (i = 0; i < N_UMAPS; i++) { + struct udpif_key *ukey; + struct umap *umap = &udpif->ukeys[i]; + + CMAP_FOR_EACH (ukey, cmap_node, &umap->cmap) { + if (ovs_u128_equals(ukey->ufid, ufid) && + pmd_id == ukey->pmd_id) { + transition_ukey(ukey, UKEY_EVICTING); + ovs_mutex_lock(&umap->mutex); + ukey_delete(umap, ukey); + ovs_mutex_unlock(&umap->mutex); + } + } + ovsrcu_quiesce(); + } + return 0; +} + static void upcall_unixctl_show(struct unixctl_conn *conn, int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, void *aux OVS_UNUSED)