From patchwork Mon Apr 12 15:19:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465328 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=LoNXI3GV; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=O1yjmEev; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsqy5t5Pz9sTD for ; Tue, 13 Apr 2021 01:20:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 4E35883BA0; Mon, 12 Apr 2021 15:20:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3QOVdM4H-1h5; Mon, 12 Apr 2021 15:20:27 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 72FFB83B58; Mon, 12 Apr 2021 15:20:26 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8CE01C0015; Mon, 12 Apr 2021 15:20:24 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id CD5B0C000C for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C556540300 for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="LoNXI3GV"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="O1yjmEev" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vrtPKG-raClS for ; Mon, 12 Apr 2021 15:20:21 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp4.osuosl.org (Postfix) with ESMTPS id C3F87402C7 for ; Mon, 12 Apr 2021 15:20:21 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id D80925C01AF; Mon, 12 Apr 2021 11:20:18 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 12 Apr 2021 11:20:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=iCli+LJZq7/y4 Q+KfAylxzeVzJC3ji50jvvhzWsAmfM=; b=LoNXI3GVRIWIV8b4j1uyTpfEpS7Y7 Pestz8NeAqWI/HbptBjBY/WBJKfeQcvmgyQUC6xzuxaAhqR+B+j4/HMcYldz82Ob k8Ouu95NN4R8H/Y6pxjlP1/JQKDNaXCMs/z9XGhFh5Hf1pVi0PkB1RdwF180VY8Z hYX1iOsInqQQNYW/XA3D19sl9G62/iHn2aofK8kT0PL6jYb5Q267kPWFdDH2ewfX sQpflMK+r/RyF8uQBHnPiD/PvPksVGCn+ZQ6YjOPSsPCwWhkqWpKnHQ7AUp7cZYo 3jqnvX0RNhrgWF2bTvm5mFpLdBC9GRVtTddrCcAlCQNasfYbfZxjBxN3w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=iCli+LJZq7/y4Q+KfAylxzeVzJC3ji50jvvhzWsAmfM=; b=O1yjmEev nvCbb0hhAny++w1oZE3vP7US/zw3CdDO7tjLxonfFuJozvUj23Ff+sYPC/fdhtVu yamsxVhfgnl0W86LFHBBeHV+MQQzMfxa1PTtpmlrjpj0vV2F686C6vFp4bOIarpb KEXa7hp3tTYMe3DmH82V1Oso8sDAjUokYLzaDbF2yJdld7Q5C/r0yTFbT3/FeaJs PUTVYd/nKgGHa7dg3vEVXOGCYyGZaaDeye6Yzj96YlUMDXcXULhKMT1dRqdej5mt taJ4mcJnG7OfNGsSvdtkFc2IAz03TAE6G5h1/Jf/NljyzRsv8BCMWindkzp9xVvB WCczBljqmMkUmQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 2CEC824006A; Mon, 12 Apr 2021 11:20:18 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:47 +0200 Message-Id: <0c479781eb449c20726196daaded9fdddf20f996.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 01/28] netdev: Add flow API uninit function X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a new operation for flow API providers to uninitialize when the API is disassociated from a netdev. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-provider.h | 3 +++ lib/netdev-offload.c | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/lib/netdev-offload-provider.h b/lib/netdev-offload-provider.h index cf859d1b4..2127599d3 100644 --- a/lib/netdev-offload-provider.h +++ b/lib/netdev-offload-provider.h @@ -90,6 +90,9 @@ struct netdev_flow_api { /* Initializies the netdev flow api. * Return 0 if successful, otherwise returns a positive errno value. */ int (*init_flow_api)(struct netdev *); + + /* Uninitializes the netdev flow api. */ + void (*uninit_flow_api)(struct netdev *); }; int netdev_register_flow_api_provider(const struct netdev_flow_api *); diff --git a/lib/netdev-offload.c b/lib/netdev-offload.c index 6237667c3..deefefd63 100644 --- a/lib/netdev-offload.c +++ b/lib/netdev-offload.c @@ -320,6 +320,10 @@ netdev_uninit_flow_api(struct netdev *netdev) return; } + if (flow_api->uninit_flow_api) { + flow_api->uninit_flow_api(netdev); + } + ovsrcu_set(&netdev->flow_api, NULL); rfa = netdev_lookup_flow_api(flow_api->type); ovs_refcount_unref(&rfa->refcnt); From patchwork Mon Apr 12 15:19:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465330 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=F0KRDcsA; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=MuDAWkw2; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsr16Zszz9sVm for ; Tue, 13 Apr 2021 01:20:33 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 2BF17838B6; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5QoqM0G2wKgc; Mon, 12 Apr 2021 15:20:28 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id A0B3683AE9; Mon, 12 Apr 2021 15:20:27 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 83792C001A; Mon, 12 Apr 2021 15:20:25 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 18773C000A for ; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E1C4D403FF for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="F0KRDcsA"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="MuDAWkw2" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 32fWINPNWL7e for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1D74C4040E for ; Mon, 12 Apr 2021 15:20:21 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 895735C01C6; Mon, 12 Apr 2021 11:20:19 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 12 Apr 2021 11:20:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=Pfe/c9DX8up1i xISr/Ejsf4QuE/U35JrChRdk/7Ru/w=; b=F0KRDcsA4nDxhisndNbLFFOW3TLMZ 6rIL8XWUUiX+H1TX2+sa7BrVqXUj7g3Rn8VLJdjYJrP5jQmOuBH+T9WgWgpeemvn fS9VMO32PxN/LhGkN99M6th9pt/wK0MPLbxnzMZokqju3j5ljZNV2DWvWRaPyUyk knO5ey/3a5RAepMjc5AJqj7fVKRyqgZplDR71nWFJ5nOgrzZA+XWPo86h2jHCzvv u+z54FLzru0fCKOgzm3cWrA0htA2M/25OOr6o14R9vGR7w4Gu6a7oUMaD3QyCKYZ wq8q5UQJVo9levkefsRaq6VxRdlMTrKxjiGCEzUoBds4CtRh6h6Jha7jA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=Pfe/c9DX8up1ixISr/Ejsf4QuE/U35JrChRdk/7Ru/w=; b=MuDAWkw2 Pg9+GthBXz/ZPjBURehcxAMIhQGSQZ6gFucrlTGqOs5e6BWoBShaDWIrYMRmAefW Gz2LQIpdZz/w+JQPnFgE8oLBFKaAw9LVzGrajQ8xMNyyAjUXQ7gUcXZMN91nKcRl PDCbWkS2xc9tS0iuMhZ/C7atDqFSRS1la6Y2ivgtTOFGC9mZ/13fVD+aKlCw5u9O QZWtoFZ5O2U9HMEhruajcDmvTmOA5jMJIqmmbh+nPVPUptquj2cxeGn5T1sIPgVl cpuCFNaxiVGXuABizZhkmvo8I969dJRps4TwQm++G9foYVBFySD3ngk+JNUmS9n2 S29kVBKWK/TNag== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id E6685240054; Mon, 12 Apr 2021 11:20:18 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:48 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 02/28] netdev-offload-dpdk: Use per-netdev offload metadata X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a per-netdev offload data field as part of netdev hw_info structure. Use this field in netdev-offload-dpdk to map offload metadata (ufid to rte_flow). Use flow API deinit ops to destroy the per-netdev metadata when deallocating a netdev. Use RCU primitives to ensure coherency during port deletion. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-dpdk.c | 131 ++++++++++++++++++++++++++++++++------ lib/netdev-offload.h | 2 + 2 files changed, 114 insertions(+), 19 deletions(-) diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index f2413f5be..01d09aca7 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -26,6 +26,7 @@ #include "netdev-provider.h" #include "openvswitch/match.h" #include "openvswitch/vlog.h" +#include "ovs-rcu.h" #include "packets.h" #include "uuid.h" @@ -52,7 +53,6 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(100, 5); /* * A mapping from ufid to dpdk rte_flow. */ -static struct cmap ufid_to_rte_flow = CMAP_INITIALIZER; struct ufid_to_rte_flow_data { struct cmap_node node; @@ -63,14 +63,81 @@ struct ufid_to_rte_flow_data { struct dpif_flow_stats stats; }; +struct netdev_offload_dpdk_data { + struct cmap ufid_to_rte_flow; +}; + +static int +offload_data_init(struct netdev *netdev) +{ + struct netdev_offload_dpdk_data *data; + + data = xzalloc(sizeof *data); + cmap_init(&data->ufid_to_rte_flow); + + ovsrcu_set(&netdev->hw_info.offload_data, (void *) data); + + return 0; +} + +static void +offload_data_destroy__(struct netdev_offload_dpdk_data *data) +{ + free(data); +} + +static void +offload_data_destroy(struct netdev *netdev) +{ + struct netdev_offload_dpdk_data *data; + struct ufid_to_rte_flow_data *node; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + if (data == NULL) { + return; + } + + if (!cmap_is_empty(&data->ufid_to_rte_flow)) { + VLOG_ERR("Incomplete flush: %s contains rte_flow elements", + netdev_get_name(netdev)); + } + + CMAP_FOR_EACH (node, node, &data->ufid_to_rte_flow) { + ovsrcu_postpone(free, node); + } + + cmap_destroy(&data->ufid_to_rte_flow); + ovsrcu_postpone(offload_data_destroy__, data); + + ovsrcu_set(&netdev->hw_info.offload_data, NULL); +} + +static struct cmap * +offload_data_map(struct netdev *netdev) +{ + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + + return data ? &data->ufid_to_rte_flow : NULL; +} + /* Find rte_flow with @ufid. */ static struct ufid_to_rte_flow_data * -ufid_to_rte_flow_data_find(const ovs_u128 *ufid, bool warn) +ufid_to_rte_flow_data_find(struct netdev *netdev, + const ovs_u128 *ufid, bool warn) { size_t hash = hash_bytes(ufid, sizeof *ufid, 0); struct ufid_to_rte_flow_data *data; + struct cmap *map = offload_data_map(netdev); + + if (!map) { + return NULL; + } - CMAP_FOR_EACH_WITH_HASH (data, node, hash, &ufid_to_rte_flow) { + CMAP_FOR_EACH_WITH_HASH (data, node, hash, map) { if (ovs_u128_equals(*ufid, data->ufid)) { return data; } @@ -85,12 +152,19 @@ ufid_to_rte_flow_data_find(const ovs_u128 *ufid, bool warn) } static inline struct ufid_to_rte_flow_data * -ufid_to_rte_flow_associate(const ovs_u128 *ufid, struct netdev *netdev, +ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, struct rte_flow *rte_flow, bool actions_offloaded) { size_t hash = hash_bytes(ufid, sizeof *ufid, 0); - struct ufid_to_rte_flow_data *data = xzalloc(sizeof *data); + struct cmap *map = offload_data_map(netdev); struct ufid_to_rte_flow_data *data_prev; + struct ufid_to_rte_flow_data *data; + + if (!map) { + return NULL; + } + + data = xzalloc(sizeof *data); /* * We should not simply overwrite an existing rte flow. @@ -98,7 +172,7 @@ ufid_to_rte_flow_associate(const ovs_u128 *ufid, struct netdev *netdev, * Thus, if following assert triggers, something is wrong: * the rte_flow is not destroyed. */ - data_prev = ufid_to_rte_flow_data_find(ufid, false); + data_prev = ufid_to_rte_flow_data_find(netdev, ufid, false); if (data_prev) { ovs_assert(data_prev->rte_flow == NULL); } @@ -108,8 +182,7 @@ ufid_to_rte_flow_associate(const ovs_u128 *ufid, struct netdev *netdev, data->rte_flow = rte_flow; data->actions_offloaded = actions_offloaded; - cmap_insert(&ufid_to_rte_flow, - CONST_CAST(struct cmap_node *, &data->node), hash); + cmap_insert(map, CONST_CAST(struct cmap_node *, &data->node), hash); return data; } @@ -117,9 +190,13 @@ static inline void ufid_to_rte_flow_disassociate(struct ufid_to_rte_flow_data *data) { size_t hash = hash_bytes(&data->ufid, sizeof data->ufid, 0); + struct cmap *map = offload_data_map(data->netdev); + + if (!map) { + return; + } - cmap_remove(&ufid_to_rte_flow, - CONST_CAST(struct cmap_node *, &data->node), hash); + cmap_remove(map, CONST_CAST(struct cmap_node *, &data->node), hash); netdev_close(data->netdev); ovsrcu_postpone(free, data); } @@ -1421,7 +1498,7 @@ netdev_offload_dpdk_add_flow(struct netdev *netdev, if (!flow) { goto out; } - flows_data = ufid_to_rte_flow_associate(ufid, netdev, flow, + flows_data = ufid_to_rte_flow_associate(netdev, ufid, flow, actions_offloaded); VLOG_DBG("%s: installed flow %p by ufid "UUID_FMT, netdev_get_name(netdev), flow, UUID_ARGS((struct uuid *)ufid)); @@ -1478,7 +1555,7 @@ netdev_offload_dpdk_flow_put(struct netdev *netdev, struct match *match, * Here destroy the old rte flow first before adding a new one. * Keep the stats for the newly created rule. */ - rte_flow_data = ufid_to_rte_flow_data_find(ufid, false); + rte_flow_data = ufid_to_rte_flow_data_find(netdev, ufid, false); if (rte_flow_data && rte_flow_data->rte_flow) { old_stats = rte_flow_data->stats; modification = true; @@ -1509,7 +1586,7 @@ netdev_offload_dpdk_flow_del(struct netdev *netdev OVS_UNUSED, { struct ufid_to_rte_flow_data *rte_flow_data; - rte_flow_data = ufid_to_rte_flow_data_find(ufid, true); + rte_flow_data = ufid_to_rte_flow_data_find(netdev, ufid, true); if (!rte_flow_data || !rte_flow_data->rte_flow) { return -1; } @@ -1523,7 +1600,21 @@ netdev_offload_dpdk_flow_del(struct netdev *netdev OVS_UNUSED, static int netdev_offload_dpdk_init_flow_api(struct netdev *netdev) { - return netdev_dpdk_flow_api_supported(netdev) ? 0 : EOPNOTSUPP; + int ret = EOPNOTSUPP; + + if (netdev_dpdk_flow_api_supported(netdev)) { + ret = offload_data_init(netdev); + } + + return ret; +} + +static void +netdev_offload_dpdk_uninit_flow_api(struct netdev *netdev) +{ + if (netdev_dpdk_flow_api_supported(netdev)) { + offload_data_destroy(netdev); + } } static int @@ -1540,7 +1631,7 @@ netdev_offload_dpdk_flow_get(struct netdev *netdev, struct rte_flow_error error; int ret = 0; - rte_flow_data = ufid_to_rte_flow_data_find(ufid, false); + rte_flow_data = ufid_to_rte_flow_data_find(netdev, ufid, false); if (!rte_flow_data || !rte_flow_data->rte_flow) { ret = -1; goto out; @@ -1575,13 +1666,14 @@ out: static int netdev_offload_dpdk_flow_flush(struct netdev *netdev) { + struct cmap *map = offload_data_map(netdev); struct ufid_to_rte_flow_data *data; - CMAP_FOR_EACH (data, node, &ufid_to_rte_flow) { - if (data->netdev != netdev) { - continue; - } + if (!map) { + return -1; + } + CMAP_FOR_EACH (data, node, map) { netdev_offload_dpdk_flow_destroy(data); } @@ -1593,6 +1685,7 @@ const struct netdev_flow_api netdev_offload_dpdk = { .flow_put = netdev_offload_dpdk_flow_put, .flow_del = netdev_offload_dpdk_flow_del, .init_flow_api = netdev_offload_dpdk_init_flow_api, + .uninit_flow_api = netdev_offload_dpdk_uninit_flow_api, .flow_get = netdev_offload_dpdk_flow_get, .flow_flush = netdev_offload_dpdk_flow_flush, }; diff --git a/lib/netdev-offload.h b/lib/netdev-offload.h index 18b48790f..d820e23ed 100644 --- a/lib/netdev-offload.h +++ b/lib/netdev-offload.h @@ -20,6 +20,7 @@ #include "openvswitch/netdev.h" #include "openvswitch/types.h" +#include "ovs-rcu.h" #include "packets.h" #include "flow.h" @@ -45,6 +46,7 @@ struct netdev_hw_info { bool oor; /* Out of Offload Resources ? */ int offload_count; /* Pending (non-offloaded) flow count */ int pending_count; /* Offloaded flow count */ + OVSRCU_TYPE(void *) offload_data; /* Offload metadata. */ }; enum hw_info_type { From patchwork Mon Apr 12 15:19:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465329 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=wiqkjWNf; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=DTkpyHeR; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsqx5Kz4z9rx6 for ; Tue, 13 Apr 2021 01:20:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id B369940440; Mon, 12 Apr 2021 15:20:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pAYgWCz_pNuP; Mon, 12 Apr 2021 15:20:24 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id E58A940001; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C224CC000C; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id BE27FC000A for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 9D62040417 for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AcnIBqw64AVS for ; Mon, 12 Apr 2021 15:20:21 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp2.osuosl.org (Postfix) with ESMTPS id C2F3F403FF for ; Mon, 12 Apr 2021 15:20:21 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 629745C01D1; Mon, 12 Apr 2021 11:20:20 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 12 Apr 2021 11:20:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=uoocCMMRcmx6U Fku7e3hdBOc+Vi4kbrufyveIo9mVHM=; b=wiqkjWNfbL5dfF9qA1cNLxGB1ofWn 6epi86nx7V4WPEYbOrWsHDzNbOi3+YRn+IFjO9KgHpYy0N4ugaYjfG3kuVKQ8UFh AY+EHECP7bLYudGuASmHESWMM2mUWUtb64OaTNyMsf+Gy/OXdvPhvsjECYI78KQY m2X+yj6g14jkOoO5mjwo7eHtNN/1igCPbSLMHkJWF0zQVc4KMSwDlhh7/k/q2guu q6RE8XOnJihHjFU3YLglEG8duLL+npA30a3RaYgjC3rrSOA0YBvskSnKTB4ZnzmB FWST7T52Uytd8UuKQnjSEkhDh5tV6ULATPWxZo6SniqOcHPeGXH3NNuYQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=uoocCMMRcmx6UFku7e3hdBOc+Vi4kbrufyveIo9mVHM=; b=DTkpyHeR nha21qK1YM56ApH+dcwwgFb0y8dwv3Yed8sLRKH+lARsfTGhwAUz/LYpIQ9OFSdu upgKXdt+P9pwmmmoE96RUrIjmMFqyOkXJbHM4qHhn4ATPyqhlj7WGCA9dVQYn81d FwLaIpmmBDuUy2jwOqSRSVFc6d4rtjFX7yeXSjzGqWNdRnbfYrUTu9nhelEXvDZg ZX4nZRsDtP9wGSd2013hlBkh/+nI5B7YiZt/2I8IFe+shNp6q0i5+6H+CeKpUT7n B3WR0xcKkNk3BnGGG30ECTqrYojzDWt8q5Y2wS4abM8eL2/rLkedLW00yCe3I0+P k/AbbD3h3SjxuQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id AA2D0240054; Mon, 12 Apr 2021 11:20:19 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:49 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 03/28] netdev-offload-dpdk: Implement hw-offload statistics read X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In the DPDK offload provider, keep track of inserted rte_flow and report it when queried. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-dpdk.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index 01d09aca7..c43e8b968 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -65,6 +65,7 @@ struct ufid_to_rte_flow_data { struct netdev_offload_dpdk_data { struct cmap ufid_to_rte_flow; + uint64_t rte_flow_counter; }; static int @@ -644,6 +645,12 @@ netdev_offload_dpdk_flow_create(struct netdev *netdev, flow = netdev_dpdk_rte_flow_create(netdev, attr, items, actions, error); if (flow) { + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + data->rte_flow_counter++; + if (!VLOG_DROP_DBG(&rl)) { dump_flow(&s, &s_extra, attr, items, actions); extra_str = ds_cstr(&s_extra); @@ -1524,6 +1531,12 @@ netdev_offload_dpdk_flow_destroy(struct ufid_to_rte_flow_data *rte_flow_data) ret = netdev_dpdk_rte_flow_destroy(netdev, rte_flow, &error); if (ret == 0) { + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + data->rte_flow_counter--; + ufid_to_rte_flow_disassociate(rte_flow_data); VLOG_DBG_RL(&rl, "%s: rte_flow 0x%"PRIxPTR " flow destroy %d ufid " UUID_FMT, @@ -1680,6 +1693,23 @@ netdev_offload_dpdk_flow_flush(struct netdev *netdev) return 0; } +static int +netdev_offload_dpdk_get_n_flows(struct netdev *netdev, + uint64_t *n_flows) +{ + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + if (!data) { + return -1; + } + + *n_flows = data->rte_flow_counter; + + return 0; +} + const struct netdev_flow_api netdev_offload_dpdk = { .type = "dpdk_flow_api", .flow_put = netdev_offload_dpdk_flow_put, @@ -1688,4 +1718,5 @@ const struct netdev_flow_api netdev_offload_dpdk = { .uninit_flow_api = netdev_offload_dpdk_uninit_flow_api, .flow_get = netdev_offload_dpdk_flow_get, .flow_flush = netdev_offload_dpdk_flow_flush, + .flow_get_n_flows = netdev_offload_dpdk_get_n_flows, }; From patchwork Mon Apr 12 15:19:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465332 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=kVgMX5O0; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=Bnw9g1uJ; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsr22QVXz9sWP for ; Tue, 13 Apr 2021 01:20:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 71A994048A; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ziujjUzJN0AM; Mon, 12 Apr 2021 15:20:29 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id C7C034046B; Mon, 12 Apr 2021 15:20:28 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 96699C0011; Mon, 12 Apr 2021 15:20:26 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6BE2AC000A for ; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 49DE8402C7 for ; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="kVgMX5O0"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="Bnw9g1uJ" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O1SlKsy2Czk7 for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp4.osuosl.org (Postfix) with ESMTPS id 328BF402CF for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 40DE65C01CF; Mon, 12 Apr 2021 11:20:21 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=c8cSj3eekuXm8 tViiD3eT6l8MozpBg9bsLdhjqj6epo=; b=kVgMX5O0OQJenrlQn99lOUnPVz8yo DOaM4OQFNe2Oz23bLHW55bCGW2kK0QaxDAHxK5Q2aTXbuk8e6Xu5er+LXPvpUOaj e+9HAh2zELVY9Exbk053KEeNjCPa6xkcxEXJabLa9Y0xRSKHdCfoxsP1HmBSKy6v ESFRNWpFDPs6VvFoLMxrjQsdYuA4NO2Xnv5uPlrISpHm0Axkz+8lOtKFkjVTqyBU zkDX50eD0+UZduWatefmZlhhqskz2OvgD0UTKx1ifgpBHhBKAUBI+UQWUAW6zkyy VekJzj355zE6qAI4fHCXjvQP8hxtuw6FLFqGfXZA/9BeDAmKTUG1yrTkQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=c8cSj3eekuXm8tViiD3eT6l8MozpBg9bsLdhjqj6epo=; b=Bnw9g1uJ 1i9DVOfbr54PUn6c6pW5spq6Pmm+1tyIhaZ7vpkz8Qe+pFxtQPwZsf1dgKSLFRH/ 0M2hLmX6dlDlg8E/xX8mEC2xbmTQhpcEzQIWQ/wT8vCCeBEgVhkJ3efQYxyxOZsp 63etI0UilkB7y0dcwEZSJMpFYvwQuZKssOLNmiXECnRgWmw+gW7plvMWem7AaQh6 wJWBcabGVoG4rMztOEeir3hTpuZ387Bb7XfUz2evP9nKbjfEs+drJ6B9HWI8tAvd Ho8kxuaDgRWLxo876V5XXqWCnRF8qwmA9S7PwhIj2VTPxDXkJKQpxOV5O5oIEwkC leqrLzpoie+tqQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 7496A240054; Mon, 12 Apr 2021 11:20:20 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:50 +0200 Message-Id: <97a444b245d0e5e5ee132a3e336529a625a63f21.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 04/28] dpctl: Add function to read hardware offload statistics X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Expose a function to query datapath offload statistics. This function is separate from the current one in netdev-offload as it exposes more detailed statistics from the datapath, instead of only from the netdev-offload provider. Each datapath is meant to use the custom counters as it sees fit for its handling of hardware offloads. Call the new API from dpctl. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpctl.c | 36 ++++++++++++++++++++++++++++++++++++ lib/dpif-netdev.c | 1 + lib/dpif-netlink.c | 1 + lib/dpif-provider.h | 7 +++++++ lib/dpif.c | 8 ++++++++ lib/dpif.h | 9 +++++++++ 6 files changed, 62 insertions(+) diff --git a/lib/dpctl.c b/lib/dpctl.c index ef8ae7402..6ff73e2d9 100644 --- a/lib/dpctl.c +++ b/lib/dpctl.c @@ -1541,6 +1541,40 @@ dpctl_del_flows(int argc, const char *argv[], struct dpctl_params *dpctl_p) return error; } +static int +dpctl_offload_stats_show(int argc, const char *argv[], + struct dpctl_params *dpctl_p) +{ + struct netdev_custom_stats stats; + struct dpif *dpif; + int error; + size_t i; + + error = opt_dpif_open(argc, argv, dpctl_p, 2, &dpif); + if (error) { + return error; + } + + memset(&stats, 0, sizeof(stats)); + error = dpif_offload_stats_get(dpif, &stats); + if (error) { + dpctl_error(dpctl_p, error, "retrieving offload statistics"); + goto close_dpif; + } + + dpctl_print(dpctl_p, "HW Offload stats:\n"); + for (i = 0; i < stats.size; i++) { + dpctl_print(dpctl_p, " %s: %6" PRIu64 "\n", + stats.counters[i].name, stats.counters[i].value); + } + + netdev_free_custom_stats_counters(&stats); + +close_dpif: + dpif_close(dpif); + return error; +} + static int dpctl_help(int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, struct dpctl_params *dpctl_p) @@ -2697,6 +2731,8 @@ static const struct dpctl_command all_commands[] = { { "add-flows", "[dp] file", 1, 2, dpctl_process_flows, DP_RW }, { "mod-flows", "[dp] file", 1, 2, dpctl_process_flows, DP_RW }, { "del-flows", "[dp] [file]", 0, 2, dpctl_del_flows, DP_RW }, + { "offload-stats-show", "[dp]", + 0, 1, dpctl_offload_stats_show, DP_RO }, { "dump-conntrack", "[-m] [-s] [dp] [zone=N]", 0, 4, dpctl_dump_conntrack, DP_RO }, { "flush-conntrack", "[dp] [zone=N] [ct-tuple]", 0, 3, diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 251788b04..2832671d8 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -8486,6 +8486,7 @@ const struct dpif_class dpif_netdev_class = { dpif_netdev_flow_dump_thread_destroy, dpif_netdev_flow_dump_next, dpif_netdev_operate, + NULL, /* offload_stats_get */ NULL, /* recv_set */ NULL, /* handlers_set */ dpif_netdev_set_config, diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index ceb56c685..924b0bd12 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -3972,6 +3972,7 @@ const struct dpif_class dpif_netlink_class = { dpif_netlink_flow_dump_thread_destroy, dpif_netlink_flow_dump_next, dpif_netlink_operate, + NULL, /* offload_stats_get */ dpif_netlink_recv_set, dpif_netlink_handlers_set, NULL, /* set_config */ diff --git a/lib/dpif-provider.h b/lib/dpif-provider.h index b817fceac..36dfa8e71 100644 --- a/lib/dpif-provider.h +++ b/lib/dpif-provider.h @@ -330,6 +330,13 @@ struct dpif_class { void (*operate)(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, enum dpif_offload_type offload_type); + /* Get hardware-offloads activity counters from a dataplane. + * Those counters are not offload statistics (which are accessible through + * netdev statistics), but a status of hardware offload management: + * how many offloads are currently waiting, inserted, etc. */ + int (*offload_stats_get)(struct dpif *dpif, + struct netdev_custom_stats *stats); + /* Enables or disables receiving packets with dpif_recv() for 'dpif'. * Turning packet receive off and then back on is allowed to change Netlink * PID assignments (see ->port_get_pid()). The client is responsible for diff --git a/lib/dpif.c b/lib/dpif.c index 56d0b4a65..30fc2db51 100644 --- a/lib/dpif.c +++ b/lib/dpif.c @@ -1426,6 +1426,14 @@ dpif_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, } } +int dpif_offload_stats_get(struct dpif *dpif, + struct netdev_custom_stats *stats) +{ + return (dpif->dpif_class->offload_stats_get + ? dpif->dpif_class->offload_stats_get(dpif, stats) + : EOPNOTSUPP); +} + /* Returns a string that represents 'type', for use in log messages. */ const char * dpif_upcall_type_to_string(enum dpif_upcall_type type) diff --git a/lib/dpif.h b/lib/dpif.h index ecda896c7..89af01a5b 100644 --- a/lib/dpif.h +++ b/lib/dpif.h @@ -787,6 +787,15 @@ struct dpif_op { void dpif_operate(struct dpif *, struct dpif_op **ops, size_t n_ops, enum dpif_offload_type); + +/* Queries the datapath for hardware offloads stats. + * + * Statistics are written in 'stats' following the 'netdev_custom_stats' + * format. They are allocated on the heap and must be freed by the caller, + * using 'netdev_free_custom_stats_counters'. + */ +int dpif_offload_stats_get(struct dpif *dpif, + struct netdev_custom_stats *stats); /* Upcalls. */ From patchwork Mon Apr 12 15:19:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465331 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=n0DBNO8r; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=g1qRUbby; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsr20zPWz9sW0 for ; Tue, 13 Apr 2021 01:20:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3873460AC9; Mon, 12 Apr 2021 15:20:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w-ksXb5-JNbf; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTP id 42FB160ABE; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 97722C0015; Mon, 12 Apr 2021 15:20:27 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 79B6DC000C for ; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 58F098378A for ; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="n0DBNO8r"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="g1qRUbby" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XFtrMxdJGxMe for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by smtp1.osuosl.org (Postfix) with ESMTPS id DD68583521 for ; Mon, 12 Apr 2021 15:20:22 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 364751814; Mon, 12 Apr 2021 11:20:22 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=Mx51GSSjVyT3K 5oXovxXFWWKKEN7/MQlZAQ6XwYvzkY=; b=n0DBNO8rgAcgB9IuT6bD0tpoz/w8n U1PFgg3IELvSKS0qw62gu1bcE+nG2hXQzbQLy0mVlNoSS+aaLv/om92jahL7uv1O aZDsG6BfVI0kEAFYEO4hd7Om9l8z5dbz0MpA48KKHaLvNJ63mb8jLhhlRCKUwlNe z0dvqjH2ylZI2vHvC8zKDkxZa3TknTc3Eg7EqguxboP4Mcqxv4nF+rrJlEV8mrY5 UlSW63fMg+kY3qgdD01atzd3cXW+j3Jo1pBc4SX2lZsgU4MOLYbfS1lx54l0iFIa ZL68orCUDrDy3pK1Kry10rxFCl1ZZk236HgNaAVl2YjSwswAMrx38gMJA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=Mx51GSSjVyT3K5oXovxXFWWKKEN7/MQlZAQ6XwYvzkY=; b=g1qRUbby cLp3o+pv/RX9LUiddQtwkLDGhWIwoJhdbj7UnpekCGuyo0eDXk8DJcyJawuSXCeI Kj2cEAsFf7yrONnA44Q5uFsdpY4Ck+/30fnXiiT7e3QJGQnTLBttsVe6yqFmRMbe +HgE87LBgYPiZTkw66Mayr7iXR1lqc0oVnsEtl0lML0zsDTVs1CwLrCM9P5eQ2Du BYvafCclx8YwKoT4FsBXQhFsBK7BlRKk+SNDMa+mQjlGbMdpg/5BB1z0k9KjMl/0 zPP6XHT+JRrptL55/EAqSqKYMtTqg8QVXXsWd63QkmgUhQUOaHu6QjAuaeHgftHm i7affzdt4JyD8A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhgggfestdekre dtredttdenucfhrhhomhepifgrvghtrghnucftihhvvghtuceoghhrihhvvgesuhdvheei rdhnvghtqeenucggtffrrghtthgvrhhnpeehgfevffekteehteefieefvdehleefjeefhe evudetjefhkeeutdekieeuvdetheenucfkphepkeeirddvheegrddvgeefrddugeeinecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 3C9D1240057; Mon, 12 Apr 2021 11:20:21 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:51 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 05/28] dpif-netdev: Rename flow offload thread X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" ovs_strlcpy silently fails to copy the thread name if it is too long. Rename the flow offload thread to differentiate it from the main thread. Fixes: 02bb2824e51d ("dpif-netdev: do hw flow offload in a thread") Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 2832671d8..ae08fa3b8 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2786,8 +2786,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, if (ovsthread_once_start(&offload_thread_once)) { xpthread_cond_init(&dp_flow_offload.cond, NULL); - ovs_thread_create("dp_netdev_flow_offload", - dp_netdev_flow_offload_main, NULL); + ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -2810,8 +2809,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, if (ovsthread_once_start(&offload_thread_once)) { xpthread_cond_init(&dp_flow_offload.cond, NULL); - ovs_thread_create("dp_netdev_flow_offload", - dp_netdev_flow_offload_main, NULL); + ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } From patchwork Mon Apr 12 15:19:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465334 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=IVmo2eCI; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=ROVB6MiK; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsr93RKzz9rx6 for ; Tue, 13 Apr 2021 01:20:41 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 82BD4404F3; Mon, 12 Apr 2021 15:20:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pEEkVUCC1kIh; Mon, 12 Apr 2021 15:20:34 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTP id 4B72940506; Mon, 12 Apr 2021 15:20:33 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 314C5C001B; Mon, 12 Apr 2021 15:20:29 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5975BC0011 for ; Mon, 12 Apr 2021 15:20:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 48845402CF for ; Mon, 12 Apr 2021 15:20:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KODtnI4sHAyf for ; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp4.osuosl.org (Postfix) with ESMTPS id 4CEC040300 for ; Mon, 12 Apr 2021 15:20:23 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 98B115C01D0; Mon, 12 Apr 2021 11:20:22 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=cCD1JFLCeu1dk T5Q5QW6qMOh34y+6WeyuQenBQIZ5fE=; b=IVmo2eCIvmE7EyHuoYXTAmtbX1Hi4 yv9tW3WO1xDruHVYJ3D48jA3Nl5GGBaa4jCULGIf5F8+YLZxHr/hekNBKjg1WNj6 1jWV/dLPjG44wDfCd6pYCvp47GCdiAAGtthg6N8OQyHQrGdXD8tGgI+ibrRj3eGH 0AhuBfb9u4MeNgsRksPhJE6Dz65184DnbTAmiI7kb4hM8Lb4WnzI0/E7DYwj/4Wv JbaT8mrgGz5OKk+Q4gwTShpWUTwU0qh39/o0gY/yCmXaEG6RfdPTVqKVKNTPJypY HsdVDDZAnvz+3RPXk0cCHdZQDYQWi3QtReoLalatp/zeOaFS3XbMvwmuQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=cCD1JFLCeu1dkT5Q5QW6qMOh34y+6WeyuQenBQIZ5fE=; b=ROVB6MiK eu8HY8KRoeJrwSm3aZ7AZf6Ek9viG4Dkax+q27slH0CnMeYsQjp+Z08fzghvJZiO ANIt4ViYJHKOd7rb0NtPhTy/nPrduTfz9BKqOl7ylN2sIAytkVJuxnFcbwRPuILa K/p/kAhDGRMyGaWB6wFEaO2WwHcFgCGm9BBknaQ3gjefUn2SnhZSuhmDiwKQr0T6 sYf2OeVAMDPL40UExL0TgGamqrghnuvFauWAkOfYFDggwNbKSra+6XnD6EnnKYqF L4DnMfVZnqOTcgHOQQqbgWFfjyaHPZUjUFZVGf/XAHqaQ718/SSfXozcNf7BCn4I MwFctf0EVHLCmg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id DF48E240066; Mon, 12 Apr 2021 11:20:21 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:52 +0200 Message-Id: <93a6afce60f6e0910c3ea10befb2b3a45bf21e21.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 06/28] dpif-netdev: Rename offload thread structure X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The offload management in userspace is done through a separate thread. The naming of the structure holding the objects used for synchronization with the dataplane is generic and nondescript. Clarify the object function by renaming it. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 52 +++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index ae08fa3b8..ebc214ec8 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -424,7 +424,7 @@ enum { DP_NETDEV_FLOW_OFFLOAD_OP_DEL, }; -struct dp_flow_offload_item { +struct dp_offload_thread_item { struct dp_netdev_pmd_thread *pmd; struct dp_netdev_flow *flow; int op; @@ -435,15 +435,15 @@ struct dp_flow_offload_item { struct ovs_list node; }; -struct dp_flow_offload { +struct dp_offload_thread { struct ovs_mutex mutex; struct ovs_list list; pthread_cond_t cond; }; -static struct dp_flow_offload dp_flow_offload = { +static struct dp_offload_thread dp_offload_thread = { .mutex = OVS_MUTEX_INITIALIZER, - .list = OVS_LIST_INITIALIZER(&dp_flow_offload.list), + .list = OVS_LIST_INITIALIZER(&dp_offload_thread.list), }; static struct ovsthread_once offload_thread_once @@ -2599,12 +2599,12 @@ mark_to_flow_find(const struct dp_netdev_pmd_thread *pmd, return NULL; } -static struct dp_flow_offload_item * +static struct dp_offload_thread_item * dp_netdev_alloc_flow_offload(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow, int op) { - struct dp_flow_offload_item *offload; + struct dp_offload_thread_item *offload; offload = xzalloc(sizeof(*offload)); offload->pmd = pmd; @@ -2618,7 +2618,7 @@ dp_netdev_alloc_flow_offload(struct dp_netdev_pmd_thread *pmd, } static void -dp_netdev_free_flow_offload(struct dp_flow_offload_item *offload) +dp_netdev_free_flow_offload(struct dp_offload_thread_item *offload) { dp_netdev_pmd_unref(offload->pmd); dp_netdev_flow_unref(offload->flow); @@ -2628,16 +2628,16 @@ dp_netdev_free_flow_offload(struct dp_flow_offload_item *offload) } static void -dp_netdev_append_flow_offload(struct dp_flow_offload_item *offload) +dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) { - ovs_mutex_lock(&dp_flow_offload.mutex); - ovs_list_push_back(&dp_flow_offload.list, &offload->node); - xpthread_cond_signal(&dp_flow_offload.cond); - ovs_mutex_unlock(&dp_flow_offload.mutex); + ovs_mutex_lock(&dp_offload_thread.mutex); + ovs_list_push_back(&dp_offload_thread.list, &offload->node); + xpthread_cond_signal(&dp_offload_thread.cond); + ovs_mutex_unlock(&dp_offload_thread.mutex); } static int -dp_netdev_flow_offload_del(struct dp_flow_offload_item *offload) +dp_netdev_flow_offload_del(struct dp_offload_thread_item *offload) { return mark_to_flow_disassociate(offload->pmd, offload->flow); } @@ -2654,7 +2654,7 @@ dp_netdev_flow_offload_del(struct dp_flow_offload_item *offload) * valid, thus only item 2 needed. */ static int -dp_netdev_flow_offload_put(struct dp_flow_offload_item *offload) +dp_netdev_flow_offload_put(struct dp_offload_thread_item *offload) { struct dp_netdev_pmd_thread *pmd = offload->pmd; struct dp_netdev_flow *flow = offload->flow; @@ -2734,22 +2734,22 @@ err_free: static void * dp_netdev_flow_offload_main(void *data OVS_UNUSED) { - struct dp_flow_offload_item *offload; + struct dp_offload_thread_item *offload; struct ovs_list *list; const char *op; int ret; for (;;) { - ovs_mutex_lock(&dp_flow_offload.mutex); - if (ovs_list_is_empty(&dp_flow_offload.list)) { + ovs_mutex_lock(&dp_offload_thread.mutex); + if (ovs_list_is_empty(&dp_offload_thread.list)) { ovsrcu_quiesce_start(); - ovs_mutex_cond_wait(&dp_flow_offload.cond, - &dp_flow_offload.mutex); + ovs_mutex_cond_wait(&dp_offload_thread.cond, + &dp_offload_thread.mutex); ovsrcu_quiesce_end(); } - list = ovs_list_pop_front(&dp_flow_offload.list); - offload = CONTAINER_OF(list, struct dp_flow_offload_item, node); - ovs_mutex_unlock(&dp_flow_offload.mutex); + list = ovs_list_pop_front(&dp_offload_thread.list); + offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); + ovs_mutex_unlock(&dp_offload_thread.mutex); switch (offload->op) { case DP_NETDEV_FLOW_OFFLOAD_OP_ADD: @@ -2782,10 +2782,10 @@ static void queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow) { - struct dp_flow_offload_item *offload; + struct dp_offload_thread_item *offload; if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_flow_offload.cond, NULL); + xpthread_cond_init(&dp_offload_thread.cond, NULL); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -2800,7 +2800,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow, struct match *match, const struct nlattr *actions, size_t actions_len) { - struct dp_flow_offload_item *offload; + struct dp_offload_thread_item *offload; int op; if (!netdev_is_flow_api_enabled()) { @@ -2808,7 +2808,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, } if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_flow_offload.cond, NULL); + xpthread_cond_init(&dp_offload_thread.cond, NULL); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } From patchwork Mon Apr 12 15:19:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465335 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=WDNP4/P7; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=haITHkE2; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrC6D9hz9rx6 for ; Tue, 13 Apr 2021 01:20:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id A52E283C61; Mon, 12 Apr 2021 15:20:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g8Zu6wcyp_78; Mon, 12 Apr 2021 15:20:39 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 1B53F83B8C; Mon, 12 Apr 2021 15:20:35 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 04A8CC0026; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id D5CFAC001D for ; Mon, 12 Apr 2021 15:20:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id B3014403A1 for ; Mon, 12 Apr 2021 15:20:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="WDNP4/P7"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="haITHkE2" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YfffRLoYJ09y for ; Mon, 12 Apr 2021 15:20:24 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp4.osuosl.org (Postfix) with ESMTPS id 62901402CF for ; Mon, 12 Apr 2021 15:20:24 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id AE24F5C01CC; Mon, 12 Apr 2021 11:20:23 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=LnCLUfD14P+SA rIFrFi7r5ieY7/RuFGCOtex0QdIMvI=; b=WDNP4/P7Ze7tweBPI3i+mwGXwTaPJ Bnh76jZ3JIkIHOQk4aWEluT9L2hB2INw9f27HdihKig4mSr1y5rryybN2V8Q4UZo blpdIbLvjU19favzFe6Cggoogx6RxScHtH+brUP9OKP5qk8Ld4pR8tJfp636J4mc M1BdMxV5lU1gIDVRElCpY3klX2nBJ2frmVTN14mN9RRRpvrqhAeQ/Pl51ydFyqxR c93v7FaVeLj3v4w6fng6PPLi03npEI85Gmh4lxAcWsluxDruBj/AzZD8vUOAtNgC 3TjgylKZVh1+1x/esx1dl51SFP8N2kSeA0/S1ZWq1TMVs/rGq90AGqgKA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=LnCLUfD14P+SArIFrFi7r5ieY7/RuFGCOtex0QdIMvI=; b=haITHkE2 FJ40ekO3QQz1ikDICQBPVMre0/JYtnZNmjcYNlgvSRsQFSgqVYropruS9+TbXN8M JY/25rEbphiJ7kIXT6hoFhFnIVDcBAVRERx+tj2CWratiYOIjFT0sOd+etZg6io8 accQQlnE9hiA/9SUhFDUK0NjqVRppgh7796HZloWbqhBXP8tlJqOawHN2Mi2YhZf NkrPLDUt702HwQYcZXABZMwEFWCkniXMCXflZrKSgxPB8N0iKsVH5cYtFtxYPucX bZkL6g7cDDNEBRlhAmnP8BE5WTSbisKUP2hoyIAzZf5lHYMrog2xfkwn0vN6QcSz aOPfjlvf3UDGfg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epleegffdttdeftdevhfeikeegveeugeehleeltdduieegvddvveelgeeugeekhffgnecu ffhomhgrihhnpegrphgrtghhvgdrohhrghdpfihikhhiphgvughirgdrohhrghenucfkph epkeeirddvheegrddvgeefrddugeeinecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepghhrihhvvgesuhdvheeirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id A3732240054; Mon, 12 Apr 2021 11:20:22 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:53 +0200 Message-Id: <292e978bee76f01148e0e91027d06f0a8efed656.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 07/28] mov-avg: Add a moving average helper structure X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a new module offering a helper to compute the Cumulative Moving Average (CMA) and the Exponential Moving Average (EMA) of a series of values. Use the new helpers to add latency metrics in dpif-netdev. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/automake.mk | 1 + lib/mov-avg.h | 171 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 172 insertions(+) create mode 100644 lib/mov-avg.h diff --git a/lib/automake.mk b/lib/automake.mk index 39901bd6d..111179736 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -166,6 +166,7 @@ lib_libopenvswitch_la_SOURCES = \ lib/memory.c \ lib/memory.h \ lib/meta-flow.c \ + lib/mov-avg.h \ lib/multipath.c \ lib/multipath.h \ lib/namemap.c \ diff --git a/lib/mov-avg.h b/lib/mov-avg.h new file mode 100644 index 000000000..4a7e62c18 --- /dev/null +++ b/lib/mov-avg.h @@ -0,0 +1,171 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef _MOV_AVG_H +#define _MOV_AVG_H 1 + +#include + +/* Moving average helpers. */ + +/* Cumulative Moving Average. + * + * Computes the arithmetic mean over a whole series of value. + * Online equivalent of sum(V) / len(V). + * + * As all values have equal weight, this average will + * be slow to show recent changes in the series. + * + */ + +struct mov_avg_cma { + unsigned long long int count; + double mean; + double sum_dsquared; +}; + +#define MOV_AVG_CMA_INITIALIZER \ + { .count = 0, .mean = .0, .sum_dsquared = .0 } + +static inline void +mov_avg_cma_init(struct mov_avg_cma *cma) +{ + *cma = (struct mov_avg_cma) MOV_AVG_CMA_INITIALIZER; +} + +static inline void +mov_avg_cma_update(struct mov_avg_cma *cma, double new_val) +{ + double new_mean; + + cma->count++; + new_mean = cma->mean + (new_val - cma->mean) / cma->count; + + cma->sum_dsquared += (new_val - new_mean) * (new_val - cma->mean); + cma->mean = new_mean; +} + +static inline double +mov_avg_cma(struct mov_avg_cma *cma) +{ + return cma->mean; +} + +static inline double +mov_avg_cma_std_dev(struct mov_avg_cma *cma) +{ + double variance = 0.0; + + if (cma->count > 1) { + variance = cma->sum_dsquared / (cma->count - 1); + } + + return sqrt(variance); +} + +/* Exponential Moving Average. + * + * Each value in the series has an exponentially decreasing weight, + * the older they get the less weight they have. + * + * The smoothing factor 'alpha' must be within 0 < alpha < 1. + * The closer this factor to zero, the more equal the weight between + * recent and older values. As it approaches one, the more recent values + * will have more weight. + * + * The EMA can be thought of as an estimator for the next value when measures + * are dependent. In this case, it can make sense to consider the mean square + * error of the prediction. An 'alpha' minimizing this error would be the + * better choice to improve the estimation. + * + * A common way to choose 'alpha' is to use the following formula: + * + * a = 2 / (N + 1) + * + * With this 'alpha', the EMA will have the same 'center of mass' as an + * equivalent N-values Simple Moving Average. + * + * When using this factor, the N last values of the EMA will have a sum weight + * converging toward 0.8647, meaning that those values will account for 86% of + * the average[1]. + * + * [1] https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + */ + +struct mov_avg_ema { + double alpha; /* 'Smoothing' factor. */ + double mean; + double variance; + bool initialized; +}; + +/* Choose alpha explicitly. */ +#define MOV_AVG_EMA_INITIALIZER_ALPHA(a) { \ + .initialized = false, \ + .alpha = (a), .variance = 0.0, .mean = 0.0 \ +} + +/* Choose alpha to consider 'N' past periods as 86% of the EMA. */ +#define MOV_AVG_EMA_INITIALIZER(n_elem) \ + MOV_AVG_EMA_INITIALIZER_ALPHA(2.0 / ((double)(n_elem) + 1.0)) + +static inline void +mov_avg_ema_init_alpha(struct mov_avg_ema *ema, + double alpha) +{ + *ema = (struct mov_avg_ema) MOV_AVG_EMA_INITIALIZER_ALPHA(alpha); +} + +static inline void +mov_avg_ema_init(struct mov_avg_ema *ema, + unsigned long long int n_elem) +{ + *ema = (struct mov_avg_ema) MOV_AVG_EMA_INITIALIZER(n_elem); +} + +static inline void +mov_avg_ema_update(struct mov_avg_ema *ema, double new_val) +{ + const double alpha = ema->alpha; + double alpha_diff; + double diff; + + if (!ema->initialized) { + ema->initialized = true; + ema->mean = new_val; + return; + } + + diff = new_val - ema->mean; + alpha_diff = alpha * diff; + + ema->variance = (1.0 - alpha) * (ema->variance + alpha_diff * diff); + ema->mean = ema->mean + alpha_diff; +} + +static inline double +mov_avg_ema(struct mov_avg_ema *ema) +{ + return ema->mean; +} + +static inline double +mov_avg_ema_std_dev(struct mov_avg_ema *ema) +{ + return sqrt(ema->variance); +} + +#endif /* _MOV_AVG_H */ From patchwork Mon Apr 12 15:19:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465340 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=aD3uB+fJ; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=IWgi9wXR; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrQ3Bl6z9sTD for ; Tue, 13 Apr 2021 01:20:54 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C35F683CD3; Mon, 12 Apr 2021 15:20:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UR9xUx11_0YL; Mon, 12 Apr 2021 15:20:48 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTP id EFE0983C4E; Mon, 12 Apr 2021 15:20:39 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 03811C0020; Mon, 12 Apr 2021 15:20:33 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 176E8C0029 for ; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id E575460AB8 for ; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="aD3uB+fJ"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="IWgi9wXR" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y7o7xxVVECcf for ; Mon, 12 Apr 2021 15:20:25 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4F92A60908 for ; Mon, 12 Apr 2021 15:20:25 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 76E385C01C8; Mon, 12 Apr 2021 11:20:24 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=aHUJ7LASCiVC9 lHimMENwi38tDSXz2mnSwqWGaf1Swk=; b=aD3uB+fJy1Gbquo1XHiNBOLncVcG1 EXVYyIZtP//t4XLaeCXSoO3xuaCqjFJ/pkgLvmikmx3WJgEUn3qFnqmtiojcInwU TrEWmHnEAvNONiuyUbvTfT4swyHMM3tmhJJEf3qOWbig1Fi+2CtrtLlcK7Gxpvwn Nrx+FPNYqSksy0GHapMyhjJH4Cq5CZ1T0mqHp5hidShCRCrrh8DgOoztzTMsw7wk Xg8Hr/dMFw11pw/d+lJ9A4LmK40qZoGj488ZuTh9BoR1Q6dG53hAfTICx3qTyiuW 5DnhVxySMgi2ShDvmsruGHLCP79Psr4Aa9f4FHyfQxv8XvNPie5AxC18Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=aHUJ7LASCiVC9lHimMENwi38tDSXz2mnSwqWGaf1Swk=; b=IWgi9wXR YkTMZOrXOxR7FUfw5PTVApDXZ3upxlWQaRzQn0YZBTDIKlRcn8qWETzldvwUxF6W CQ9FH8XjnfZC5VPM159Lvz9tQCKaAqwDTWvEdidTO2qGLWblGwFTaVEUreJgT0Tq 7blhVxvR0j3ARNowAAVw7c7Mx53GIApLycCCvYYaxrGCa2KVV+oGNZ4KC7HXBcEn eSrASRYiVsgHvxPbPynODx7duz/7dPewpJDhIoPjCSToUXIf2AdAhC/Di1rZQ5+Z aIuiL9LboSa4fQ3oSf63dQ/FBo5VZz63ofynEMYYeC30s/TSTu6CGZNbFidrA/eF kLKdzwVQQO46Uw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id B77F2240054; Mon, 12 Apr 2021 11:20:23 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:54 +0200 Message-Id: <5c3ebd512bc3aa81ebbbcfbfed9429b51f7e9a09.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 08/28] dpif-netdev: Implement hardware offloads stats query X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In the netdev datapath, keep track of the enqueued offloads between the PMDs and the offload thread. Additionally, query each netdev for their hardware offload counters. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 89 insertions(+), 1 deletion(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index ebc214ec8..d458bcb12 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -51,6 +51,7 @@ #include "hmapx.h" #include "id-pool.h" #include "ipf.h" +#include "mov-avg.h" #include "netdev.h" #include "netdev-offload.h" #include "netdev-provider.h" @@ -431,6 +432,7 @@ struct dp_offload_thread_item { struct match match; struct nlattr *actions; size_t actions_len; + long long int timestamp; struct ovs_list node; }; @@ -438,12 +440,18 @@ struct dp_offload_thread_item { struct dp_offload_thread { struct ovs_mutex mutex; struct ovs_list list; + uint64_t enqueued_item; + struct mov_avg_cma cma; + struct mov_avg_ema ema; pthread_cond_t cond; }; static struct dp_offload_thread dp_offload_thread = { .mutex = OVS_MUTEX_INITIALIZER, .list = OVS_LIST_INITIALIZER(&dp_offload_thread.list), + .enqueued_item = 0, + .cma = MOV_AVG_CMA_INITIALIZER, + .ema = MOV_AVG_EMA_INITIALIZER(100), }; static struct ovsthread_once offload_thread_once @@ -2632,6 +2640,7 @@ dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) { ovs_mutex_lock(&dp_offload_thread.mutex); ovs_list_push_back(&dp_offload_thread.list, &offload->node); + dp_offload_thread.enqueued_item++; xpthread_cond_signal(&dp_offload_thread.cond); ovs_mutex_unlock(&dp_offload_thread.mutex); } @@ -2736,6 +2745,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) { struct dp_offload_thread_item *offload; struct ovs_list *list; + long long int latency_us; const char *op; int ret; @@ -2748,6 +2758,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) ovsrcu_quiesce_end(); } list = ovs_list_pop_front(&dp_offload_thread.list); + dp_offload_thread.enqueued_item--; offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); ovs_mutex_unlock(&dp_offload_thread.mutex); @@ -2768,6 +2779,10 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) OVS_NOT_REACHED(); } + latency_us = time_usec() - offload->timestamp; + mov_avg_cma_update(&dp_offload_thread.cma, latency_us); + mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + VLOG_DBG("%s to %s netdev flow "UUID_FMT, ret == 0 ? "succeed" : "failed", op, UUID_ARGS((struct uuid *) &offload->flow->mega_ufid)); @@ -2792,6 +2807,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, offload = dp_netdev_alloc_flow_offload(pmd, flow, DP_NETDEV_FLOW_OFFLOAD_OP_DEL); + offload->timestamp = pmd->ctx.now; dp_netdev_append_flow_offload(offload); } @@ -2824,6 +2840,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, memcpy(offload->actions, actions, actions_len); offload->actions_len = actions_len; + offload->timestamp = pmd->ctx.now; dp_netdev_append_flow_offload(offload); } @@ -4209,6 +4226,77 @@ dpif_netdev_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, } } +static int +dpif_netdev_offload_stats_get(struct dpif *dpif, + struct netdev_custom_stats *stats) +{ + enum { + DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED, + DP_NETDEV_HW_OFFLOADS_STATS_INSERTED, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV, + }; + const char *names[] = { + [DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED] = + " Enqueued offloads", + [DP_NETDEV_HW_OFFLOADS_STATS_INSERTED] = + " Inserted offloads", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN] = + " Cumulative Average latency (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV] = + " Cumulative Latency stddev (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN] = + " Exponential Average latency (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV] = + " Exponential Latency stddev (us)", + }; + struct dp_netdev *dp = get_dp_netdev(dpif); + struct dp_netdev_port *port; + uint64_t nb_offloads; + size_t i; + + if (!netdev_is_flow_api_enabled()) { + return EINVAL; + } + + stats->size = ARRAY_SIZE(names); + stats->counters = xcalloc(stats->size, sizeof *stats->counters); + + nb_offloads = 0; + + ovs_mutex_lock(&dp->port_mutex); + HMAP_FOR_EACH (port, node, &dp->ports) { + uint64_t port_nb_offloads = 0; + + /* Do not abort on read error from a port, just report 0. */ + if (!netdev_flow_get_n_flows(port->netdev, &port_nb_offloads)) { + nb_offloads += port_nb_offloads; + } + } + ovs_mutex_unlock(&dp->port_mutex); + + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value = + dp_offload_thread.enqueued_item; + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED].value = nb_offloads; + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].value = + mov_avg_cma(&dp_offload_thread.cma); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV].value = + mov_avg_cma_std_dev(&dp_offload_thread.cma); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN].value = + mov_avg_ema(&dp_offload_thread.ema); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV].value = + mov_avg_ema_std_dev(&dp_offload_thread.ema); + + for (i = 0; i < ARRAY_SIZE(names); i++) { + snprintf(stats->counters[i].name, sizeof(stats->counters[i].name), + "%s", names[i]); + } + + return 0; +} + /* Enable or Disable PMD auto load balancing. */ static void set_pmd_auto_lb(struct dp_netdev *dp, bool always_log) @@ -8484,7 +8572,7 @@ const struct dpif_class dpif_netdev_class = { dpif_netdev_flow_dump_thread_destroy, dpif_netdev_flow_dump_next, dpif_netdev_operate, - NULL, /* offload_stats_get */ + dpif_netdev_offload_stats_get, NULL, /* recv_set */ NULL, /* handlers_set */ dpif_netdev_set_config, From patchwork Mon Apr 12 15:19:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465337 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=hPZA8ln8; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=oE8Z5MM6; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrJ2xhtz9rx6 for ; Tue, 13 Apr 2021 01:20:48 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 927F283C9A; Mon, 12 Apr 2021 15:20:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YKWAKYw4DOHb; Mon, 12 Apr 2021 15:20:43 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 933AD83C0F; Mon, 12 Apr 2021 15:20:36 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 07AE9C0027; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4E95DC001E for ; Mon, 12 Apr 2021 15:20:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id ED08E60AA9 for ; Mon, 12 Apr 2021 15:20:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="hPZA8ln8"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="oE8Z5MM6" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id racWjVUBwpDl for ; Mon, 12 Apr 2021 15:20:26 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp3.osuosl.org (Postfix) with ESMTPS id 012BF60A95 for ; Mon, 12 Apr 2021 15:20:25 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 3E8665C01CC; Mon, 12 Apr 2021 11:20:25 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=Yvl4r9kQh64HU ASVvgT1bAiJYkfZycRowrVksd8jWC8=; b=hPZA8ln8+r4tca+gsCvbS2EKvkd2c 7AcQ/XZ4jIYOHGceRS6ctehtb14NC/B9MKHIIa62Z6P+jcrHN9wpBbkM7gKNuj2w n4HPoxd2WnQNvPXZSkCWip6ThCU4+ClWWsxM0b4dN28dsYkF387mHUNzxVuw/YUZ Ra1n7NSQKA2+743P9ZUMWglqBjGRq3V3QDuwJ74g0jAyQh0KdRLLF87aqjUEcIP0 LCD7nLkULbhgXkHexKgrol+xsEh1ECle/no3qy1MD7JgvYht+EvIX/3x/k+eZBqY 8iIaVJjfSQZNkxWH3twMC9oZOEgtZxHB62LVZpMi60yIFLUq0QnxY1GCw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=Yvl4r9kQh64HUASVvgT1bAiJYkfZycRowrVksd8jWC8=; b=oE8Z5MM6 wzdSgPIGjP2Azr2FWweZR5w0EjN3C0ujPM6R3ErTphwidmDDSNmI0CstPmTEhLuY Iswd+Svww4zL5PdNkpbUg7VsMbOTAXMi1VejXxREIzsxexoh469oKrVWYg+JBLzD 1KmvH8KYqkLt+kdZ4tS0X/o3qQ8KnCxe1IsQlKac5kNBz3HhM0sjsNHMEERMjZVF duowQM5bD33mtINnZdFBmlbXZb+SHNsUXswJQVDbGRspBfS31jO/IGKkUDp9ehB3 E/oIVKHSLyi/n+ZrX2gf3b4RkRRhZ1PbodwFaOBNm2LqUm4uZ30OyYcFYA8bAaO8 HTWgfUtpVW51SA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 84B27240054; Mon, 12 Apr 2021 11:20:24 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:55 +0200 Message-Id: <5594928ae3a162b7f899ebc1772e54183523536e.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 09/28] ovs-atomic: Expose atomic exchange operation X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The atomic exchange operation is a useful primitive that should be available as well. Most compiler already expose or offer a way to use it, but a single symbol needs to be defined. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein --- lib/ovs-atomic-c++.h | 3 +++ lib/ovs-atomic-clang.h | 5 +++++ lib/ovs-atomic-gcc4+.h | 5 +++++ lib/ovs-atomic-gcc4.7+.h | 5 +++++ lib/ovs-atomic-i586.h | 5 +++++ lib/ovs-atomic-locked.h | 9 +++++++++ lib/ovs-atomic-msvc.h | 22 ++++++++++++++++++++++ lib/ovs-atomic-pthreads.h | 5 +++++ lib/ovs-atomic-x86_64.h | 5 +++++ lib/ovs-atomic.h | 8 +++++++- 10 files changed, 71 insertions(+), 1 deletion(-) diff --git a/lib/ovs-atomic-c++.h b/lib/ovs-atomic-c++.h index d47b8dd39..8605fa9d3 100644 --- a/lib/ovs-atomic-c++.h +++ b/lib/ovs-atomic-c++.h @@ -29,6 +29,9 @@ using std::atomic_compare_exchange_strong_explicit; using std::atomic_compare_exchange_weak; using std::atomic_compare_exchange_weak_explicit; +using std::atomic_exchange; +using std::atomic_exchange_explicit; + #define atomic_read(SRC, DST) \ atomic_read_explicit(SRC, DST, memory_order_seq_cst) #define atomic_read_explicit(SRC, DST, ORDER) \ diff --git a/lib/ovs-atomic-clang.h b/lib/ovs-atomic-clang.h index 34cc2faa7..cdf02a512 100644 --- a/lib/ovs-atomic-clang.h +++ b/lib/ovs-atomic-clang.h @@ -67,6 +67,11 @@ typedef enum { #define atomic_compare_exchange_weak_explicit(DST, EXP, SRC, ORD1, ORD2) \ __c11_atomic_compare_exchange_weak(DST, EXP, SRC, ORD1, ORD2) +#define atomic_exchange(RMW, ARG) \ + atomic_exchange_explicit(RMW, ARG, memory_order_seq_cst) +#define atomic_exchange_explicit(RMW, ARG, ORDER) \ + __c11_atomic_exchange(RMW, ARG, ORDER) + #define atomic_add(RMW, ARG, ORIG) \ atomic_add_explicit(RMW, ARG, ORIG, memory_order_seq_cst) #define atomic_sub(RMW, ARG, ORIG) \ diff --git a/lib/ovs-atomic-gcc4+.h b/lib/ovs-atomic-gcc4+.h index 25bcf20a0..f9accde1a 100644 --- a/lib/ovs-atomic-gcc4+.h +++ b/lib/ovs-atomic-gcc4+.h @@ -128,6 +128,11 @@ atomic_signal_fence(memory_order order) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +#define atomic_exchange_explicit(DST, SRC, ORDER) \ + __sync_lock_test_and_set(DST, SRC) +#define atomic_exchange(DST, SRC) \ + atomic_exchange_explicit(DST, SRC, memory_order_seq_cst) + #define atomic_op__(RMW, OP, ARG, ORIG) \ ({ \ typeof(RMW) rmw__ = (RMW); \ diff --git a/lib/ovs-atomic-gcc4.7+.h b/lib/ovs-atomic-gcc4.7+.h index 4c197ebe0..846e05775 100644 --- a/lib/ovs-atomic-gcc4.7+.h +++ b/lib/ovs-atomic-gcc4.7+.h @@ -61,6 +61,11 @@ typedef enum { #define atomic_compare_exchange_weak_explicit(DST, EXP, SRC, ORD1, ORD2) \ __atomic_compare_exchange_n(DST, EXP, SRC, true, ORD1, ORD2) +#define atomic_exchange_explicit(DST, SRC, ORDER) \ + __atomic_exchange_n(DST, SRC, ORDER) +#define atomic_exchange(DST, SRC) \ + atomic_exchange_explicit(DST, SRC, memory_order_seq_cst) + #define atomic_add(RMW, OPERAND, ORIG) \ atomic_add_explicit(RMW, OPERAND, ORIG, memory_order_seq_cst) #define atomic_sub(RMW, OPERAND, ORIG) \ diff --git a/lib/ovs-atomic-i586.h b/lib/ovs-atomic-i586.h index 9a385ce84..35a0959ff 100644 --- a/lib/ovs-atomic-i586.h +++ b/lib/ovs-atomic-i586.h @@ -400,6 +400,11 @@ atomic_signal_fence(memory_order order) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +#define atomic_exchange_explicit(RMW, ARG, ORDER) \ + atomic_exchange__(RMW, ARG, ORDER) +#define atomic_exchange(RMW, ARG) \ + atomic_exchange_explicit(RMW, ARG, memory_order_seq_cst) + #define atomic_add__(RMW, ARG, CLOB) \ asm volatile("lock; xadd %0,%1 ; " \ "# atomic_add__ " \ diff --git a/lib/ovs-atomic-locked.h b/lib/ovs-atomic-locked.h index f8f0ba2a5..bf38c4a43 100644 --- a/lib/ovs-atomic-locked.h +++ b/lib/ovs-atomic-locked.h @@ -31,6 +31,15 @@ void atomic_unlock__(void *); atomic_unlock__(DST), \ false))) +#define atomic_exchange_locked(DST, SRC) \ + ({ \ + atomic_lock__(DST); \ + typeof(*(DST)) __tmp = *(DST); \ + *(DST) = SRC; \ + atomic_unlock__(DST); \ + __tmp; \ + }) + #define atomic_op_locked_add += #define atomic_op_locked_sub -= #define atomic_op_locked_or |= diff --git a/lib/ovs-atomic-msvc.h b/lib/ovs-atomic-msvc.h index 9def887d3..ef8310269 100644 --- a/lib/ovs-atomic-msvc.h +++ b/lib/ovs-atomic-msvc.h @@ -345,6 +345,28 @@ atomic_signal_fence(memory_order order) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +/* While intrinsics offering different memory ordering + * are available in MSVC C compiler, they are not defined + * in the C++ compiler. Ignore for compatibility. + * + * Use nested ternary operators as the GNU extension ({}) + * is not available. + */ + +#define atomic_exchange_explicit(DST, SRC, ORDER) \ + ((sizeof *(DST) == 1) ? \ + _InterlockedExchange8((char volatile *) DST, SRC) \ + : (sizeof *(DST) == 2) ? \ + _InterlockedExchange16((short volatile *) DST, SRC) \ + : (sizeof *(DST) == 4) ? \ + _InterlockedExchange((long int volatile *) DST, SRC) \ + : (sizeof *(DST) == 8) ? \ + _InterlockedExchange64((__int64 volatile *) DST, SRC) \ + : (ovs_abort(), 0)) + +#define atomic_exchange(DST, SRC) \ + atomic_exchange_explicit(DST, SRC, memory_order_seq_cst) + /* MSVCs c++ compiler implements c11 atomics and looking through its * implementation (in xatomic.h), orders are ignored for x86 platform. * Do the same here. */ diff --git a/lib/ovs-atomic-pthreads.h b/lib/ovs-atomic-pthreads.h index 12234e79e..570a67fe4 100644 --- a/lib/ovs-atomic-pthreads.h +++ b/lib/ovs-atomic-pthreads.h @@ -77,6 +77,11 @@ atomic_signal_fence(memory_order order OVS_UNUSED) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +#define atomic_exchange(DST, SRC) \ + atomic_exchange_locked(DST, SRC) +#define atomic_exchange_explicit(DST, SRC, ORDER) \ + ((void) (ORDER), atomic_exchange(DST, SRC)) + #define atomic_add(RMW, ARG, ORIG) atomic_op_locked(RMW, add, ARG, ORIG) #define atomic_sub(RMW, ARG, ORIG) atomic_op_locked(RMW, sub, ARG, ORIG) #define atomic_or( RMW, ARG, ORIG) atomic_op_locked(RMW, or, ARG, ORIG) diff --git a/lib/ovs-atomic-x86_64.h b/lib/ovs-atomic-x86_64.h index 1e7d42707..3bdaf2f08 100644 --- a/lib/ovs-atomic-x86_64.h +++ b/lib/ovs-atomic-x86_64.h @@ -274,6 +274,11 @@ atomic_signal_fence(memory_order order) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +#define atomic_exchange_explicit(RMW, ARG, ORDER) \ + atomic_exchange__(RMW, ARG, ORDER) +#define atomic_exchange(RMW, ARG) \ + atomic_exchange_explicit(RMW, ARG, memory_order_seq_cst) + #define atomic_add__(RMW, ARG, CLOB) \ asm volatile("lock; xadd %0,%1 ; " \ "# atomic_add__ " \ diff --git a/lib/ovs-atomic.h b/lib/ovs-atomic.h index 11fa19268..8fdce0cf8 100644 --- a/lib/ovs-atomic.h +++ b/lib/ovs-atomic.h @@ -210,7 +210,7 @@ * In this section, A is an atomic type and C is the corresponding non-atomic * type. * - * The "store" and "compare_exchange" primitives match C11: + * The "store", "exchange", and "compare_exchange" primitives match C11: * * void atomic_store(A *object, C value); * void atomic_store_explicit(A *object, C value, memory_order); @@ -244,6 +244,12 @@ * efficiently, so it should be used if the application will need to * loop anyway. * + * C atomic_exchange(A *object, C desired); + * C atomic_exchange_explicit(A *object, C desired, memory_order); + * + * Atomically stores 'desired' into '*object', returning the value + * previously held. + * * The following primitives differ from the C11 ones (and have different names) * because there does not appear to be a way to implement the standard * primitives in standard C: From patchwork Mon Apr 12 15:19:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465344 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=Hdssqmh4; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=Gg6Es9Va; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrw50dYz9rx6 for ; Tue, 13 Apr 2021 01:21:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 1345183DD3; Mon, 12 Apr 2021 15:21:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q9UTCWCo-VAW; Mon, 12 Apr 2021 15:21:12 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 7E8A683BB7; Mon, 12 Apr 2021 15:20:49 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0D43EC0049; Mon, 12 Apr 2021 15:20:39 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 199E7C002F for ; Mon, 12 Apr 2021 15:20:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 7C958404F0 for ; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="Hdssqmh4"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="Gg6Es9Va" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eUnnpoxEa19g for ; Mon, 12 Apr 2021 15:20:27 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp4.osuosl.org (Postfix) with ESMTPS id F3166404A9 for ; Mon, 12 Apr 2021 15:20:26 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 1E62A5C01AF; Mon, 12 Apr 2021 11:20:26 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=E75HxnpSIzh8c sL2QIVWHdG/uo+08b1Bhou/czsl4jQ=; b=Hdssqmh4kyja4IEgy0hlB6t2e/u/Y o58pnL04t5MFCBNyo1JVpfLPSeH4GLGcO2MPhG77uT+l/4s2Va4oDxchcAcL6xVT Qq7Ku0mM1omRgEQhHTJhQRaTxmBTnM99/0m4O7MhlVjGKmyEeM58DHZY0VTagSRF 6wYIw2Ya/mYMXMAROpOZJVzznLJe74QlPVdOIrX89RoeYNrYHibg0qjg8OujU2Am ZEea83wWn2JVtxM6qcEJxQuAlRgsq97UxMzjAwxCPnA3Drh49BIF7SD0Orf0YX4z POkKpmpCbn4jM3lHqZ/BfdWSaJq/T/9dmakMBYrMjOnrXyUMQtbBeiZBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=E75HxnpSIzh8csL2QIVWHdG/uo+08b1Bhou/czsl4jQ=; b=Gg6Es9Va hJnd1ItF5o3L+/PVWuytXfIbtC1u7RP3xk7stNxgoMKUAQbz3/BiVjneai8KZrO7 H0usPUip3WjAqwR1hDL3v9l7ylxeUZdl2S/uzLblaKXAujsnWwUQlsMb+stjOCxX Do04iC/QOe8r4qz8r0Ej59oJXbevlE706Y4NWMNt5d8kJCnULrozCb6j4V4ibpLO hG/F0GC3OOTUX5gmk7pLqboXkxgmI8++orbWI/5o9RGk0g4ie0mw8+jY9aw4EttE upHL9P7cGs+18EhuitX9ebqCRwrrcSklRGcHwzcrv/9PQBOAiR52HJyMOJ5Lhjyj jksAIiiTpYU7qA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epuefhffetffdtffdtheehteehiedvkeeufeehvdffteetvdegudduffdutdejhefgnecu ffhomhgrihhnpedutddvgegtohhrvghsrdhnvghtpdhrohgthhgvshhtvghrrdgvughupd grphgrtghhvgdrohhrghenucfkphepkeeirddvheegrddvgeefrddugeeinecuvehluhhs thgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvgesuhdvhe eirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 4AF51240054; Mon, 12 Apr 2021 11:20:25 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:56 +0200 Message-Id: <4b9ee9ea994d6fe08e5c9effe71dd8aa5ceb6456.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 10/28] mpsc-queue: Module for lock-free message passing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a lockless multi-producer/single-consumer (MPSC), linked-list based, intrusive, unbounded queue that does not require deferred memory management. The queue is an implementation of the structure described by Dmitri Vyukov[1]. It adds a slightly more explicit API explaining the proper use of the queue. Alternatives were considered such as a Treiber Stack [2] or a Michael-Scott queue [3], but this one is faster, simpler and scalable. [1]: http://www.1024cores.net/home/lock-free-algorithms/queues/intrusive-mpsc-node-based-queue [2]: R. K. Treiber. Systems programming: Coping with parallelism. Technical Report RJ 5118, IBM Almaden Research Center, April 1986. [3]: M. M. Michael, Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms. https://www.cs.rochester.edu/research/synchronization/pseudocode/queues.html The queue is designed to improve the specific MPSC setup. A benchmark accompanies the unit tests to measure the difference in this configuration. A single reader thread polls the queue while N writers enqueue elements as fast as possible. The mpsc-queue is compared against the regular ovs-list as well as the guarded list. The latter usually offers a slight improvement by batching the element removal, however the mpsc-queue is faster. The average is of each producer threads time: $ ./tests/ovstest test-mpsc-queue benchmark 3000000 1 Benchmarking n=3000000 on 1 + 1 threads. type\thread: Reader 1 Avg mpsc-queue: 161 161 161 ms list: 803 803 803 ms guarded list: 665 665 665 ms $ ./tests/ovstest test-mpsc-queue benchmark 3000000 2 Benchmarking n=3000000 on 1 + 2 threads. type\thread: Reader 1 2 Avg mpsc-queue: 102 101 97 99 ms list: 246 212 246 229 ms guarded list: 264 263 214 238 ms $ ./tests/ovstest test-mpsc-queue benchmark 3000000 3 Benchmarking n=3000000 on 1 + 3 threads. type\thread: Reader 1 2 3 Avg mpsc-queue: 92 91 92 91 91 ms list: 520 517 515 520 517 ms guarded list: 405 395 401 404 400 ms $ ./tests/ovstest test-mpsc-queue benchmark 3000000 4 Benchmarking n=3000000 on 1 + 4 threads. type\thread: Reader 1 2 3 4 Avg mpsc-queue: 77 73 73 77 75 74 ms list: 371 359 361 287 370 344 ms guarded list: 389 388 359 363 357 366 ms Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/automake.mk | 2 + lib/mpsc-queue.c | 251 ++++++++++++++ lib/mpsc-queue.h | 189 +++++++++++ tests/automake.mk | 1 + tests/library.at | 5 + tests/test-mpsc-queue.c | 727 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 1175 insertions(+) create mode 100644 lib/mpsc-queue.c create mode 100644 lib/mpsc-queue.h create mode 100644 tests/test-mpsc-queue.c diff --git a/lib/automake.mk b/lib/automake.mk index 111179736..b45801852 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -167,6 +167,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/memory.h \ lib/meta-flow.c \ lib/mov-avg.h \ + lib/mpsc-queue.c \ + lib/mpsc-queue.h \ lib/multipath.c \ lib/multipath.h \ lib/namemap.c \ diff --git a/lib/mpsc-queue.c b/lib/mpsc-queue.c new file mode 100644 index 000000000..cd26ed207 --- /dev/null +++ b/lib/mpsc-queue.c @@ -0,0 +1,251 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "ovs-atomic.h" + +#include "mpsc-queue.h" + +/* Multi-producer, single-consumer queue + * ===================================== + * + * This an implementation of the MPSC queue described by Dmitri Vyukov [1]. + * + * One atomic exchange operation is done per insertion. Removal in most cases + * will not require atomic operation and will use one atomic exchange to close + * the queue chain. + * + * Insertion + * ========= + * + * The queue is implemented using a linked-list. Insertion is done at the + * back of the queue, by swapping the current end with the new node atomically, + * then pointing the previous end toward the new node. To follow Vyukov + * nomenclature, the end-node of the chain is called head. A producer will + * only manipulate the head. + * + * The head swap is atomic, however the link from the previous head to the new + * one is done in a separate operation. This means that the chain is + * momentarily broken, when the previous head still points to NULL and the + * current head has been inserted. + * + * Considering a series of insertions, the queue state will remain consistent + * and the insertions order is compatible with their precedence, thus the + * queue is serializable. However, because an insertion consists in two + * separate memory transactions, it is not linearizable. + * + * Removal + * ======= + * + * The consumer must deal with the queue inconsistency. It will manipulate + * the tail of the queue and move it along the latest consumed elements. + * When an end of the chain of elements is found (the next pointer is NULL), + * the tail is compared with the head. + * + * If both points to different addresses, then the queue is in an inconsistent + * state: the tail cannot move forward as the next is NULL, but the head is not + * the last element in the chain: this can only happen if the chain is broken. + * + * In this case, the consumer must wait for the producer to finish writing the + * next pointer of its current tail: 'MPSC_QUEUE_RETRY' is returned. + * + * Removal is thus in most cases (when there are elements in the queue) + * accomplished without using atomics, until the last element of the queue. + * There, the head is atomically loaded. If the queue is in a consistent state, + * the head is moved back to the queue stub by inserting the stub in the queue: + * ending the queue is the same as an insertion, which is one atomic XCHG. + * + * Forward guarantees + * ================== + * + * Insertion and peeking are wait-free: they will execute in a known bounded + * number of instructions, regardless of the state of the queue. + * + * However, while removal consists in peeking and a constant write to + * update the tail, it can repeatedly fail until the queue become consistent. + * It is thus dependent on other threads progressing. This means that the + * queue forward progress is obstruction-free only. It has a potential for + * livelocking. + * + * The chain will remain broken as long as a producer is not finished writing + * its next pointer. If a producer is cancelled for example, the queue could + * remain broken for any future readings. This queue should either be used + * with cooperative threads or insertion must only be done outside cancellable + * sections. + * + * Performances + * ============ + * + * In benchmarks this structure was better than alternatives such as: + * + * * A reversed Treiber stack [2], using 1 CAS per operations + * and requiring reversal of the node list on removal. + * + * * Michael-Scott lock-free queue [3], using 2 CAS per operations. + * + * While it is not linearizable, this queue is well-suited for message passing. + * If a proper hardware XCHG operation is used, it scales better than + * CAS-based implementations. + * + * References + * ========== + * + * [1]: http://www.1024cores.net/home/lock-free-algorithms/queues/intrusive-mpsc-node-based-queue + * + * [2]: R. K. Treiber. Systems programming: Coping with parallelism. + * Technical Report RJ 5118, IBM Almaden Research Center, April 1986. + * + * [3]: M. M. Michael, Simple, Fast, and Practical Non-Blocking and + * Blocking Concurrent Queue Algorithms + * [3]: https://www.cs.rochester.edu/research/synchronization/pseudocode/queues.html + * + */ + +void +mpsc_queue_init(struct mpsc_queue *queue) +{ + atomic_store_relaxed(&queue->head, &queue->stub); + atomic_store_relaxed(&queue->tail, &queue->stub); + atomic_store_relaxed(&queue->stub.next, NULL); + + ovs_mutex_init(&queue->read_lock); +} + +void +mpsc_queue_destroy(struct mpsc_queue *queue) + OVS_EXCLUDED(queue->read_lock) +{ + ovs_mutex_destroy(&queue->read_lock); +} + +enum mpsc_queue_poll_result +mpsc_queue_poll(struct mpsc_queue *queue, struct mpsc_queue_node **node) + OVS_REQUIRES(queue->read_lock) +{ + struct mpsc_queue_node *tail; + struct mpsc_queue_node *next; + struct mpsc_queue_node *head; + + atomic_read_relaxed(&queue->tail, &tail); + atomic_read_explicit(&tail->next, &next, memory_order_acquire); + + if (tail == &queue->stub) { + if (next == NULL) { + return MPSC_QUEUE_EMPTY; + } + + atomic_store_relaxed(&queue->tail, next); + tail = next; + atomic_read_explicit(&tail->next, &next, memory_order_acquire); + } + + if (next != NULL) { + atomic_store_relaxed(&queue->tail, next); + *node = tail; + return MPSC_QUEUE_ITEM; + } + + atomic_read_explicit(&queue->head, &head, memory_order_acquire); + if (tail != head) { + return MPSC_QUEUE_RETRY; + } + + mpsc_queue_insert(queue, &queue->stub); + + atomic_read_explicit(&tail->next, &next, memory_order_acquire); + if (next != NULL) { + atomic_store_relaxed(&queue->tail, next); + *node = tail; + return MPSC_QUEUE_ITEM; + } + + return MPSC_QUEUE_EMPTY; +} + +struct mpsc_queue_node * +mpsc_queue_pop(struct mpsc_queue *queue) + OVS_REQUIRES(queue->read_lock) +{ + enum mpsc_queue_poll_result result; + struct mpsc_queue_node *node; + + do { + result = mpsc_queue_poll(queue, &node); + if (result == MPSC_QUEUE_EMPTY) { + return NULL; + } + } while (result == MPSC_QUEUE_RETRY); + + return node; +} + +void +mpsc_queue_push_back(struct mpsc_queue *queue, struct mpsc_queue_node *node) + OVS_REQUIRES(queue->read_lock) +{ + struct mpsc_queue_node *tail; + + atomic_read_relaxed(&queue->tail, &tail); + atomic_store_relaxed(&node->next, tail); + atomic_store_relaxed(&queue->tail, node); +} + +struct mpsc_queue_node * +mpsc_queue_tail(struct mpsc_queue *queue) + OVS_REQUIRES(queue->read_lock) +{ + struct mpsc_queue_node *tail; + struct mpsc_queue_node *next; + + atomic_read_relaxed(&queue->tail, &tail); + atomic_read_explicit(&tail->next, &next, memory_order_acquire); + + if (tail == &queue->stub) { + if (next == NULL) { + return NULL; + } + + atomic_store_relaxed(&queue->tail, next); + tail = next; + } + + return tail; +} + +/* Get the next element of a node. */ +struct mpsc_queue_node *mpsc_queue_next(struct mpsc_queue *queue, + struct mpsc_queue_node *prev) + OVS_REQUIRES(queue->read_lock) +{ + struct mpsc_queue_node *next; + + atomic_read_explicit(&prev->next, &next, memory_order_acquire); + if (next == &queue->stub) { + atomic_read_explicit(&next->next, &next, memory_order_acquire); + } + return next; +} + +void +mpsc_queue_insert(struct mpsc_queue *queue, struct mpsc_queue_node *node) +{ + struct mpsc_queue_node *prev; + + atomic_store_relaxed(&node->next, NULL); + prev = atomic_exchange_explicit(&queue->head, node, memory_order_acq_rel); + atomic_store_explicit(&prev->next, node, memory_order_release); +} diff --git a/lib/mpsc-queue.h b/lib/mpsc-queue.h new file mode 100644 index 000000000..4f8c32cc6 --- /dev/null +++ b/lib/mpsc-queue.h @@ -0,0 +1,189 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef MPSC_QUEUE_H +#define MPSC_QUEUE_H 1 + +#include +#include +#include + +#include +#include + +#include "ovs-atomic.h" + +/* Multi-producer, single-consumer queue + * ===================================== + * + * This data structure is a lockless queue implementation with + * the following properties: + * + * * Multi-producer: multiple threads can write concurrently. + * Insertion in the queue is thread-safe, no inter-thread + * synchronization is necessary. + * + * * Single-consumer: only a single thread can safely remove + * nodes from the queue. The queue must be 'acquired' using + * 'mpsc_queue_acquire()' before removing nodes. + * + * * Unbounded: the queue is backed by a linked-list and is not + * limited in number of elements. + * + * * Intrusive: queue elements are allocated as part of larger + * objects. Objects are retrieved by offset manipulation. + * + * * per-producer FIFO: Elements in the queue are kept in the + * order their producer inserted them. The consumer retrieves + * them in in the same insertion order. When multiple + * producers insert at the same time, either will proceed. + * + * This queue is well-suited for message passing between threads, + * where any number of thread can insert a message and a single + * thread is meant to receive and process it. + * + * Thread-safety + * ============= + * + * The consumer thread must acquire the queue using 'mpsc_queue_acquire()'. + * Once the queue is protected against concurrent reads, the thread can call + * the consumer API: + * + * * mpsc_queue_poll() to peek and return the head of the queue + * * mpsc_queue_pop() to remove the head of the queue + * * mpsc_queue_tail() to read the current tail + * * MPSC_QUEUE_FOR_EACH() to iterate over the current elements, + * without removing them. + * * MPSC_QUEUE_FOR_EACH_POP() to iterate over the elements while + * removing them. + * + * When a thread is finished with reading the queue, it can release the + * reader lock using 'mpsc_queue_release()'. + * + * Producers can always insert elements in the queue, even if no consumer + * acquired the reader lock. No inter-producer synchronization is needed. + * + * The consumer thread is also allowed to insert elements while it holds the + * reader lock. + * + * Producer threads must never be cancelled while writing to the queue. + * This will block the consumer, that will then lose any subsequent elements + * in the queue. Producers should ideally be cooperatively managed or + * the queue insertion should be within non-cancellable sections. + * + * Queue state + * =========== + * + * When polling the queue, three states can be observed: 'empty', 'non-empty', + * and 'inconsistent'. Three polling results are defined, respectively: + * + * * MPSC_QUEUE_EMPTY: the queue is empty. + * * MPSC_QUEUE_ITEM: an item was available and has been removed. + * * MPSC_QUEUE_RETRY: the queue is inconsistent. + * + * If 'MPSC_QUEUE_RETRY' is returned, then a producer has not yet finished + * writing to the queue and the list of nodes is not coherent. The consumer + * can retry shortly to check if the producer has finished. + * + * This behavior is the reason the removal function is called + * 'mpsc_queue_poll()'. + * + */ + +struct mpsc_queue_node { + ATOMIC(struct mpsc_queue_node *) next; +}; + +struct mpsc_queue { + ATOMIC(struct mpsc_queue_node *) head; + ATOMIC(struct mpsc_queue_node *) tail; + struct mpsc_queue_node stub; + struct ovs_mutex read_lock; +}; + +#define MPSC_QUEUE_INITIALIZER(Q) { \ + .head = ATOMIC_VAR_INIT(&(Q)->stub), \ + .tail = ATOMIC_VAR_INIT(&(Q)->stub), \ + .stub = { .next = ATOMIC_VAR_INIT(NULL) }, \ + .read_lock = OVS_MUTEX_INITIALIZER, \ +} + +/* Consumer API. */ + +/* Initialize the queue. Not necessary is 'MPSC_QUEUE_INITIALIZER' was used. */ +void mpsc_queue_init(struct mpsc_queue *queue); +/* The reader lock must be released prior to destroying the queue. */ +void mpsc_queue_destroy(struct mpsc_queue *queue); + +/* Acquire and release the consumer lock. */ +#define mpsc_queue_acquire(q) do { \ + ovs_mutex_lock(&(q)->read_lock); \ + } while (0) +#define mpsc_queue_release(q) do { \ + ovs_mutex_unlock(&(q)->read_lock); \ + } while (0) + +enum mpsc_queue_poll_result { + /* Queue is empty. */ + MPSC_QUEUE_EMPTY, + /* Polling the queue returned an item. */ + MPSC_QUEUE_ITEM, + /* Data has been enqueued but one or more producer thread have not + * finished writing it. The queue is in an inconsistent state. + * Retrying shortly, if the producer threads are still active, will + * return the data. + */ + MPSC_QUEUE_RETRY, +}; + +/* Set 'node' to a removed item from the queue if 'MPSC_QUEUE_ITEM' is + * returned, otherwise 'node' is not set. + */ +enum mpsc_queue_poll_result mpsc_queue_poll(struct mpsc_queue *queue, + struct mpsc_queue_node **node) + OVS_REQUIRES(queue->read_lock); + +/* Pop an element if there is any in the queue. */ +struct mpsc_queue_node *mpsc_queue_pop(struct mpsc_queue *queue) + OVS_REQUIRES(queue->read_lock); + +/* Insert at the back of the queue. Only the consumer can do it. */ +void mpsc_queue_push_back(struct mpsc_queue *queue, + struct mpsc_queue_node *node) + OVS_REQUIRES(queue->read_lock); + +/* Get the current queue tail. */ +struct mpsc_queue_node *mpsc_queue_tail(struct mpsc_queue *queue) + OVS_REQUIRES(queue->read_lock); + +/* Get the next element of a node. */ +struct mpsc_queue_node *mpsc_queue_next(struct mpsc_queue *queue, + struct mpsc_queue_node *prev) + OVS_REQUIRES(queue->read_lock); + +#define MPSC_QUEUE_FOR_EACH(node, queue) \ + for (node = mpsc_queue_tail(queue); node != NULL; \ + node = mpsc_queue_next((queue), node)) + +#define MPSC_QUEUE_FOR_EACH_POP(node, queue) \ + for (node = mpsc_queue_pop(queue); node != NULL; \ + node = mpsc_queue_pop(queue)) + +/* Producer API. */ + +void mpsc_queue_insert(struct mpsc_queue *queue, struct mpsc_queue_node *node); + +#endif /* MPSC_QUEUE_H */ diff --git a/tests/automake.mk b/tests/automake.mk index 1a528aa39..e95eb0180 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -465,6 +465,7 @@ tests_ovstest_SOURCES = \ tests/test-list.c \ tests/test-lockfile.c \ tests/test-multipath.c \ + tests/test-mpsc-queue.c \ tests/test-netflow.c \ tests/test-odp.c \ tests/test-ofpbuf.c \ diff --git a/tests/library.at b/tests/library.at index 1702b7556..ace6234b5 100644 --- a/tests/library.at +++ b/tests/library.at @@ -254,3 +254,8 @@ AT_SETUP([stopwatch module]) AT_CHECK([ovstest test-stopwatch], [0], [...... ], [ignore]) AT_CLEANUP + +AT_SETUP([mpsc-queue module]) +AT_CHECK([ovstest test-mpsc-queue check], [0], [.... +]) +AT_CLEANUP diff --git a/tests/test-mpsc-queue.c b/tests/test-mpsc-queue.c new file mode 100644 index 000000000..26d48b9ff --- /dev/null +++ b/tests/test-mpsc-queue.c @@ -0,0 +1,727 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#undef NDEBUG +#include +#include +#include + +#include + +#include "command-line.h" +#include "guarded-list.h" +#include "mpsc-queue.h" +#include "openvswitch/list.h" +#include "openvswitch/util.h" +#include "ovs-thread.h" +#include "ovstest.h" +#include "timeval.h" +#include "util.h" + +struct element { + union { + struct mpsc_queue_node mpscq; + struct ovs_list list; + } node; + uint64_t mark; +}; + +static void +test_mpsc_queue_mark_element(struct mpsc_queue_node *node, + uint64_t mark, + unsigned int *counter) +{ + struct element *elem; + + elem = CONTAINER_OF(node, struct element, node.mpscq); + elem->mark = mark; + *counter += 1; +} + +static void +test_mpsc_queue_insert(void) +{ + struct element elements[100]; + struct mpsc_queue_node *node; + struct mpsc_queue queue; + unsigned int counter; + size_t i; + + memset(elements, 0, sizeof(elements)); + mpsc_queue_init(&queue); + mpsc_queue_acquire(&queue); + + for (i = 0; i < ARRAY_SIZE(elements); i++) { + mpsc_queue_insert(&queue, &elements[i].node.mpscq); + } + + counter = 0; + while (mpsc_queue_poll(&queue, &node) == MPSC_QUEUE_ITEM) { + test_mpsc_queue_mark_element(node, 1, &counter); + } + + mpsc_queue_release(&queue); + mpsc_queue_destroy(&queue); + + ovs_assert(counter == ARRAY_SIZE(elements)); + for (i = 0; i < ARRAY_SIZE(elements); i++) { + ovs_assert(elements[i].mark == 1); + } + + printf("."); +} + +static void +test_mpsc_queue_removal_fifo(void) +{ + struct element elements[100]; + struct mpsc_queue_node *node; + struct mpsc_queue queue; + unsigned int counter; + size_t i; + + memset(elements, 0, sizeof(elements)); + + mpsc_queue_init(&queue); + mpsc_queue_acquire(&queue); + + for (i = 0; i < ARRAY_SIZE(elements); i++) { + mpsc_queue_insert(&queue, &elements[i].node.mpscq); + } + + /* Elements are in the same order in the list as they + * were declared / initialized. + */ + counter = 0; + while (mpsc_queue_poll(&queue, &node) == MPSC_QUEUE_ITEM) { + test_mpsc_queue_mark_element(node, counter, &counter); + } + + /* The list is valid once extracted from the queue, + * the queue can be destroyed here. + */ + mpsc_queue_release(&queue); + mpsc_queue_destroy(&queue); + + for (i = 0; i < ARRAY_SIZE(elements) - 1; i++) { + struct element *e1, *e2; + + e1 = &elements[i]; + e2 = &elements[i + 1]; + + ovs_assert(e1->mark < e2->mark); + } + + printf("."); +} + +/* Partial insert: + * + * Those functions are 'mpsc_queue_insert()' divided in two parts. + * They serve to test the behavior of the queue when forcing the potential + * condition of a thread starting an insertion then yielding. + */ +static struct mpsc_queue_node * +mpsc_queue_insert_begin(struct mpsc_queue *queue, struct mpsc_queue_node *node) +{ + struct mpsc_queue_node *prev; + + atomic_store_explicit(&node->next, NULL, memory_order_relaxed); + prev = atomic_exchange_explicit(&queue->head, node, memory_order_acq_rel); + return prev; +} + +static void +mpsc_queue_insert_end(struct mpsc_queue_node *prev, + struct mpsc_queue_node *node) +{ + atomic_store_explicit(&prev->next, node, memory_order_release); +} + +static void +test_mpsc_queue_insert_partial(void) +{ + struct element elements[10]; + struct mpsc_queue_node *prevs[ARRAY_SIZE(elements)]; + struct mpsc_queue_node *node; + struct mpsc_queue queue, *q = &queue; + size_t i; + + mpsc_queue_init(q); + + /* Insert the first half of elements entirely, + * insert the second hald of elements partially. + */ + for (i = 0; i < ARRAY_SIZE(elements); i++) { + elements[i].mark = i; + if (i > ARRAY_SIZE(elements) / 2) { + prevs[i] = mpsc_queue_insert_begin(q, &elements[i].node.mpscq); + } else { + prevs[i] = NULL; + mpsc_queue_insert(q, &elements[i].node.mpscq); + } + } + + mpsc_queue_acquire(q); + + /* Verify that when the chain is broken, iterators will stop. */ + i = 0; + MPSC_QUEUE_FOR_EACH (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e == &elements[i]); + i++; + } + ovs_assert(i < ARRAY_SIZE(elements)); + + for (i = 0; i < ARRAY_SIZE(elements); i++) { + if (prevs[i] != NULL) { + mpsc_queue_insert_end(prevs[i], &elements[i].node.mpscq); + } + } + + i = 0; + MPSC_QUEUE_FOR_EACH (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e == &elements[i]); + i++; + } + ovs_assert(i == ARRAY_SIZE(elements)); + + MPSC_QUEUE_FOR_EACH_POP (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e->mark == (unsigned int)(e - elements)); + } + + mpsc_queue_release(q); + mpsc_queue_destroy(q); + + printf("."); +} + +static void +test_mpsc_queue_push_back(void) +{ + struct mpsc_queue queue, *q = &queue; + struct mpsc_queue_node *node; + struct element elements[10]; + size_t i; + + mpsc_queue_init(q); + mpsc_queue_acquire(q); + + ovs_assert(mpsc_queue_pop(q) == NULL); + mpsc_queue_push_back(q, &elements[0].node.mpscq); + node = mpsc_queue_pop(q); + ovs_assert(node == &elements[0].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == NULL); + + mpsc_queue_push_back(q, &elements[0].node.mpscq); + mpsc_queue_push_back(q, &elements[1].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[1].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[0].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == NULL); + + mpsc_queue_push_back(q, &elements[1].node.mpscq); + mpsc_queue_push_back(q, &elements[0].node.mpscq); + mpsc_queue_insert(q, &elements[2].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[0].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[1].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[2].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == NULL); + + for (i = 0; i < ARRAY_SIZE(elements); i++) { + elements[i].mark = i; + mpsc_queue_insert(q, &elements[i].node.mpscq); + } + + node = mpsc_queue_pop(q); + mpsc_queue_push_back(q, node); + ovs_assert(mpsc_queue_pop(q) == node); + mpsc_queue_push_back(q, node); + + i = 0; + MPSC_QUEUE_FOR_EACH (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e == &elements[i]); + i++; + } + ovs_assert(i == ARRAY_SIZE(elements)); + + MPSC_QUEUE_FOR_EACH_POP (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e->mark == (unsigned int)(e - elements)); + } + + mpsc_queue_release(q); + mpsc_queue_destroy(q); + + printf("."); +} + +static void +run_tests(struct ovs_cmdl_context *ctx OVS_UNUSED) +{ + /* Verify basic insertion. */ + test_mpsc_queue_insert(); + /* Test partial insertion. */ + test_mpsc_queue_insert_partial(); + /* Verify removal order is respected. */ + test_mpsc_queue_removal_fifo(); + /* Verify tail-end insertion works. */ + test_mpsc_queue_push_back(); + printf("\n"); +} + +static struct element *elements; +static uint64_t *thread_working_ms; /* Measured work time. */ + +static unsigned int n_threads; +static unsigned int n_elems; + +static struct ovs_barrier barrier; +static volatile bool working; + +static int +elapsed(const struct timeval *start) +{ + struct timeval end; + + xgettimeofday(&end); + return timeval_to_msec(&end) - timeval_to_msec(start); +} + +struct mpscq_aux { + struct mpsc_queue *queue; + atomic_uint thread_id; +}; + +static void * +mpsc_queue_insert_thread(void *aux_) +{ + unsigned int n_elems_per_thread; + struct element *th_elements; + struct mpscq_aux *aux = aux_; + struct timeval start; + unsigned int id; + size_t i; + + atomic_add(&aux->thread_id, 1u, &id); + n_elems_per_thread = n_elems / n_threads; + th_elements = &elements[id * n_elems_per_thread]; + + ovs_barrier_block(&barrier); + xgettimeofday(&start); + + for (i = 0; i < n_elems_per_thread; i++) { + mpsc_queue_insert(aux->queue, &th_elements[i].node.mpscq); + } + + thread_working_ms[id] = elapsed(&start); + ovs_barrier_block(&barrier); + + working = false; + + return NULL; +} + +static void +benchmark_mpsc_queue(void) +{ + struct mpsc_queue_node *node; + struct mpsc_queue queue; + struct timeval start; + unsigned int counter; + bool work_complete; + pthread_t *threads; + struct mpscq_aux aux; + uint64_t epoch; + uint64_t avg; + size_t i; + + memset(elements, 0, n_elems & sizeof *elements); + memset(thread_working_ms, 0, n_threads & sizeof *thread_working_ms); + + mpsc_queue_init(&queue); + + aux.queue = &queue; + atomic_store(&aux.thread_id, 0); + + for (i = n_elems - (n_elems % n_threads); i < n_elems; i++) { + mpsc_queue_insert(&queue, &elements[i].node.mpscq); + } + + working = true; + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("sc_queue_insert", + mpsc_queue_insert_thread, &aux); + } + + mpsc_queue_acquire(&queue); + xgettimeofday(&start); + + counter = 0; + epoch = 1; + do { + while (mpsc_queue_poll(&queue, &node) == MPSC_QUEUE_ITEM) { + test_mpsc_queue_mark_element(node, epoch, &counter); + } + if (epoch == UINT64_MAX) { + epoch = 0; + } + epoch++; + } while (working); + + avg = 0; + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + avg += thread_working_ms[i]; + } + avg /= n_threads; + + /* Elements might have been inserted before threads were joined. */ + while (mpsc_queue_poll(&queue, &node) == MPSC_QUEUE_ITEM) { + test_mpsc_queue_mark_element(node, epoch, &counter); + } + + printf(" mpsc-queue: %6d", elapsed(&start)); + for (i = 0; i < n_threads; i++) { + printf(" %6" PRIu64, thread_working_ms[i]); + } + printf(" %6" PRIu64 " ms\n", avg); + + mpsc_queue_release(&queue); + mpsc_queue_destroy(&queue); + ovs_barrier_destroy(&barrier); + free(threads); + + work_complete = true; + for (i = 0; i < n_elems; i++) { + if (elements[i].mark == 0) { + printf("Element %" PRIuSIZE " was never consumed.\n", i); + work_complete = false; + } + } + ovs_assert(work_complete); + ovs_assert(counter == n_elems); +} + +struct list_aux { + struct ovs_list *list; + struct ovs_mutex *lock; + atomic_uint thread_id; +}; + +static void * +locked_list_insert_thread(void *aux_) +{ + unsigned int n_elems_per_thread; + struct element *th_elements; + struct list_aux *aux = aux_; + struct timeval start; + unsigned int id; + size_t i; + + atomic_add(&aux->thread_id, 1u, &id); + n_elems_per_thread = n_elems / n_threads; + th_elements = &elements[id * n_elems_per_thread]; + + ovs_barrier_block(&barrier); + xgettimeofday(&start); + + for (i = 0; i < n_elems_per_thread; i++) { + ovs_mutex_lock(aux->lock); + ovs_list_push_front(aux->list, &th_elements[i].node.list); + ovs_mutex_unlock(aux->lock); + } + + thread_working_ms[id] = elapsed(&start); + ovs_barrier_block(&barrier); + + working = false; + + return NULL; +} + +static void +benchmark_list(void) +{ + struct ovs_mutex lock; + struct ovs_list list; + struct element *elem; + struct timeval start; + unsigned int counter; + bool work_complete; + pthread_t *threads; + struct list_aux aux; + uint64_t epoch; + uint64_t avg; + size_t i; + + memset(elements, 0, n_elems * sizeof *elements); + memset(thread_working_ms, 0, n_threads * sizeof *thread_working_ms); + + ovs_mutex_init(&lock); + ovs_list_init(&list); + + aux.list = &list; + aux.lock = &lock; + atomic_store(&aux.thread_id, 0); + + ovs_mutex_lock(&lock); + for (i = n_elems - (n_elems % n_threads); i < n_elems; i++) { + ovs_list_push_front(&list, &elements[i].node.list); + } + ovs_mutex_unlock(&lock); + + working = true; + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("locked_list_insert", + locked_list_insert_thread, &aux); + } + + xgettimeofday(&start); + + counter = 0; + epoch = 1; + do { + ovs_mutex_lock(&lock); + LIST_FOR_EACH_POP (elem, node.list, &list) { + elem->mark = epoch; + counter++; + } + ovs_mutex_unlock(&lock); + if (epoch == UINT64_MAX) { + epoch = 0; + } + epoch++; + } while (working); + + avg = 0; + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + avg += thread_working_ms[i]; + } + avg /= n_threads; + + /* Elements might have been inserted before threads were joined. */ + ovs_mutex_lock(&lock); + LIST_FOR_EACH_POP (elem, node.list, &list) { + elem->mark = epoch; + counter++; + } + ovs_mutex_unlock(&lock); + + printf(" list: %6d", elapsed(&start)); + for (i = 0; i < n_threads; i++) { + printf(" %6" PRIu64, thread_working_ms[i]); + } + printf(" %6" PRIu64 " ms\n", avg); + ovs_barrier_destroy(&barrier); + free(threads); + + work_complete = true; + for (i = 0; i < n_elems; i++) { + if (elements[i].mark == 0) { + printf("Element %" PRIuSIZE " was never consumed.\n", i); + work_complete = false; + } + } + ovs_assert(work_complete); + ovs_assert(counter == n_elems); +} + +struct guarded_list_aux { + struct guarded_list *glist; + atomic_uint thread_id; +}; + +static void * +guarded_list_insert_thread(void *aux_) +{ + unsigned int n_elems_per_thread; + struct element *th_elements; + struct guarded_list_aux *aux = aux_; + struct timeval start; + unsigned int id; + size_t i; + + atomic_add(&aux->thread_id, 1u, &id); + n_elems_per_thread = n_elems / n_threads; + th_elements = &elements[id * n_elems_per_thread]; + + ovs_barrier_block(&barrier); + xgettimeofday(&start); + + for (i = 0; i < n_elems_per_thread; i++) { + guarded_list_push_back(aux->glist, &th_elements[i].node.list, n_elems); + } + + thread_working_ms[id] = elapsed(&start); + ovs_barrier_block(&barrier); + + working = false; + + return NULL; +} + +static void +benchmark_guarded_list(void) +{ + struct guarded_list_aux aux; + struct ovs_list extracted; + struct guarded_list glist; + struct element *elem; + struct timeval start; + unsigned int counter; + bool work_complete; + pthread_t *threads; + uint64_t epoch; + uint64_t avg; + size_t i; + + memset(elements, 0, n_elems * sizeof *elements); + memset(thread_working_ms, 0, n_threads * sizeof *thread_working_ms); + + guarded_list_init(&glist); + ovs_list_init(&extracted); + + aux.glist = &glist; + atomic_store(&aux.thread_id, 0); + + for (i = n_elems - (n_elems % n_threads); i < n_elems; i++) { + guarded_list_push_back(&glist, &elements[i].node.list, n_elems); + } + + working = true; + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("guarded_list_insert", + guarded_list_insert_thread, &aux); + } + + xgettimeofday(&start); + + counter = 0; + epoch = 1; + do { + guarded_list_pop_all(&glist, &extracted); + LIST_FOR_EACH_POP (elem, node.list, &extracted) { + elem->mark = epoch; + counter++; + } + if (epoch == UINT64_MAX) { + epoch = 0; + } + epoch++; + } while (working); + + avg = 0; + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + avg += thread_working_ms[i]; + } + avg /= n_threads; + + /* Elements might have been inserted before threads were joined. */ + guarded_list_pop_all(&glist, &extracted); + LIST_FOR_EACH_POP (elem, node.list, &extracted) { + elem->mark = epoch; + counter++; + } + + printf("guarded list: %6d", elapsed(&start)); + for (i = 0; i < n_threads; i++) { + printf(" %6" PRIu64, thread_working_ms[i]); + } + printf(" %6" PRIu64 " ms\n", avg); + ovs_barrier_destroy(&barrier); + free(threads); + guarded_list_destroy(&glist); + + work_complete = true; + for (i = 0; i < n_elems; i++) { + if (elements[i].mark == 0) { + printf("Element %" PRIuSIZE " was never consumed.\n", i); + work_complete = false; + } + } + ovs_assert(work_complete); + ovs_assert(counter == n_elems); +} + +static void +run_benchmarks(struct ovs_cmdl_context *ctx) +{ + long int l_threads; + long int l_elems; + size_t i; + + l_elems = strtol(ctx->argv[1], NULL, 10); + l_threads = strtol(ctx->argv[2], NULL, 10); + ovs_assert(l_elems > 0 && l_threads > 0); + + n_elems = l_elems; + n_threads = l_threads; + + elements = xcalloc(n_elems, sizeof *elements); + thread_working_ms = xcalloc(n_threads, sizeof *thread_working_ms); + + printf("Benchmarking n=%u on 1 + %u threads.\n", n_elems, n_threads); + + printf(" type\\thread: Reader "); + for (i = 0; i < n_threads; i++) { + printf(" %3" PRIuSIZE " ", i + 1); + } + printf(" Avg\n"); + + benchmark_mpsc_queue(); + benchmark_list(); + benchmark_guarded_list(); + + free(thread_working_ms); + free(elements); +} + +static const struct ovs_cmdl_command commands[] = { + {"check", NULL, 0, 0, run_tests, OVS_RO}, + {"benchmark", " ", 2, 2, run_benchmarks, OVS_RO}, + {NULL, NULL, 0, 0, NULL, OVS_RO}, +}; + +static void +test_mpsc_queue_main(int argc, char *argv[]) +{ + struct ovs_cmdl_context ctx = { + .argc = argc - optind, + .argv = argv + optind, + }; + + set_program_name(argv[0]); + ovs_cmdl_run_command(&ctx, commands); +} + +OVSTEST_REGISTER("test-mpsc-queue", test_mpsc_queue_main); From patchwork Mon Apr 12 15:19:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465342 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=p2zp8l56; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=NGAMTPF2; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrf0Vwxz9rx6 for ; Tue, 13 Apr 2021 01:21:06 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 1BAA183C6C; Mon, 12 Apr 2021 15:21:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id l8hC2Tep9DfZ; Mon, 12 Apr 2021 15:21:00 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id A9CD083C8C; Mon, 12 Apr 2021 15:20:44 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E0636C002F; Mon, 12 Apr 2021 15:20:35 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 098CDC0022 for ; Mon, 12 Apr 2021 15:20:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 35BE1838FE for ; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V2KjRmjW4F5o for ; Mon, 12 Apr 2021 15:20:28 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp1.osuosl.org (Postfix) with ESMTPS id B682583B8B for ; Mon, 12 Apr 2021 15:20:27 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id C9A7E5C01CF; Mon, 12 Apr 2021 11:20:26 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=qsqnYH4oalJ1x en7JXhc+GjxfYcViQ+TVeIXB7e/uEg=; b=p2zp8l56ZYZVZL/VEc/i6rmDibzs5 S/b5dZpYpmr1K21RQho+KgE9qefZm5xT2NKrQ3y3OEplJJ0WkTJp8AVg14PdzeuV b2BgzBR2fJ23f0fXT871qdzws4eTOt/+C14o+CKtmQSeE8VaXvUvTFbDI8xzbmWv X4loOmK83yq0wDSe+Mxk4PMZH3GAf5Bm4vFy7KvmkO/5974USBv8UnLlBukpaRsD oMz9dTlHA3FYI6/B1uazA137kGEbFITre0kNKnfjpN9rudBBb1eV+Ps7xjLJsM7z JdPQ0AkLOQqrIDksS3w3VxEMQ1iG0EIVeKVx1RBqBjovJPRa7uLK05Ryw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=qsqnYH4oalJ1xen7JXhc+GjxfYcViQ+TVeIXB7e/uEg=; b=NGAMTPF2 szB+2Puxij4C92l+/uoqEETOes306QJj5md2wpkDH2bh6I68bndOjgaR0Re3A3AK eL8K7aig2NyofnL9zpXeyFFtvOQo08xTVVHM2mr2w6kUOBJjoLpLWaWPvmTODmFo N3csxmzwTdkdkqcdzuTSeO957NpAZTCsSMkXUyf1lT5WWbYZ4qcltDC4Gi5EZIzy ctrtM+5kU1iYVErRj5aVt3gK+/B8iJ8/JmAQUziG+sIckMRsoB0Lkp1D2jTy3yjT k5GonciIY3AyCXIJWWFdxz6Kv00imVmoHfqZwedgJnWjLpLPwwNqqjbd30TB1M0N 5xmQI9ixo9jprA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epudfhfeeikeeiiedutefgheekledukedtkeejheduheffteeiudehiefgkedvleefnecu ffhomhgrihhnpegrphgrtghhvgdrohhrghenucfkphepkeeirddvheegrddvgeefrdduge einecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhr ihhvvgesuhdvheeirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 2414C240054; Mon, 12 Apr 2021 11:20:26 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:57 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 11/28] llring: Add lockless MPMC bounded queue structure X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a lockless multi-producer/multi-consumer, array-based, non-intrusive, bounded queue that will fail on overflow. Each operation (enqueue, dequeue) uses a CAS(). As such, both producer and consumer sides guarantee lock-free forward progress. If the queue is full, enqueuing will fail. Conversely, if the queue is empty, dequeueing will fail. The bound of the queue are restricted to power-of-twos, to allow simpler overflow on unsigned position markers. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/automake.mk | 2 + lib/llring.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++++ lib/llring.h | 76 ++++++++++++++++++++++++ 3 files changed, 231 insertions(+) create mode 100644 lib/llring.c create mode 100644 lib/llring.h diff --git a/lib/automake.mk b/lib/automake.mk index b45801852..f71781438 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -156,6 +156,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/learn.h \ lib/learning-switch.c \ lib/learning-switch.h \ + lib/llring.c \ + lib/llring.h \ lib/lockfile.c \ lib/lockfile.h \ lib/mac-learning.c \ diff --git a/lib/llring.c b/lib/llring.c new file mode 100644 index 000000000..1fb930017 --- /dev/null +++ b/lib/llring.c @@ -0,0 +1,153 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "ovs-atomic.h" + +#include "llring.h" + +/* A queue element. + * Calling 'llring_create' will allocate an array of such elements, + * that will hold the inserted data. + */ +struct llring_node { + atomic_uint32_t seq; + uint32_t data; +}; + +/* A ring description. + * The head and tail of the ring are padded to avoid false-sharing, + * which improves slightly multi-thread performance, at the cost + * of some memory. + */ +struct llring { + PADDED_MEMBERS(CACHE_LINE_SIZE, atomic_uint32_t head;); + PADDED_MEMBERS(CACHE_LINE_SIZE, atomic_uint32_t tail;); + uint32_t mask; + struct llring_node nodes[0]; +}; + +struct llring * +llring_create(uint32_t size) +{ + struct llring *r; + uint32_t i; + + if (size < 2 || !IS_POW2(size)) { + return NULL; + } + + r = xmalloc(sizeof *r + size * sizeof r->nodes[0]); + + r->mask = size - 1; + for (i = 0; i < size; i++) { + atomic_store_relaxed(&r->nodes[i].seq, i); + } + atomic_store_relaxed(&r->head, 0); + atomic_store_relaxed(&r->tail, 0); + + return r; +} + +void +llring_destroy(struct llring *r) +{ + free(r); +} + +bool +llring_enqueue(struct llring *r, uint32_t data) +{ + struct llring_node *node; + uint32_t pos; + + atomic_read_relaxed(&r->head, &pos); + while (true) { + int64_t diff; + uint32_t seq; + + node = &r->nodes[pos & r->mask]; + atomic_read_explicit(&node->seq, &seq, memory_order_acquire); + diff = (int64_t) seq - (int64_t) pos; + + if (diff < 0) { + /* Current ring[head].seq is from previous ring generation, + * ring is full and enqueue fails. */ + return false; + } + + if (diff == 0) { + /* If head == ring[head].seq, then the slot is free, + * attempt to take it by moving the head, if no one moved it since. + */ + if (atomic_compare_exchange_weak_explicit(&r->head, &pos, pos + 1, + memory_order_relaxed, + memory_order_relaxed)) { + break; + } + } else { + /* Someone changed the head since last read, retry. */ + atomic_read_relaxed(&r->head, &pos); + } + } + + node->data = data; + atomic_store_explicit(&node->seq, pos + 1, memory_order_release); + return true; +} + +bool +llring_dequeue(struct llring *r, uint32_t *data) +{ + struct llring_node *node; + uint32_t pos; + + atomic_read_relaxed(&r->tail, &pos); + while (true) { + int64_t diff; + uint32_t seq; + + node = &r->nodes[pos & r->mask]; + atomic_read_explicit(&node->seq, &seq, memory_order_acquire); + diff = (int64_t) seq - (int64_t) (pos + 1); + + if (diff < 0) { + /* Current ring[tail + 1].seq is from previous ring generation, + * ring is empty and dequeue fails. */ + return false; + } + + if (diff == 0) { + /* If tail + 1 == ring[tail + 1].seq, then the slot is allocated, + * attempt to free it by moving the tail, if no one moved it since. + */ + if (atomic_compare_exchange_weak_explicit(&r->tail, &pos, pos + 1, + memory_order_relaxed, + memory_order_relaxed)) { + break; + } + } else { + /* Someone changed the tail since last read, retry. */ + atomic_read_relaxed(&r->tail, &pos); + } + } + + *data = node->data; + /* Advance the slot to next gen by adding r->mask + 1 to its sequence. */ + atomic_store_explicit(&node->seq, pos + r->mask + 1, memory_order_release); + return true; +} diff --git a/lib/llring.h b/lib/llring.h new file mode 100644 index 000000000..f97baa343 --- /dev/null +++ b/lib/llring.h @@ -0,0 +1,76 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include + +#include "ovs-atomic.h" + +/* Bounded lockless queue + * ====================== + * + * A lockless FIFO queue bounded to a known size. + * Each operation (insert, remove) uses one CAS(). + * + * The structure is: + * + * Multi-producer: multiple threads can write to it + * concurrently. + * + * Multi-consumer: multiple threads can read from it + * concurrently. + * + * Bounded: the queue is backed by external memory. + * No new allocation is made on insertion, only the + * used elements in the queue are marked as such. + * The boundary of the queue is defined as the size given + * at init, which must be a power of two. + * + * Failing: when an operation (enqueue, dequeue) cannot + * be performed due to the queue being full/empty, the + * operation immediately fails, instead of waiting on + * a state change. + * + * Non-intrusive: queue elements are allocated prior to + * initialization. Data is shallow-copied to those + * allocated elements. + * + * Thread safety + * ============= + * + * The queue is thread-safe for MPMC case. + * No lock is taken by the queue. The queue guarantees + * lock-free forward progress for each of its operations. + * + */ + +/* Create a circular lockless ring. + * The 'size' parameter must be a power-of-two higher than 2, + * otherwise allocation will fail. + */ +struct llring; +struct llring *llring_create(uint32_t size); + +/* Free a lockless ring. */ +void llring_destroy(struct llring *r); + +/* 'data' is copied to the latest free slot in the queue. */ +bool llring_enqueue(struct llring *r, uint32_t data); + +/* The value within the oldest slot taken in the queue is copied + * to the address pointed by 'data'. + */ +bool llring_dequeue(struct llring *r, uint32_t *data); From patchwork Mon Apr 12 15:19:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465350 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=X8fm2GeE; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=mEuCOchx; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssC6YZJz9rx6 for ; Tue, 13 Apr 2021 01:21:35 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6041D4055E; Mon, 12 Apr 2021 15:21:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1lIIqyAL0ade; Mon, 12 Apr 2021 15:21:24 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id CB785404E6; Mon, 12 Apr 2021 15:20:57 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6F33BC0030; Mon, 12 Apr 2021 15:20:48 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0BEE0C000A for ; Mon, 12 Apr 2021 15:20:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6E290404BD for ; Mon, 12 Apr 2021 15:20:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QTpSt5rXKoSA for ; Mon, 12 Apr 2021 15:20:28 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp2.osuosl.org (Postfix) with ESMTPS id 7563E40462 for ; Mon, 12 Apr 2021 15:20:28 +0000 (UTC) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 9B52F5C01CC; Mon, 12 Apr 2021 11:20:27 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 12 Apr 2021 11:20:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=+ZNyyaeNKaNNP L1a6928P1VPc61KubyLgS7fDaksEV8=; b=X8fm2GeETVXIbPEqKHt8haGE/O2yi QY9jIeb8EuiakybguUGLYNVBKVUNdF8K9fFKo6Trlli2VRTb8Bx+GywX7FDkZ3L8 RXvmrqm9bzBOidTjO+QtQVTZ7YP25F17Y1X5a2j4gW0meT/BcBOgwN70TWReJH78 2NimEqVCwuXxl2zPZj4cvKjycHuaBfRcye6pCp4dQJaQq9Hio6kIm+DKujJcnO6r JCejvN4866P3W1o7a3lfmf/PTeAZLdA6IRPNEiQE9fgPt2Gb8Nbgv+qj+oqZnC4J Brme8Y3dPKn/qhj9t2NlHhXympUPmNqkefV2ACYLA2gGJuvRjiP6uk+xg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=+ZNyyaeNKaNNPL1a6928P1VPc61KubyLgS7fDaksEV8=; b=mEuCOchx uzogNbmCelfaxrpRSbYGskFzqX9Hc2fbPx7WHWTQfuuL/fKKq6Tbc6efi9B4Qqf4 lmxlq4q4wIj+EaR+fpHFpwdFi+Qzh4jlsAvqt14KqyFFgm5LxBGoQ3WHiSABelAA X2q9Z121PwFRNMf9cmhyJsECcbCkAoospS0Lk2sr1YeerRG4I/bBwBUXJjWLKBkv zVVPCZUrrSwqkFjCvOdMpI5XN9Mhy4f+jyzHPAquhMudN531ypO4OYp3GJe3T9k4 03ZNNxaw1ahdLkX0Ei/CiAB3sR1xiMDxC4ijcb0zcm9cepn75+8a/WC28jrGJBy4 TdVChIm7GNpsrA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epudfhfeeikeeiiedutefgheekledukedtkeejheduheffteeiudehiefgkedvleefnecu ffhomhgrihhnpegrphgrtghhvgdrohhrghenucfkphepkeeirddvheegrddvgeefrdduge einecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhr ihhvvgesuhdvheeirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id E34FB240054; Mon, 12 Apr 2021 11:20:26 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:58 +0200 Message-Id: <951b1d0edfd4305678facaebc3ea1e1b41c9103b.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 12/28] seq-pool: Module for faster ID generation X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The current id-pool module is slow to allocate the next valid ID, and can be optimized when restricting some properties of the pool. Those restrictions are: * No ability to add a random ID to the pool. * A new ID is no more the smallest possible ID. It is however guaranteed to be in the range of [base, next_id]. Multiple users of the pool are registered, each with a thread-local cache for better scalability and the next_id is one after the latest ID added to any user cache. The allocation range can be written as: [base, last_alloc + nb-user * cache-size + 1]. * A user should never free an ID that is not allocated. No checks are done and doing so will duplicate the spurious ID. Refcounting or other memory management scheme should be used to ensure an object and its ID are only freed once. This pool is designed to scale reasonably well in multi-thread setup. As it is aimed at being a faster replacement to the current id-pool, a benchmark has been implemented alongside unit tests. The benchmark is composed of 4 rounds: 'new', 'del', 'mix', and 'rnd'. Respectively + 'new': only allocate IDs + 'del': only free IDs + 'mix': allocate, sequential free, then allocate ID. + 'rnd': allocate, random free, allocate ID. Randomized freeing is done by swapping the latest allocated ID with any from the range of currently allocated ID, which is reminiscent of the Fisher-Yates shuffle. This evaluates freeing non-sequential IDs, which is the more natural use-case. For this specific round, the id-pool performance is such that a timeout of 10 seconds is added to the benchmark: $ ./tests/ovstest test-seq-pool benchmark 10000 1 Benchmarking n=10000 on 1 thread. type\thread: 1 Avg seq-pool new: 1 1 ms seq-pool del: 0 0 ms seq-pool mix: 1 1 ms seq-pool rnd: 1 1 ms id-pool new: 0 0 ms id-pool del: 1 1 ms id-pool mix: 1 1 ms id-pool rnd: 1201 1201 ms $ ./tests/ovstest test-seq-pool benchmark 100000 1 Benchmarking n=100000 on 1 thread. type\thread: 1 Avg seq-pool new: 2 2 ms seq-pool del: 5 5 ms seq-pool mix: 5 5 ms seq-pool rnd: 5 5 ms id-pool new: 8 8 ms id-pool del: 5 5 ms id-pool mix: 11 11 ms id-pool rnd: 10000+ ****** ms $ ./tests/ovstest test-seq-pool benchmark 1000000 1 Benchmarking n=1000000 on 1 thread. type\thread: 1 Avg seq-pool new: 23 23 ms seq-pool del: 49 49 ms seq-pool mix: 53 53 ms seq-pool rnd: 53 53 ms id-pool new: 190 190 ms id-pool del: 173 173 ms id-pool mix: 273 273 ms id-pool rnd: 10042+ ****** ms $ ./tests/ovstest test-seq-pool benchmark 1000000 2 Benchmarking n=1000000 on 2 threads. type\thread: 1 2 Avg seq-pool new: 40 39 39 ms seq-pool del: 33 33 33 ms seq-pool mix: 89 91 90 ms seq-pool rnd: 146 151 148 ms id-pool new: 485 485 485 ms id-pool del: 541 542 541 ms id-pool mix: 550 600 575 ms id-pool rnd: 10048+ 10003+ ****** ms $ ./tests/ovstest test-seq-pool benchmark 1000000 4 Benchmarking n=1000000 on 4 threads. type\thread: 1 2 3 4 Avg seq-pool new: 40 39 40 40 39 ms seq-pool del: 24 28 28 30 27 ms seq-pool mix: 60 63 69 69 65 ms seq-pool rnd: 195 197 202 202 199 ms id-pool new: 478 471 482 485 479 ms id-pool del: 474 469 467 474 471 ms id-pool mix: 558 558 611 545 568 ms id-pool rnd: 10121+ 10076+ 10030+ 10167+ ****** ms Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/automake.mk | 2 + lib/seq-pool.c | 198 +++++++++++++++ lib/seq-pool.h | 66 +++++ tests/automake.mk | 1 + tests/library.at | 5 + tests/test-seq-pool.c | 543 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 815 insertions(+) create mode 100644 lib/seq-pool.c create mode 100644 lib/seq-pool.h create mode 100644 tests/test-seq-pool.c diff --git a/lib/automake.mk b/lib/automake.mk index f71781438..3ca43b57a 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -295,6 +295,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/sat-math.h \ lib/seq.c \ lib/seq.h \ + lib/seq-pool.c \ + lib/seq-pool.h \ lib/sha1.c \ lib/sha1.h \ lib/shash.c \ diff --git a/lib/seq-pool.c b/lib/seq-pool.c new file mode 100644 index 000000000..4426d11d8 --- /dev/null +++ b/lib/seq-pool.c @@ -0,0 +1,198 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "openvswitch/list.h" +#include "openvswitch/thread.h" +#include "openvswitch/util.h" +#include "ovs-atomic.h" +#include "llring.h" +#include "seq-pool.h" + +#define SEQPOOL_CACHE_SIZE 32 +BUILD_ASSERT_DECL(IS_POW2(SEQPOOL_CACHE_SIZE)); + +struct seq_node { + struct ovs_list list_node; + uint32_t id; +}; + +struct seq_pool { + uint32_t next_id; + struct llring **cache; /* per-user id cache. */ + size_t nb_user; /* Number of user threads. */ + struct ovs_mutex lock; /* Protects free_ids access. */ + struct ovs_list free_ids; /* Set of currently free IDs. */ + uint32_t base; /* IDs in the range of [base, base + n_ids). */ + uint32_t n_ids; /* Total number of ids in the pool. */ +}; + +struct seq_pool * +seq_pool_create(unsigned int nb_user, uint32_t base, uint32_t n_ids) +{ + struct seq_pool *pool; + size_t i; + + ovs_assert(nb_user != 0); + ovs_assert(base <= UINT32_MAX - n_ids); + + pool = xmalloc(sizeof *pool); + + pool->cache = xcalloc(nb_user, sizeof *pool->cache); + for (i = 0; i < nb_user; i++) { + pool->cache[i] = llring_create(SEQPOOL_CACHE_SIZE); + } + pool->nb_user = nb_user; + + pool->next_id = base; + pool->base = base; + pool->n_ids = n_ids; + + ovs_mutex_init(&pool->lock); + ovs_list_init(&pool->free_ids); + + return pool; +} + +void +seq_pool_destroy(struct seq_pool *pool) +{ + struct seq_node *node; + struct seq_node *next; + size_t i; + + if (!pool) { + return; + } + + ovs_mutex_lock(&pool->lock); + LIST_FOR_EACH_SAFE (node, next, list_node, &pool->free_ids) { + free(node); + } + ovs_list_poison(&pool->free_ids); + ovs_mutex_unlock(&pool->lock); + ovs_mutex_destroy(&pool->lock); + + for (i = 0; i < pool->nb_user; i++) { + llring_destroy(pool->cache[i]); + } + free(pool->cache); + + free(pool); +} + +bool +seq_pool_new_id(struct seq_pool *pool, unsigned int uid, uint32_t *id) +{ + struct llring *cache; + struct ovs_list *front; + struct seq_node *node; + + uid %= pool->nb_user; + cache = pool->cache[uid]; + + if (llring_dequeue(cache, id)) { + return true; + } + + ovs_mutex_lock(&pool->lock); + + while (!ovs_list_is_empty(&pool->free_ids)) { + front = ovs_list_front(&pool->free_ids); + node = CONTAINER_OF(front, struct seq_node, list_node); + if (llring_enqueue(cache, node->id)) { + ovs_list_remove(front); + free(node); + } else { + break; + } + } + + while (pool->next_id < pool->base + pool->n_ids) { + if (llring_enqueue(cache, pool->next_id)) { + pool->next_id++; + } else { + break; + } + } + + ovs_mutex_unlock(&pool->lock); + + if (llring_dequeue(cache, id)) { + return true; + } else { + struct llring *c2; + size_t i; + + /* If no ID was available either from shared counter, + * free-list or local cache, steal an ID from another + * user cache. + */ + for (i = 0; i < pool->nb_user; i++) { + if (i == uid) { + continue; + } + c2 = pool->cache[i]; + if (llring_dequeue(c2, id)) { + return true; + } + } + } + + return false; +} + +void +seq_pool_free_id(struct seq_pool *pool, unsigned int uid, uint32_t id) +{ + struct seq_node *nodes[SEQPOOL_CACHE_SIZE + 1]; + struct llring *cache; + uint32_t node_id; + size_t i; + + if (id < pool->base || id >= pool->base + pool->n_ids) { + return; + } + + uid %= pool->nb_user; + cache = pool->cache[uid]; + + if (llring_enqueue(cache, id)) { + return; + } + + /* Flush the cache. */ + for (i = 0; llring_dequeue(cache, &node_id); i++) { + nodes[i] = xmalloc(sizeof *nodes[i]); + nodes[i]->id = node_id; + } + + /* Finish with the last freed node. */ + nodes[i] = xmalloc(sizeof **nodes); + nodes[i]->id = id; + i++; + + if (i < ARRAY_SIZE(nodes)) { + nodes[i] = NULL; + } + + ovs_mutex_lock(&pool->lock); + for (i = 0; i < ARRAY_SIZE(nodes) && nodes[i] != NULL; i++) { + ovs_list_push_back(&pool->free_ids, &nodes[i]->list_node); + } + ovs_mutex_unlock(&pool->lock); +} diff --git a/lib/seq-pool.h b/lib/seq-pool.h new file mode 100644 index 000000000..c992a0988 --- /dev/null +++ b/lib/seq-pool.h @@ -0,0 +1,66 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef SEQ_POOL_H +#define SEQ_POOL_H + +#include +#include +#include + +/* + * Sequential ID pool. + * =================== + * + * Pool of unique 32bits IDs. + * + * Multiple users are registered at initialization. Each user uses a cache + * of ID. When each thread using the pool uses its own user ID, the pool + * scales reasonably for concurrent allocation. + * + * New IDs are always in the range of '[base, next_id]', where 'next_id' is + * in the range of '[last_alloc_ID + nb_user * cache_size + 1]'. + * This means that a new ID is not always the smallest available ID, but it is + * still from a limited range. + * + * Users should ensure that an ID is *never* freed twice. Not doing so will + * have the effect of double-allocating such ID afterward. + * + * Thread-safety + * ============= + * + * APIs are thread safe. + * + * Multiple threads can share the same user ID if necessary, but it can hurt + * performance if threads are not otherwise synchronized. + */ + +struct seq_pool; + +/* nb_user is the number of expected users of the pool, + * in terms of execution threads. */ +struct seq_pool *seq_pool_create(unsigned int nb_user, + uint32_t base, uint32_t n_ids); +void seq_pool_destroy(struct seq_pool *pool); + +/* uid is the thread user-id. It should be within '[0, nb_user['. */ +bool seq_pool_new_id(struct seq_pool *pool, unsigned int uid, uint32_t *id); + +/* uid is the thread user-id. It should be within '[0, nb_user['. + * An allocated ID must *never* be freed twice. + */ +void seq_pool_free_id(struct seq_pool *pool, unsigned int uid, uint32_t id); +#endif /* seq-pool.h */ diff --git a/tests/automake.mk b/tests/automake.mk index e95eb0180..8265df4ba 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -474,6 +474,7 @@ tests_ovstest_SOURCES = \ tests/test-rcu.c \ tests/test-reconnect.c \ tests/test-rstp.c \ + tests/test-seq-pool.c \ tests/test-sflow.c \ tests/test-sha1.c \ tests/test-skiplist.c \ diff --git a/tests/library.at b/tests/library.at index ace6234b5..8ddd04fa0 100644 --- a/tests/library.at +++ b/tests/library.at @@ -259,3 +259,8 @@ AT_SETUP([mpsc-queue module]) AT_CHECK([ovstest test-mpsc-queue check], [0], [.... ]) AT_CLEANUP + +AT_SETUP([seq-pool module]) +AT_CHECK([ovstest test-seq-pool check], [0], [.... +]) +AT_CLEANUP diff --git a/tests/test-seq-pool.c b/tests/test-seq-pool.c new file mode 100644 index 000000000..a1e0d5500 --- /dev/null +++ b/tests/test-seq-pool.c @@ -0,0 +1,543 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#undef NDEBUG +#include +#include +#include + +#include + +#include "command-line.h" +#include "id-pool.h" +#include "openvswitch/util.h" +#include "ovs-thread.h" +#include "ovs-rcu.h" +#include "ovstest.h" +#include "random.h" +#include "seq-pool.h" +#include "timeval.h" +#include "util.h" + +#define SEQ_POOL_CACHE_SIZE 32 + +#define N_IDS 100 + +static void +test_seq_pool_alloc_full_range(void) +{ + bool ids[N_IDS]; + struct seq_pool *pool; + size_t i; + + memset(ids, 0, sizeof ids); + pool = seq_pool_create(1, 0, N_IDS); + + for (i = 0; i < N_IDS; i++) { + uint32_t id; + + ovs_assert(seq_pool_new_id(pool, 0, &id)); + /* No double alloc.*/ + ovs_assert(ids[id] == false); + ids[id] = true; + } + + for (i = 0; i < N_IDS; i++) { + ovs_assert(ids[i]); + } + + seq_pool_destroy(pool); + printf("."); +} + +static void +test_seq_pool_alloc_steal(void) +{ + /* N must be less than a pool cache size to force the second user + * to steal from the first. + */ +#define N (SEQ_POOL_CACHE_SIZE / 4) + bool ids[N]; + struct seq_pool *pool; + uint32_t id; + size_t i; + + memset(ids, 0, sizeof ids); + pool = seq_pool_create(2, 0, N); + + /* Fill up user 0 cache. */ + ovs_assert(seq_pool_new_id(pool, 0, &id)); + for (i = 0; i < N - 1; i++) { + /* Check that user 1 can still alloc from user 0 cache. */ + ovs_assert(seq_pool_new_id(pool, 1, &id)); + } + + seq_pool_destroy(pool); + printf("."); +#undef N +} + +static void +test_seq_pool_alloc_monotonic(void) +{ + uint32_t ids[N_IDS]; + struct seq_pool *pool; + size_t i; + + memset(ids, 0, sizeof ids); + pool = seq_pool_create(1, 0, N_IDS); + + for (i = 0; i < N_IDS; i++) { + ovs_assert(seq_pool_new_id(pool, 0, &ids[i])); + ovs_assert(ids[i] == i); + } + + seq_pool_destroy(pool); + printf("."); +} + +static void +test_seq_pool_alloc_under_limit(void) +{ + uint32_t ids[N_IDS]; + unsigned int limit; + struct seq_pool *pool; + size_t i; + + memset(ids, 0, sizeof ids); + pool = seq_pool_create(1, 0, N_IDS); + + for (limit = 1; limit < N_IDS; limit++) { + /* Allocate until arbitrary limit then free allocated ids. */ + for (i = 0; i < limit; i++) { + ovs_assert(seq_pool_new_id(pool, 0, &ids[i])); + } + for (i = 0; i < limit; i++) { + seq_pool_free_id(pool, 0, ids[i]); + } + /* Verify that the N='limit' next allocations are under limit. */ + for (i = 0; i < limit; i++) { + ovs_assert(seq_pool_new_id(pool, 0, &ids[i])); + ovs_assert(ids[i] < limit + SEQ_POOL_CACHE_SIZE); + } + for (i = 0; i < limit; i++) { + seq_pool_free_id(pool, 0, ids[i]); + } + } + + seq_pool_destroy(pool); + printf("."); +} + +static void +run_tests(struct ovs_cmdl_context *ctx OVS_UNUSED) +{ + /* Check that all ids can be allocated. */ + test_seq_pool_alloc_full_range(); + /* Check that all ids can be allocated with multiple users. */ + test_seq_pool_alloc_steal(); + /* Check that id allocation is always increasing. */ + test_seq_pool_alloc_monotonic(); + /* Check that id allocation stays under some limit. */ + test_seq_pool_alloc_under_limit(); + printf("\n"); +} + +static uint32_t *ids; +static uint64_t *thread_working_ms; /* Measured work time. */ + +static unsigned int n_threads; +static unsigned int n_ids; + +static struct ovs_barrier barrier; + +#define TIMEOUT_MS (10 * 1000) /* 10 sec timeout */ +static int running_time_ms; +static volatile bool stop = false; + +static int +elapsed(int *start) +{ + return running_time_ms - *start; +} + +static void +swap_u32(uint32_t *a, uint32_t *b) +{ + uint32_t t; + t = *a; + *a = *b; + *b = t; +} + +static void +shuffle(uint32_t *p, size_t n) +{ + for (; n > 1; n--, p++) { + uint32_t *q = &p[random_range(n)]; + swap_u32(p, q); + } +} + +static void +print_result(const char *prefix) +{ + uint64_t avg; + size_t i; + + avg = 0; + for (i = 0; i < n_threads; i++) { + avg += thread_working_ms[i]; + } + avg /= n_threads; + printf("%s: ", prefix); + for (i = 0; i < n_threads; i++) { + if (thread_working_ms[i] >= TIMEOUT_MS) { + printf("%6" PRIu64 "+", thread_working_ms[i]); + } else { + printf(" %6" PRIu64, thread_working_ms[i]); + } + } + if (avg >= TIMEOUT_MS) { + printf(" ****** ms\n"); + } else { + printf(" %6" PRIu64 " ms\n", avg); + } +} + +struct seq_pool_aux { + struct seq_pool *pool; + atomic_uint thread_id; +}; + +static void * +seq_pool_thread(void *aux_) +{ + unsigned int n_ids_per_thread; + struct seq_pool_aux *aux = aux_; + uint32_t *th_ids; + unsigned int tid; + int start; + size_t i; + + atomic_add(&aux->thread_id, 1u, &tid); + n_ids_per_thread = n_ids / n_threads; + th_ids = &ids[tid * n_ids_per_thread]; + + /* NEW / ALLOC */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ignore(seq_pool_new_id(aux->pool, tid, &th_ids[i])); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* DEL */ + + shuffle(th_ids, n_ids_per_thread); + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + seq_pool_free_id(aux->pool, tid, th_ids[i]); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* MIX */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ignore(seq_pool_new_id(aux->pool, tid, &th_ids[i])); + seq_pool_free_id(aux->pool, tid, th_ids[i]); + ignore(seq_pool_new_id(aux->pool, tid, &th_ids[i])); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* MIX SHUFFLED */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + if (elapsed(&start) >= TIMEOUT_MS) { + break; + } + ignore(seq_pool_new_id(aux->pool, tid, &th_ids[i])); + swap_u32(&th_ids[i], &th_ids[random_range(i + 1)]); + seq_pool_free_id(aux->pool, tid, th_ids[i]); + ignore(seq_pool_new_id(aux->pool, tid, &th_ids[i])); + } + thread_working_ms[tid] = elapsed(&start); + + return NULL; +} + +static void +benchmark_seq_pool(void) +{ + pthread_t *threads; + struct seq_pool_aux aux; + size_t i; + + memset(ids, 0, n_ids & sizeof *ids); + memset(thread_working_ms, 0, n_threads & sizeof *thread_working_ms); + + aux.pool = seq_pool_create(n_threads, 0, n_ids); + atomic_store(&aux.thread_id, 0); + + for (i = n_ids - (n_ids % n_threads); i < n_ids; i++) { + uint32_t id; + + seq_pool_new_id(aux.pool, 0, &id); + ids[i] = id; + } + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads + 1); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("seq_pool_alloc", + seq_pool_thread, &aux); + } + + ovs_barrier_block(&barrier); + + print_result("seq-pool new"); + + ovs_barrier_block(&barrier); + + print_result("seq-pool del"); + + ovs_barrier_block(&barrier); + + print_result("seq-pool mix"); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + print_result("seq-pool rnd"); + + seq_pool_destroy(aux.pool); + ovs_barrier_destroy(&barrier); + free(threads); +} + +struct id_pool_aux { + struct id_pool *pool; + struct ovs_mutex *lock; + atomic_uint thread_id; +}; + +static void * +id_pool_thread(void *aux_) +{ + unsigned int n_ids_per_thread; + struct id_pool_aux *aux = aux_; + uint32_t *th_ids; + unsigned int tid; + int start; + size_t i; + + atomic_add(&aux->thread_id, 1u, &tid); + n_ids_per_thread = n_ids / n_threads; + th_ids = &ids[tid * n_ids_per_thread]; + + /* NEW */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ovs_mutex_lock(aux->lock); + ovs_assert(id_pool_alloc_id(aux->pool, &th_ids[i])); + ovs_mutex_unlock(aux->lock); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* DEL */ + + shuffle(th_ids, n_ids_per_thread); + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ovs_mutex_lock(aux->lock); + id_pool_free_id(aux->pool, th_ids[i]); + ovs_mutex_unlock(aux->lock); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* MIX */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ovs_mutex_lock(aux->lock); + ignore(id_pool_alloc_id(aux->pool, &th_ids[i])); + id_pool_free_id(aux->pool, th_ids[i]); + ignore(id_pool_alloc_id(aux->pool, &th_ids[i])); + ovs_mutex_unlock(aux->lock); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* MIX SHUFFLED */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + if (elapsed(&start) >= TIMEOUT_MS) { + break; + } + ovs_mutex_lock(aux->lock); + ignore(id_pool_alloc_id(aux->pool, &th_ids[i])); + swap_u32(&th_ids[i], &th_ids[random_range(i + 1)]); + id_pool_free_id(aux->pool, th_ids[i]); + ignore(id_pool_alloc_id(aux->pool, &th_ids[i])); + ovs_mutex_unlock(aux->lock); + } + thread_working_ms[tid] = elapsed(&start); + + return NULL; +} + +static void +benchmark_id_pool(void) +{ + pthread_t *threads; + struct id_pool_aux aux; + struct ovs_mutex lock; + size_t i; + + memset(ids, 0, n_ids & sizeof *ids); + memset(thread_working_ms, 0, n_threads & sizeof *thread_working_ms); + + aux.pool = id_pool_create(0, n_ids); + aux.lock = &lock; + ovs_mutex_init(&lock); + atomic_store(&aux.thread_id, 0); + + for (i = n_ids - (n_ids % n_threads); i < n_ids; i++) { + id_pool_alloc_id(aux.pool, &ids[i]); + } + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads + 1); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("id_pool_alloc", id_pool_thread, &aux); + } + + ovs_barrier_block(&barrier); + + print_result(" id-pool new"); + + ovs_barrier_block(&barrier); + + print_result(" id-pool del"); + + ovs_barrier_block(&barrier); + + print_result(" id-pool mix"); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + print_result(" id-pool rnd"); + + id_pool_destroy(aux.pool); + ovs_barrier_destroy(&barrier); + free(threads); +} + +static void * +clock_main(void *arg OVS_UNUSED) +{ + struct timeval start; + struct timeval end; + + xgettimeofday(&start); + while (!stop) { + xgettimeofday(&end); + running_time_ms = timeval_to_msec(&end) - timeval_to_msec(&start); + xnanosleep(1000); + } + + return NULL; +} + +static void +run_benchmarks(struct ovs_cmdl_context *ctx) +{ + pthread_t clock; + long int l_threads; + long int l_ids; + size_t i; + + l_ids = strtol(ctx->argv[1], NULL, 10); + l_threads = strtol(ctx->argv[2], NULL, 10); + ovs_assert(l_ids > 0 && l_threads > 0); + + n_ids = l_ids; + n_threads = l_threads; + + ids = xcalloc(n_ids, sizeof *ids); + thread_working_ms = xcalloc(n_threads, sizeof *thread_working_ms); + + clock = ovs_thread_create("clock", clock_main, NULL); + + printf("Benchmarking n=%u on %u thread%s.\n", n_ids, n_threads, + n_threads > 1 ? "s" : ""); + + printf(" type\\thread: "); + for (i = 0; i < n_threads; i++) { + printf(" %3" PRIuSIZE " ", i + 1); + } + printf(" Avg\n"); + + benchmark_seq_pool(); + benchmark_id_pool(); + + stop = true; + + free(thread_working_ms); + xpthread_join(clock, NULL); +} + +static const struct ovs_cmdl_command commands[] = { + {"check", NULL, 0, 0, run_tests, OVS_RO}, + {"benchmark", " ", 2, 2, run_benchmarks, OVS_RO}, + {NULL, NULL, 0, 0, NULL, OVS_RO}, +}; + +static void +test_seq_pool_main(int argc, char *argv[]) +{ + struct ovs_cmdl_context ctx = { + .argc = argc - optind, + .argv = argv + optind, + }; + + set_program_name(argv[0]); + ovs_cmdl_run_command(&ctx, commands); +} + +OVSTEST_REGISTER("test-seq-pool", test_seq_pool_main); From patchwork Mon Apr 12 15:19:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465339 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=H/lqfE/5; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=jN0LZezx; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrQ27QLz9rx6 for ; Tue, 13 Apr 2021 01:20:54 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 9D41B40573; Mon, 12 Apr 2021 15:20:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Daqb5i34gbhu; Mon, 12 Apr 2021 15:20:50 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id 3A1FD4050C; Mon, 12 Apr 2021 15:20:43 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 16698C002E; Mon, 12 Apr 2021 15:20:35 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2C0ABC002D for ; Mon, 12 Apr 2021 15:20:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 6281F404D4 for ; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="H/lqfE/5"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="jN0LZezx" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h0tydFmN0XHk for ; Mon, 12 Apr 2021 15:20:29 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp4.osuosl.org (Postfix) with ESMTPS id 2AF244047C for ; Mon, 12 Apr 2021 15:20:29 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 4BE285C01CA; Mon, 12 Apr 2021 11:20:28 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=4sqkPsPZs+gAe KrtrdEPrtCzvpHEQQdFoL+rEyWFiFM=; b=H/lqfE/5aAVDCcxXxPq92LYu9vH5x MQZzCg1k26J9Gec5LeJx3HYU48+n77MezIIVIrFrz1bEDSgMGFPOkNA1r2PEMb4f cSgrx/XMzk/CGX+t8alYKzRQT4oE/7nq9/nVPl4XJYN0Pk9WNBBcIfI2nxv4YGHT Ioyi1I2n+L/wKKmqTvpdIwoyHhyxi4EnqlNw/aK0rDOk5diXQqd/KCNQtjMbUt7c s3CCPTsYkaWzJ0U0FgzZXVO/TghrhqbObyF2q/A8AQgUM/BE61gm7DRgUr94wBrn i3EVbbu/7IC/i3gknbSLOTt6wtJZkYyozehxMecs0Gzw4a4r1TswWEi5w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=4sqkPsPZs+gAeKrtrdEPrtCzvpHEQQdFoL+rEyWFiFM=; b=jN0LZezx sjqBe4vEqLZQkKF712uux1r3WhekCnpogYvuxSE7fBYxXYOj7hFiJQTnCtJfJZUv BJ95k5aoxDu7i3BCEx8zc4E8xBkw0ln5hTOFQlSqC3H4A1XnQn4aesk1t9HV5gID UzpgzsK0jvMhHm+MwyxUFnk/DlyOOgPVjuCnOWv/KBls8tQzJFBZLeyymY3d5Axu QoN8yfhch/4e5CibxgrBNXLlyQgM1dK9fsE0eIYMoC13QodsmiD8pKDIea0W10uu ev8ESFqWddwd8YyTfOTamEH9O9Wkz24NO4Hm+d1vnWoZHBYd9oZMBmYY4+xK9m/8 S0PbQmzt66ayGQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpeehnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id A7C88240054; Mon, 12 Apr 2021 11:20:27 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:19:59 +0200 Message-Id: <83adaf50925328b5e6635fb4af0bc60f80ef049e.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 13/28] netdev-offload: Add multi-thread API X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Expose functions reporting user configuration of offloading threads, as well as utility functions for multithreading. This will only expose the configuration knob to the user, while no datapath will implement the multiple thread request. This will allow implementations to use this API for offload thread management in relevant layers before enabling the actual dataplane implementation. The offload thread ID is lazily allocated and can as such be in a different order than the offload thread start sequence. The RCU thread will sometime access hardware-offload objects from a provider for reclamation purposes. In such case, it will get a default offload thread ID of 0. Care must be taken that using this thread ID is safe concurrently with the offload threads. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-provider.h | 1 + lib/netdev-offload.c | 88 ++++++++++++++++++++++++++++++++++- lib/netdev-offload.h | 19 ++++++++ vswitchd/vswitch.xml | 16 +++++++ 4 files changed, 122 insertions(+), 2 deletions(-) diff --git a/lib/netdev-offload-provider.h b/lib/netdev-offload-provider.h index 2127599d3..e02330a43 100644 --- a/lib/netdev-offload-provider.h +++ b/lib/netdev-offload-provider.h @@ -84,6 +84,7 @@ struct netdev_flow_api { struct dpif_flow_stats *); /* Get the number of flows offloaded to netdev. + * 'n_flows' is an array of counters, one per offload thread. * Return 0 if successful, otherwise returns a positive errno value. */ int (*flow_get_n_flows)(struct netdev *, uint64_t *n_flows); diff --git a/lib/netdev-offload.c b/lib/netdev-offload.c index deefefd63..087302fd3 100644 --- a/lib/netdev-offload.c +++ b/lib/netdev-offload.c @@ -60,6 +60,12 @@ VLOG_DEFINE_THIS_MODULE(netdev_offload); static bool netdev_flow_api_enabled = false; +#define DEFAULT_OFFLOAD_THREAD_NB 1 +#define MAX_OFFLOAD_THREAD_NB 10 + +static unsigned int offload_thread_nb = DEFAULT_OFFLOAD_THREAD_NB; +DEFINE_EXTERN_PER_THREAD_DATA(netdev_offload_thread_id, OVSTHREAD_ID_UNSET); + /* Protects 'netdev_flow_apis'. */ static struct ovs_mutex netdev_flow_api_provider_mutex = OVS_MUTEX_INITIALIZER; @@ -436,6 +442,64 @@ netdev_is_flow_api_enabled(void) return netdev_flow_api_enabled; } +unsigned int +netdev_offload_thread_nb(void) +{ + return offload_thread_nb; +} + +unsigned int +netdev_offload_ufid_to_thread_id(const ovs_u128 ufid) +{ + uint32_t ufid_hash; + + if (netdev_offload_thread_nb() == 1) { + return 0; + } + + ufid_hash = hash_words64_inline( + (const uint64_t [2]){ ufid.u64.lo, + ufid.u64.hi }, 2, 1); + return ufid_hash % netdev_offload_thread_nb(); +} + +unsigned int +netdev_offload_thread_init(void) +{ + static atomic_count next_id = ATOMIC_COUNT_INIT(0); + bool thread_is_hw_offload; + bool thread_is_rcu; + + thread_is_hw_offload = !strncmp(get_subprogram_name(), + "hw_offload", strlen("hw_offload")); + thread_is_rcu = !strncmp(get_subprogram_name(), "urcu", strlen("urcu")); + + /* Panic if any other thread besides offload and RCU tries + * to initialize their thread ID. */ + ovs_assert(thread_is_hw_offload || thread_is_rcu); + + if (*netdev_offload_thread_id_get() == OVSTHREAD_ID_UNSET) { + unsigned int id; + + if (thread_is_rcu) { + /* RCU will compete with other threads for shared object access. + * Reclamation functions using a thread ID must be thread-safe. + * For that end, and because RCU must consider all potential shared + * objects anyway, its thread-id can be whichever, so return 0. + */ + id = 0; + } else { + /* Only the actual offload threads have their own ID. */ + id = atomic_count_inc(&next_id); + } + /* Panic if any offload thread is getting a spurious ID. */ + ovs_assert(id < netdev_offload_thread_nb()); + return *netdev_offload_thread_id_get() = id; + } else { + return *netdev_offload_thread_id_get(); + } +} + void netdev_ports_flow_flush(const char *dpif_type) { @@ -627,7 +691,16 @@ netdev_ports_get_n_flows(const char *dpif_type, odp_port_t port_no, ovs_rwlock_rdlock(&netdev_hmap_rwlock); data = netdev_ports_lookup(port_no, dpif_type); if (data) { - ret = netdev_flow_get_n_flows(data->netdev, n_flows); + uint64_t thread_n_flows[MAX_OFFLOAD_THREAD_NB] = {0}; + unsigned int tid; + + ret = netdev_flow_get_n_flows(data->netdev, thread_n_flows); + *n_flows = 0; + if (!ret) { + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + *n_flows += thread_n_flows[tid]; + } + } } ovs_rwlock_unlock(&netdev_hmap_rwlock); return ret; @@ -680,7 +753,18 @@ netdev_set_flow_api_enabled(const struct smap *ovs_other_config) if (ovsthread_once_start(&once)) { netdev_flow_api_enabled = true; - VLOG_INFO("netdev: Flow API Enabled"); + offload_thread_nb = smap_get_ullong(ovs_other_config, + "n-offload-threads", + DEFAULT_OFFLOAD_THREAD_NB); + if (offload_thread_nb > MAX_OFFLOAD_THREAD_NB) { + VLOG_WARN("netdev: Invalid number of threads requested: %u", + offload_thread_nb); + offload_thread_nb = DEFAULT_OFFLOAD_THREAD_NB; + } + + VLOG_INFO("netdev: Flow API Enabled, using %u thread%s", + offload_thread_nb, + offload_thread_nb > 1 ? "s" : ""); #ifdef __linux__ tc_set_policy(smap_get_def(ovs_other_config, "tc-policy", diff --git a/lib/netdev-offload.h b/lib/netdev-offload.h index d820e23ed..6e72456e1 100644 --- a/lib/netdev-offload.h +++ b/lib/netdev-offload.h @@ -21,6 +21,7 @@ #include "openvswitch/netdev.h" #include "openvswitch/types.h" #include "ovs-rcu.h" +#include "ovs-thread.h" #include "packets.h" #include "flow.h" @@ -80,6 +81,24 @@ struct offload_info { * to delete the original flow. */ }; +DECLARE_EXTERN_PER_THREAD_DATA(unsigned int, netdev_offload_thread_id); + +unsigned int netdev_offload_thread_nb(void); +unsigned int netdev_offload_thread_init(void); +unsigned int netdev_offload_ufid_to_thread_id(const ovs_u128 ufid); + +static inline unsigned int +netdev_offload_thread_id(void) +{ + unsigned int id = *netdev_offload_thread_id_get(); + + if (OVS_UNLIKELY(id == OVSTHREAD_ID_UNSET)) { + id = netdev_offload_thread_init(); + } + + return id; +} + int netdev_flow_flush(struct netdev *); int netdev_flow_dump_create(struct netdev *, struct netdev_flow_dump **dump, bool terse); diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 4597a215d..022de0fe7 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -247,6 +247,22 @@

+ +

+ Set this value to the number of threads created to manage hardware + offloads. +

+

+ The default value is 1. Changing this value requires + restarting the daemon. +

+

+ This is only relevant if + is enabled. +

+ + From patchwork Mon Apr 12 15:20:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465338 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=G4xLo//X; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=b7At9I3A; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrM3hPTz9rx6 for ; Tue, 13 Apr 2021 01:20:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 7A1E383CBD; Mon, 12 Apr 2021 15:20:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wXLG4lM-sgvX; Mon, 12 Apr 2021 15:20:46 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTP id 52F3483C2E; Mon, 12 Apr 2021 15:20:38 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0A538C002B; Mon, 12 Apr 2021 15:20:32 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id CA1ABC001C for ; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id A956560AC4 for ; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="G4xLo//X"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="b7At9I3A" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rW4SXxwlrxxB for ; Mon, 12 Apr 2021 15:20:29 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp3.osuosl.org (Postfix) with ESMTPS id D418460AB8 for ; Mon, 12 Apr 2021 15:20:29 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 257565C01C8; Mon, 12 Apr 2021 11:20:29 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=SqjgQICJqkVCf 89/PmKlrbNL63XT/ymLM3oWFdKwFoY=; b=G4xLo//XoCamttR5U58/hvYygJPgI XZQCpFUTgiMRD2Uqx05i4vg9y4ITRCN8az8lhJ6m4dTg6GP8aw4WVe3qSCXsvPY/ 9/wbO8CdvS9FOg4y/sry6Xm+JkQ4XxzmDb+NmX+9RThBsSRx1jxDPLwOUV4ttl2C u+oHcyXclzXmtKZ8Vbet4d8Oh30h3ho5NHUfWUZj5uTKgemqb53IlsDg5lirnFEO 6Ffql58lAc3wPMK4nqqooJZprW1xRI0kZOAFoXncnkgmGJQ/tvDu0RH5Da5ZS5ON 3yuMNuP8FxLlBtNQkXlr/7dYjmid1GzKohEQUjzeRqzMbIPBdlhf08DmA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=SqjgQICJqkVCf89/PmKlrbNL63XT/ymLM3oWFdKwFoY=; b=b7At9I3A pCUPXJxiG/Xhi/+KB4Xdm6Gz11cR7Sp6g3SXKTjmzxXgcHKkIKlgomzWOMOxgGvM pYN3kR30CETVon+dekdQP8vXyzPDI/pNR08P5ga4Gv7Q3qmHtvVidAdzjv0f4gkD i5fwd7KPxSent34x50BMbJSWqY2tVT92V37/U3+csGb9sF4lk6f7trVUm6h2cZSk od03WWt+zMBg26W4j4nCWFkWlgr/TJFmWJsBfAArJgjvuyEunuosE32Ir9KNrX6n G6BCBTUAk4xFAer6GsIHx61kIRlpG6r2SV++g19UqrHFlRBQXhRhsHRAQq4SR3L/ IH6FcRbUERAJ3g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpeehnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 6BC41240054; Mon, 12 Apr 2021 11:20:28 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:00 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 14/28] dpif-netdev: Quiesce offload thread periodically X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Similar to what was done for the PMD threads [1], reduce the performance impact of quiescing too often in the offload thread. After each processed offload, the offload thread currently quiesce and will sync with RCU. This synchronization can be lengthy and make the thread unnecessary slow. Instead attempt to quiesce every 10 ms at most. While the queue is empty, the offload thread remains quiescent. [1]: 81ac8b3b194c ("dpif-netdev: Do RCU synchronization at fixed interval in PMD main loop.") Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein --- lib/dpif-netdev.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index d458bcb12..4e403a9e5 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2740,15 +2740,20 @@ err_free: return -1; } +#define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ + static void * dp_netdev_flow_offload_main(void *data OVS_UNUSED) { struct dp_offload_thread_item *offload; struct ovs_list *list; long long int latency_us; + long long int next_rcu; + long long int now; const char *op; int ret; + next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; for (;;) { ovs_mutex_lock(&dp_offload_thread.mutex); if (ovs_list_is_empty(&dp_offload_thread.list)) { @@ -2756,6 +2761,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) ovs_mutex_cond_wait(&dp_offload_thread.cond, &dp_offload_thread.mutex); ovsrcu_quiesce_end(); + next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; } list = ovs_list_pop_front(&dp_offload_thread.list); dp_offload_thread.enqueued_item--; @@ -2779,7 +2785,9 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) OVS_NOT_REACHED(); } - latency_us = time_usec() - offload->timestamp; + now = time_usec(); + + latency_us = now - offload->timestamp; mov_avg_cma_update(&dp_offload_thread.cma, latency_us); mov_avg_ema_update(&dp_offload_thread.ema, latency_us); @@ -2787,7 +2795,13 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) ret == 0 ? "succeed" : "failed", op, UUID_ARGS((struct uuid *) &offload->flow->mega_ufid)); dp_netdev_free_flow_offload(offload); - ovsrcu_quiesce(); + + /* Do RCU synchronization at fixed interval. */ + if (now > next_rcu) { + if (!ovsrcu_try_quiesce()) { + next_rcu += DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; + } + } } return NULL; From patchwork Mon Apr 12 15:20:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465341 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=Fif1JyjY; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=oXR5o5uk; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrV05QVz9sTD for ; Tue, 13 Apr 2021 01:20:57 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 3398583CFE; Mon, 12 Apr 2021 15:20:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vl4Z1y2bWS2T; Mon, 12 Apr 2021 15:20:52 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 7D8AA83C5F; Mon, 12 Apr 2021 15:20:41 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 07963C002D; Mon, 12 Apr 2021 15:20:34 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 05519C002A for ; Mon, 12 Apr 2021 15:20:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id CEA654049A for ; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="Fif1JyjY"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="oXR5o5uk" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HwVtAF7tUkDW for ; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp2.osuosl.org (Postfix) with ESMTPS id B34C240454 for ; Mon, 12 Apr 2021 15:20:30 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id E62CD5C0043; Mon, 12 Apr 2021 11:20:29 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=qAX1ww2pgeQUJ DMLjsxoEBre1womue/4cytJym5O+cU=; b=Fif1JyjYWSDvP4UVldN4la9Jx01FI y8h8sCstwcIUpbe5XEZzkFM42pTgiY/s33GoGDHD6OAmUH556GZT8NkAgBbBSSlU tpgaBT0GPHVKo7TwRM/hdLxbz3nAtcmird8YVmBoxoGeTwZ9LNk7bH/WWiLG8QVj vsZJ3OyODEwEx4fDFY2hY5UMZU2zUoJHiq9UxaVtF6y97G92QjqBki5iFxyMiYei ICwmegG5oBT/EqHJpJ3JkqhvxdmO0xfxCtmsg17AnWFaJ3EuAedroipA8hRpZ/++ rdB+pWEy2YfRGvsFH27psSHNBRm0uWmq2yMkUi5/MTVDd6SczFnlBKYdw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=qAX1ww2pgeQUJDMLjsxoEBre1womue/4cytJym5O+cU=; b=oXR5o5uk ai40moRyAgs4X+u+g9CGkseGu0XCwbb6JFTt8KQYqb5i7WpGaqIkWA5MvEd/O2Hc PH8IXAhlht/IOl/2r9oS4bA8EXed+CRWzFTxJVN6+6xKyNLWkRshoCT/Limynj7W RjDxbrG+nzlomi5U1xRgAms8FyUbrR3PDax6oMk4A3a47P/25gJAJzmT6hxgITRZ snyCYplpo/EAbVzxWucI3Nl0vlg4fOal8DrPQJOC3OeDE1dDqBJG40fT40GbWcXW Hj02nM/gsEFiMeEp8XASKMrfZ8r+U8yvQ+0Z/EdMZKAF3B7jzp5zCcLPB3ixJ4yR XBeyNVB7A+gC2Q== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpeehnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 34EDE240054; Mon, 12 Apr 2021 11:20:29 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:01 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 15/28] dpif-netdev: Postpone flow offload item freeing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Profiling the HW offload thread, the flow offload freeing takes approximatively 25% of the time. Most of this time is spent waiting on the futex used by the libc free(), as it triggers a syscall and reschedule the thread. Avoid the syscall and its expensive context switch. Batch the offload messages freeing using the RCU. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 4e403a9e5..0b439d90b 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2625,14 +2625,19 @@ dp_netdev_alloc_flow_offload(struct dp_netdev_pmd_thread *pmd, return offload; } +static void +dp_netdev_free_flow_offload__(struct dp_offload_thread_item *offload) +{ + free(offload->actions); + free(offload); +} + static void dp_netdev_free_flow_offload(struct dp_offload_thread_item *offload) { dp_netdev_pmd_unref(offload->pmd); dp_netdev_flow_unref(offload->flow); - - free(offload->actions); - free(offload); + ovsrcu_postpone(dp_netdev_free_flow_offload__, offload); } static void From patchwork Mon Apr 12 15:20:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465343 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=i9vE4e0y; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=hzcytqlE; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsrh3zPrz9rx6 for ; Tue, 13 Apr 2021 01:21:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id F37A483BE6; Mon, 12 Apr 2021 15:21:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B8GB1wnpU1EV; Mon, 12 Apr 2021 15:21:04 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 4262383BE2; Mon, 12 Apr 2021 15:20:46 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C9041C0026; Mon, 12 Apr 2021 15:20:36 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 05BA6C0022 for ; Mon, 12 Apr 2021 15:20:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 4CFDB60ACE for ; Mon, 12 Apr 2021 15:20:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="i9vE4e0y"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="hzcytqlE" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4aRh9sSrfDO6 for ; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp3.osuosl.org (Postfix) with ESMTPS id 7480E60ABF for ; Mon, 12 Apr 2021 15:20:31 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id AB54C5C01C8; Mon, 12 Apr 2021 11:20:30 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=21ULPZgSz7YSP Sg+VkT7IrW66hBoj7VbnjVCukH4Iio=; b=i9vE4e0y6S9at+HE10MKuOa7v6dvJ Aw+SszInyqQ3e+huRQ/S/5zsmxwKeEO02LIk3xCNpJJ7X82wbqKmYxdfC1cEJBig sJTiCiEjAIVLTNTQSBJRv1FJVA20WADEIOnaU2JCVaxN2Vf8b0/DCh+RShlvnMl9 GMeIrUFWmXMCVhJyQU7DLGHk/Ucc955Q8Ic1F81eiB5asdJb4KNWmnF44VSZKQUR AP+lYVgyoj1YUn573fJKFTRR3Whr1JMDgqKDPJD6CaDG4CpuLZpJ04RL2ocYBYOL 8d0RsRIi4KsWwCmczNLTfTXSj8ebYGtm75bdBel00CVo+R902oxAtQTjg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=21ULPZgSz7YSPSg+VkT7IrW66hBoj7VbnjVCukH4Iio=; b=hzcytqlE 0wfXFMzLrkQXWHovmFB8AReJOHRNW8qvHnqBcaadgryUwFCcUYmDuM5uVvfq7uig ec9MY4Rv5GVT/CAeZzUXZ+Bt8sgV/07lIUz8DBO4EHKiRYv2Taebfz55mF/ZteEy 8mecJeXBKxcEW4vE/hlAf/dWMUoKljz7EIv3vPY9OzlO8LLH1kGhrBTppJ01vyn9 CPLWonH8uj9AW4Pp70P81KgTDk72cVfauoXyOKua3KgjS3Ii9l2dQrhdLJqniQrY 0ngwqa/Xi6voHXzuKLb2We+S3THgHuR2LHs7SKWa3ovywLUaTN/RmqT26diUqeaC vhZhai7orK4DnA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpeehnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id EC81F240067; Mon, 12 Apr 2021 11:20:29 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:02 +0200 Message-Id: <307451d62af9796cb963fa0d424ea31407a8927c.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 16/28] dpif-netdev: Use seq-pool for mark allocation X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Use the netdev-offload multithread API to allow multiple thread allocating marks concurrently. Initialize only once the pool in a multithread context by using the ovsthread_once type. Use the seq-pool module for faster concurrent ID allocation. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 0b439d90b..851e33f85 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -74,6 +74,7 @@ #include "pvector.h" #include "random.h" #include "seq.h" +#include "seq-pool.h" #include "smap.h" #include "sset.h" #include "timeval.h" @@ -2418,7 +2419,7 @@ struct megaflow_to_mark_data { struct flow_mark { struct cmap megaflow_to_mark; struct cmap mark_to_flow; - struct id_pool *pool; + struct seq_pool *pool; }; static struct flow_mark flow_mark = { @@ -2429,14 +2430,18 @@ static struct flow_mark flow_mark = { static uint32_t flow_mark_alloc(void) { + static struct ovsthread_once pool_init = OVSTHREAD_ONCE_INITIALIZER; + unsigned int tid = netdev_offload_thread_id(); uint32_t mark; - if (!flow_mark.pool) { + if (ovsthread_once_start(&pool_init)) { /* Haven't initiated yet, do it here */ - flow_mark.pool = id_pool_create(1, MAX_FLOW_MARK); + flow_mark.pool = seq_pool_create(netdev_offload_thread_nb(), + 1, MAX_FLOW_MARK); + ovsthread_once_done(&pool_init); } - if (id_pool_alloc_id(flow_mark.pool, &mark)) { + if (seq_pool_new_id(flow_mark.pool, tid, &mark)) { return mark; } @@ -2446,7 +2451,9 @@ flow_mark_alloc(void) static void flow_mark_free(uint32_t mark) { - id_pool_free_id(flow_mark.pool, mark); + unsigned int tid = netdev_offload_thread_id(); + + seq_pool_free_id(flow_mark.pool, tid, mark); } /* associate megaflow with a mark, which is a 1:1 mapping */ From patchwork Mon Apr 12 15:20:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465346 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=N6s/KUFp; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=d8iu/7aR; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJss72KZmz9rx6 for ; Tue, 13 Apr 2021 01:21:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 3855383BB8; Mon, 12 Apr 2021 15:21:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D7KHkY1_poTB; Mon, 12 Apr 2021 15:21:26 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 15C9883C0D; Mon, 12 Apr 2021 15:20:54 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 46CEFC001B; Mon, 12 Apr 2021 15:20:45 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1DA16C0018 for ; Mon, 12 Apr 2021 15:20:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id CA95C83BF2 for ; Mon, 12 Apr 2021 15:20:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id msXd9opodUJH for ; Mon, 12 Apr 2021 15:20:33 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by smtp1.osuosl.org (Postfix) with ESMTPS id 89989838F2 for ; Mon, 12 Apr 2021 15:20:32 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id AF3781810; Mon, 12 Apr 2021 11:20:31 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 12 Apr 2021 11:20:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=Iv1rzePYZrxxx 5qjMEv3VP02TJLoFeWfrG+GeUVDhCQ=; b=N6s/KUFpYOMmwGBQVLdsl89R4/tqG HWk9zZeVfVeOAoDowxUFzMMhwZABbpkAnpdq7N+ddLErX4i3BGUYxu8zGKIt2pid oIeD5rJ4XD4gy+Jm+sch91M3gtTxKqjoUXBfKRWbVPw/tNoqesM8p2w8dC+1gqHF VtCIrr9BL+tev1a7sdOCLuKJ2QsQwNqVVAG1ECm5vati5occbpUyRkNFMV1v2+sy CCHQw15sc2XFZXVP6krvqR2mC28Fo04rAozUXx1dqj01Yi9ATtF9Z2S/lUYe3nEX cbfHFDAD0dVay7rsRtUaWThs+cIOvv/jC2iDTWShYlfBScBYF1GTylfcA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=Iv1rzePYZrxxx5qjMEv3VP02TJLoFeWfrG+GeUVDhCQ=; b=d8iu/7aR S0J9jREZrkrasBmvTNbvxTfBdPqzM28rpiHpuXramoioMaWSDRXYvCL+Y6N59Vxy JoGOSfnlAPRuwjjp/YOs3eKRNwod2UIp80tQWNXSEzY99kofI+JJ4bX1UQmo0Uox uIUTkFrsyq+bKZABaPSQuDiL4Bq3jFYJHq3e3KlAaQDj6MkCTjzxcXuDsUsDzbiL u5Nn8urI3UEMvvUrCAALcTMlEmygaMoUrPpyz80uLpkpPV+Avm9JFf8GBOBolC7n QtM5CVCGmcF+CbDieRnlVIbaKhXEDSI4+2kv+M+QrHgs0hrKsDRZXmqYnXcBwyCg lytBi3dzkokh5w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhgggfestdekre dtredttdenucfhrhhomhepifgrvghtrghnucftihhvvghtuceoghhrihhvvgesuhdvheei rdhnvghtqeenucggtffrrghtthgvrhhnpeehgfevffekteehteefieefvdehleefjeefhe evudetjefhkeeutdekieeuvdetheenucfkphepkeeirddvheegrddvgeefrddugeeinecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id B6C8D24005C; Mon, 12 Apr 2021 11:20:30 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:03 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 17/28] dpif-netdev: Introduce tagged union of offload requests X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Offload requests are currently only supporting flow offloads. As a pre-step before supporting an offload flush request, modify the layout of an offload request item, to become a tagged union. Future offload types won't be forced to re-use the full flow offload structure, which consumes a lot of memory. Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 128 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 89 insertions(+), 39 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 851e33f85..b8fccdb43 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -420,22 +420,34 @@ enum rxq_cycles_counter_type { RXQ_N_CYCLES }; +enum dp_offload_type { + DP_OFFLOAD_FLOW, +}; + enum { DP_NETDEV_FLOW_OFFLOAD_OP_ADD, DP_NETDEV_FLOW_OFFLOAD_OP_MOD, DP_NETDEV_FLOW_OFFLOAD_OP_DEL, }; -struct dp_offload_thread_item { +struct dp_offload_flow_item { struct dp_netdev_pmd_thread *pmd; struct dp_netdev_flow *flow; int op; struct match match; struct nlattr *actions; size_t actions_len; - long long int timestamp; +}; +union dp_offload_thread_data { + struct dp_offload_flow_item flow; +}; + +struct dp_offload_thread_item { struct ovs_list node; + enum dp_offload_type type; + long long int timestamp; + union dp_offload_thread_data data[0]; }; struct dp_offload_thread { @@ -2619,34 +2631,55 @@ dp_netdev_alloc_flow_offload(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow, int op) { - struct dp_offload_thread_item *offload; + struct dp_offload_thread_item *item; + struct dp_offload_flow_item *flow_offload; + + item = xzalloc(sizeof *item + sizeof *flow_offload); + flow_offload = &item->data->flow; - offload = xzalloc(sizeof(*offload)); - offload->pmd = pmd; - offload->flow = flow; - offload->op = op; + item->type = DP_OFFLOAD_FLOW; + + flow_offload->pmd = pmd; + flow_offload->flow = flow; + flow_offload->op = op; dp_netdev_flow_ref(flow); dp_netdev_pmd_try_ref(pmd); - return offload; + return item; } static void dp_netdev_free_flow_offload__(struct dp_offload_thread_item *offload) { - free(offload->actions); + struct dp_offload_flow_item *flow_offload = &offload->data->flow; + + free(flow_offload->actions); free(offload); } static void dp_netdev_free_flow_offload(struct dp_offload_thread_item *offload) { - dp_netdev_pmd_unref(offload->pmd); - dp_netdev_flow_unref(offload->flow); + struct dp_offload_flow_item *flow_offload = &offload->data->flow; + + dp_netdev_pmd_unref(flow_offload->pmd); + dp_netdev_flow_unref(flow_offload->flow); ovsrcu_postpone(dp_netdev_free_flow_offload__, offload); } +static void +dp_netdev_free_offload(struct dp_offload_thread_item *offload) +{ + switch (offload->type) { + case DP_OFFLOAD_FLOW: + dp_netdev_free_flow_offload(offload); + break; + default: + OVS_NOT_REACHED(); + }; +} + static void dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) { @@ -2658,7 +2691,7 @@ dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) } static int -dp_netdev_flow_offload_del(struct dp_offload_thread_item *offload) +dp_netdev_flow_offload_del(struct dp_offload_flow_item *offload) { return mark_to_flow_disassociate(offload->pmd, offload->flow); } @@ -2675,7 +2708,7 @@ dp_netdev_flow_offload_del(struct dp_offload_thread_item *offload) * valid, thus only item 2 needed. */ static int -dp_netdev_flow_offload_put(struct dp_offload_thread_item *offload) +dp_netdev_flow_offload_put(struct dp_offload_flow_item *offload) { struct dp_netdev_pmd_thread *pmd = offload->pmd; struct dp_netdev_flow *flow = offload->flow; @@ -2752,6 +2785,35 @@ err_free: return -1; } +static void +dp_offload_flow(struct dp_offload_thread_item *item) +{ + struct dp_offload_flow_item *flow_offload = &item->data->flow; + const char *op; + int ret; + + switch (flow_offload->op) { + case DP_NETDEV_FLOW_OFFLOAD_OP_ADD: + op = "add"; + ret = dp_netdev_flow_offload_put(flow_offload); + break; + case DP_NETDEV_FLOW_OFFLOAD_OP_MOD: + op = "modify"; + ret = dp_netdev_flow_offload_put(flow_offload); + break; + case DP_NETDEV_FLOW_OFFLOAD_OP_DEL: + op = "delete"; + ret = dp_netdev_flow_offload_del(flow_offload); + break; + default: + OVS_NOT_REACHED(); + } + + VLOG_DBG("%s to %s netdev flow "UUID_FMT, + ret == 0 ? "succeed" : "failed", op, + UUID_ARGS((struct uuid *) &flow_offload->flow->mega_ufid)); +} + #define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ static void * @@ -2762,8 +2824,6 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) long long int latency_us; long long int next_rcu; long long int now; - const char *op; - int ret; next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; for (;;) { @@ -2780,18 +2840,9 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); ovs_mutex_unlock(&dp_offload_thread.mutex); - switch (offload->op) { - case DP_NETDEV_FLOW_OFFLOAD_OP_ADD: - op = "add"; - ret = dp_netdev_flow_offload_put(offload); - break; - case DP_NETDEV_FLOW_OFFLOAD_OP_MOD: - op = "modify"; - ret = dp_netdev_flow_offload_put(offload); - break; - case DP_NETDEV_FLOW_OFFLOAD_OP_DEL: - op = "delete"; - ret = dp_netdev_flow_offload_del(offload); + switch (offload->type) { + case DP_OFFLOAD_FLOW: + dp_offload_flow(offload); break; default: OVS_NOT_REACHED(); @@ -2803,10 +2854,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) mov_avg_cma_update(&dp_offload_thread.cma, latency_us); mov_avg_ema_update(&dp_offload_thread.ema, latency_us); - VLOG_DBG("%s to %s netdev flow "UUID_FMT, - ret == 0 ? "succeed" : "failed", op, - UUID_ARGS((struct uuid *) &offload->flow->mega_ufid)); - dp_netdev_free_flow_offload(offload); + dp_netdev_free_offload(offload); /* Do RCU synchronization at fixed interval. */ if (now > next_rcu) { @@ -2842,7 +2890,8 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow, struct match *match, const struct nlattr *actions, size_t actions_len) { - struct dp_offload_thread_item *offload; + struct dp_offload_thread_item *item; + struct dp_offload_flow_item *flow_offload; int op; if (!netdev_is_flow_api_enabled()) { @@ -2860,14 +2909,15 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, } else { op = DP_NETDEV_FLOW_OFFLOAD_OP_ADD; } - offload = dp_netdev_alloc_flow_offload(pmd, flow, op); - offload->match = *match; - offload->actions = xmalloc(actions_len); - memcpy(offload->actions, actions, actions_len); - offload->actions_len = actions_len; + item = dp_netdev_alloc_flow_offload(pmd, flow, op); + flow_offload = &item->data->flow; + flow_offload->match = *match; + flow_offload->actions = xmalloc(actions_len); + memcpy(flow_offload->actions, actions, actions_len); + flow_offload->actions_len = actions_len; - offload->timestamp = pmd->ctx.now; - dp_netdev_append_flow_offload(offload); + item->timestamp = pmd->ctx.now; + dp_netdev_append_flow_offload(item); } static void From patchwork Mon Apr 12 15:20:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465349 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=W2Pk1mb6; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=IWaiVZNF; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssB6z5pz9sTD for ; Tue, 13 Apr 2021 01:21:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id B1E0583C9C; Mon, 12 Apr 2021 15:21:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NH_52vzimCIR; Mon, 12 Apr 2021 15:21:30 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 8F62083D11; Mon, 12 Apr 2021 15:20:55 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1EF6DC002E; Mon, 12 Apr 2021 15:20:46 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0639DC001E for ; Mon, 12 Apr 2021 15:20:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 19A45404FE for ; Mon, 12 Apr 2021 15:20:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="W2Pk1mb6"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="IWaiVZNF" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C0oB16gDbDeA for ; Mon, 12 Apr 2021 15:20:33 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by smtp4.osuosl.org (Postfix) with ESMTPS id 01C1A404A9 for ; Mon, 12 Apr 2021 15:20:32 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 4C01417D3; Mon, 12 Apr 2021 11:20:32 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=LRkM0sjiwEliW 6i/jnxcXampjR0c3vRrNeDcDJr6FmE=; b=W2Pk1mb6tbjnMVUCfe1bJ4+0SfmnP +CKHG8PRdvaxKCLJZbkh06yfL8s3GY8fX21cWb+EGTuNIT0Zu691bCNj+epTQ1UQ OWkywZKwCQBzTfGYhA1LAnN2PyLdoCSjaFgc69SmVTdJXTaXTSGh7dwnWyVaNymX qFcB6YwXesJQ9wW93UJCxhRyRm+kHlCUSbTRqtk+mnn+z9Ip2Fze3rng4uBdnErL ruWdSHUyazQgc+zVSvGye6RLTTJQGTTEtEaFjNGiuxxqhTX0YlDyA3Xk/LRFJDC7 cruYIC6aXNAjt7kcdt/645xfYor8RhjfNMSVzcJPpdB8yjT42jkcXGeGg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=LRkM0sjiwEliW6i/jnxcXampjR0c3vRrNeDcDJr6FmE=; b=IWaiVZNF +yFtvuvBNQ82I1jdtbZLaC5FPwIcsdfNEEmXQYMISLEELxGvcJCqUbF34DnEJGrM pjpUENZVoJr/AiMwJzbj/0qh4UXx64aF0116pFqz9eNfag8Wy0JUvNGPTuPdFfcn 6VpL6N/Ndk1qOvd80JGs5qQam/6GswD4wFu5UeesBPfR+k5kBIorih5pl1UKoJN4 QVq6H+aARUe6Jb/E8o1Kt5qghR54A84WIz57PA7+Fr5Hs9dTJ7/DCWS1mC86dlRz 11EUG9lMgFtkWm8bi73CM/RFdZxpQcuCCajK/Kg5nTPSP9McIMkQ7lJ/Qari1CDW 7lmdGCrHmxyFQg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhgggfestdekre dtredttdenucfhrhhomhepifgrvghtrghnucftihhvvghtuceoghhrihhvvgesuhdvheei rdhnvghtqeenucggtffrrghtthgvrhhnpeduhfefieekieeiudetgfehkeeludektdekje ehudehffetieduheeigfekvdelfeenucffohhmrghinheprghprggthhgvrdhorhhgnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 698F1240054; Mon, 12 Apr 2021 11:20:31 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:04 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 18/28] tests: Add ovs-barrier unit test X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" No unit test exist currently for the ovs-barrier type. It is however crucial as a building block and should be verified to work as expected. Create a simple test verifying the basic function of ovs-barrier. Integrate the test as part of the test suite. Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- tests/automake.mk | 1 + tests/library.at | 5 + tests/test-barrier.c | 264 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 270 insertions(+) create mode 100644 tests/test-barrier.c diff --git a/tests/automake.mk b/tests/automake.mk index 8265df4ba..66f0ed678 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -448,6 +448,7 @@ tests_ovstest_SOURCES = \ tests/ovstest.h \ tests/test-aes128.c \ tests/test-atomic.c \ + tests/test-barrier.c \ tests/test-bundle.c \ tests/test-byte-order.c \ tests/test-classifier.c \ diff --git a/tests/library.at b/tests/library.at index 8ddd04fa0..5ffe2fd0a 100644 --- a/tests/library.at +++ b/tests/library.at @@ -246,6 +246,11 @@ AT_SETUP([ofpbuf module]) AT_CHECK([ovstest test-ofpbuf], [0], []) AT_CLEANUP +AT_SETUP([barrier module]) +AT_KEYWORDS([barrier]) +AT_CHECK([ovstest test-barrier], [0], []) +AT_CLEANUP + AT_SETUP([rcu]) AT_CHECK([ovstest test-rcu-quiesce], [0], []) AT_CLEANUP diff --git a/tests/test-barrier.c b/tests/test-barrier.c new file mode 100644 index 000000000..3bc5291cc --- /dev/null +++ b/tests/test-barrier.c @@ -0,0 +1,264 @@ +/* + * Copyright (c) 2021 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include + +#include "ovs-thread.h" +#include "ovs-rcu.h" +#include "ovstest.h" +#include "random.h" +#include "util.h" + +#define DEFAULT_N_THREADS 4 +#define NB_STEPS 4 + +static bool verbose; +static struct ovs_barrier barrier; + +struct blocker_aux { + unsigned int tid; + bool leader; + int step; +}; + +static void * +basic_blocker_main(void *aux_) +{ + struct blocker_aux *aux = aux_; + size_t i; + + aux->step = 0; + for (i = 0; i < NB_STEPS; i++) { + ovs_barrier_block(&barrier); + aux->step++; + ovs_barrier_block(&barrier); + } + + return NULL; +} + +static void +basic_block_check(struct blocker_aux *aux, size_t n, int expected) +{ + size_t i; + + for (i = 0; i < n; i++) { + if (verbose) { + printf("aux[%" PRIuSIZE "]=%d == %d", i, aux[i].step, expected); + if (aux[i].step != expected) { + printf(" <--- X"); + } + printf("\n"); + } else { + ovs_assert(aux[i].step == expected); + } + } + ovs_barrier_block(&barrier); + ovs_barrier_block(&barrier); +} + +/* + * Basic barrier test. + * + * N writers and 1 reader participate in the test. + * Each thread goes through M steps (=NB_STEPS). + * The main thread participates as the reader. + * + * A Step is divided in three parts: + * 1. before + * (barrier) + * 2. during + * (barrier) + * 3. after + * + * Each writer updates a thread-local variable with the + * current step number within part 2 and waits. + * + * The reader checks all variables during part 3, expecting + * all variables to be equal. If any variable differs, it means + * its thread was not properly blocked by the barrier. + */ +static void +test_barrier_basic(size_t n_threads) +{ + struct blocker_aux *aux; + pthread_t *threads; + size_t i; + + ovs_barrier_init(&barrier, n_threads + 1); + + aux = xcalloc(n_threads, sizeof *aux); + threads = xmalloc(n_threads * sizeof *threads); + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("ovs-barrier", + basic_blocker_main, &aux[i]); + } + + for (i = 0; i < NB_STEPS; i++) { + basic_block_check(aux, n_threads, i); + } + ovs_barrier_destroy(&barrier); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + free(threads); + free(aux); +} + +static unsigned int *shared_mem; + +static void * +lead_blocker_main(void *aux_) +{ + struct blocker_aux *aux = aux_; + size_t i; + + aux->step = 0; + for (i = 0; i < NB_STEPS; i++) { + if (aux->leader) { + shared_mem = xmalloc(sizeof *shared_mem); + if (verbose) { + printf("*T1: allocated shmem\n"); + } + } + xnanosleep(random_range(100) * 1000); + + ovs_barrier_block(&barrier); + + if (verbose) { + printf("%cT%u: ENTER, writing\n", + (aux->leader ? '*' : ' '), aux->tid); + } + + shared_mem[0] = 42; + + ovs_barrier_block(&barrier); + + if (verbose) { + printf("%cT%u: EXIT\n", + (aux->leader ? '*' : ' '), aux->tid); + } + + if (aux->leader) { + free(shared_mem); + if (verbose) { + printf("*T1: freed shmem\n"); + } + } + xnanosleep(random_range(100) * 1000); + } + + return NULL; +} + +/* + * Leader barrier test. + * + * N threads participates, one of which is marked as + * the leader (thread 0). The main thread does not + * participate. + * + * The test is divided in M steps (=NB_STEPS). + * A Step is divided in three parts: + * 1. before + * (barrier) + * 2. during + * (barrier) + * 3. after + * + * Part 1, the leader allocates a block of shared memory. + * Part 2, all threads write to the shared memory. + * Part 3: the leader frees the shared memory. + * + * If any thread is improperly blocked by the barrier, + * the shared memory accesses will trigger a segfault + * or a use-after-free if ASAN is enabled. + */ +static void +test_barrier_lead(size_t n_threads) +{ + struct blocker_aux *aux; + pthread_t *threads; + size_t i; + + ovs_barrier_init(&barrier, n_threads); + + aux = xcalloc(n_threads, sizeof *aux); + threads = xmalloc(n_threads * sizeof *threads); + + aux[0].leader = true; + + for (i = 0; i < n_threads; i++) { + aux[i].tid = i + 1; + threads[i] = ovs_thread_create("ovs-barrier", + lead_blocker_main, &aux[i]); + } + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + /* If the main thread does not participate in the barrier, + * it must wait for all threads to join before destroying it. + */ + ovs_barrier_destroy(&barrier); + + free(threads); + free(aux); +} + +static void +usage(char *test_name) +{ + fprintf(stderr, "Usage: %s [n_threads=%d] [-v]\n", + test_name, DEFAULT_N_THREADS); +} + +static void +test_barrier(int argc, char *argv[]) +{ + size_t n_threads = DEFAULT_N_THREADS; + char **args = argv + optind - 1; + + set_program_name(argv[0]); + + argc -= optind; + if (argc > 2) { + usage(args[0]); + return; + } + + while (argc-- > 0) { + args++; + if (!strcmp(args[0], "-v")) { + verbose = true; + } else { + n_threads = strtol(args[0], NULL, 10); + if (n_threads > 20) { + n_threads = 20; + } + } + } + + test_barrier_basic(n_threads); + test_barrier_lead(n_threads); +} + +OVSTEST_REGISTER("test-barrier", test_barrier); From patchwork Mon Apr 12 15:20:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465345 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=HjLPq0J+; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=mrs85mYU; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJss16j9zz9sTD for ; Tue, 13 Apr 2021 01:21:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 937F983C6F; Mon, 12 Apr 2021 15:21:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bARY1vuWDzYe; Mon, 12 Apr 2021 15:21:20 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 52BD683CE7; Mon, 12 Apr 2021 15:20:52 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 76064C0017; Mon, 12 Apr 2021 15:20:43 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 21264C001E for ; Mon, 12 Apr 2021 15:20:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C572240519 for ; Mon, 12 Apr 2021 15:20:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="HjLPq0J+"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="mrs85mYU" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8mL8NNTe_OHd for ; Mon, 12 Apr 2021 15:20:33 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by smtp4.osuosl.org (Postfix) with ESMTPS id 8867A404DF for ; Mon, 12 Apr 2021 15:20:33 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id EEF8417E3; Mon, 12 Apr 2021 11:20:32 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=lJ6+Jhax9UCDn izTO8oOunFeh88ZoBz10j40A7uY5iM=; b=HjLPq0J+yIQJXMHr27vX/TMQKYBLG RZnV+bBUOXAnO2z41Bj6uC5N1YMBzsyUGo7k0rze4wnewpbwCDDTaijxuAgkCe0H 1Hhdgthl6vb+qVo7XTwyzKZjvJWMXwheDgoMICQqdO2e68TWAl5jnLjmgnjDHy5l pjGtVU4llu9rHlCXwCBUG8D0EmIYKqy6xeZBhqRGxBNHlDb9NAZgMdnmj/CCPbcg O30W33fMbxOQqfWmkoLWFP005Ipxv/d46hvf839kqWZNh/C/hZy28GHUtxMpep7w Opj3WALh4RXHHxuWSqzVUJndvFEHI2vmc5frYXzkFpdkwWi58RzKux7ag== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=lJ6+Jhax9UCDnizTO8oOunFeh88ZoBz10j40A7uY5iM=; b=mrs85mYU 783WhlDZyS2VS19ts/VySht//RTsH9MuPt6cxgxu38J2sqHOVWfx174tZsqmR1s1 bRMY+sYt1ewxphEVlO9ovuOU7Seqv5tK1ivvLodKHv+spBujS627lTH8R9LGofTv 32i8y0ISayQ+LpbO7An+f5I8Dw2wALL4iNbNeEFPqJDOVNgZFI7dUKTQDtTSBgSH LUjMQ9SRHbOdDfukYKSruD4hOmORAtpgGeci9H15handGorScfIh2srj7m4Ea2eH 1T6wkKOlqkrOdK60EDSlhxcI5WNUcj/POEEZ0cQwDyH5MIKwwl5n6d2v5Cq/YPY/ dt7tDkfHmRATvA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhgggfestdekre dtredttdenucfhrhhomhepifgrvghtrghnucftihhvvghtuceoghhrihhvvgesuhdvheei rdhnvghtqeenucggtffrrghtthgvrhhnpeehgfevffekteehteefieefvdehleefjeefhe evudetjefhkeeutdekieeuvdetheenucfkphepkeeirddvheegrddvgeefrddugeeinecu vehluhhsthgvrhfuihiivgepleenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 17904240066; Mon, 12 Apr 2021 11:20:32 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:05 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 19/28] ovs-thread: Fix barrier use-after-free X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When a thread is blocked on a barrier, there is no guarantee regarding the moment it will resume, only that it will at some point in the future. One thread can resume first then proceed to destroy the barrier while another thread has not yet awoken. When it finally happens, the second thread will attempt a seq_read() on the barrier seq, while the first thread have already destroyed it, triggering a use-after-free. Introduce an additional indirection layer within the barrier. A internal barrier implementation holds all the necessary elements for a thread to safely block and destroy. Whenever a barrier is destroyed, the internal implementation is left available to still blocking threads if necessary. A reference counter is used to track threads still using the implementation. Note that current uses of ovs-barrier are not affected: RCU and revalidators will not destroy their barrier immediately after blocking on it. Fixes: d8043da7182a ("ovs-thread: Implement OVS specific barrier.") Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/ovs-thread.c | 61 +++++++++++++++++++++++++++++++++++++++--------- lib/ovs-thread.h | 6 ++--- 2 files changed, 53 insertions(+), 14 deletions(-) diff --git a/lib/ovs-thread.c b/lib/ovs-thread.c index b686e4548..805cba622 100644 --- a/lib/ovs-thread.c +++ b/lib/ovs-thread.c @@ -299,21 +299,53 @@ ovs_spin_init(const struct ovs_spin *spin) } #endif +struct ovs_barrier_impl { + uint32_t size; /* Number of threads to wait. */ + atomic_count count; /* Number of threads already hit the barrier. */ + struct seq *seq; + struct ovs_refcount refcnt; +}; + +static void +ovs_barrier_impl_ref(struct ovs_barrier_impl *impl) +{ + ovs_refcount_ref(&impl->refcnt); +} + +static void +ovs_barrier_impl_unref(struct ovs_barrier_impl *impl) +{ + if (ovs_refcount_unref(&impl->refcnt) == 1) { + seq_destroy(impl->seq); + free(impl); + } +} + /* Initializes the 'barrier'. 'size' is the number of threads * expected to hit the barrier. */ void ovs_barrier_init(struct ovs_barrier *barrier, uint32_t size) { - barrier->size = size; - atomic_count_init(&barrier->count, 0); - barrier->seq = seq_create(); + struct ovs_barrier_impl *impl; + + impl = xmalloc(sizeof *impl); + impl->size = size; + atomic_count_init(&impl->count, 0); + impl->seq = seq_create(); + ovs_refcount_init(&impl->refcnt); + + ovsrcu_set(&barrier->impl, impl); } /* Destroys the 'barrier'. */ void ovs_barrier_destroy(struct ovs_barrier *barrier) { - seq_destroy(barrier->seq); + struct ovs_barrier_impl *impl; + + impl = ovsrcu_get(struct ovs_barrier_impl *, &barrier->impl); + ovsrcu_set(&barrier->impl, NULL); + ovs_barrier_impl_unref(impl); } /* Makes the calling thread block on the 'barrier' until all @@ -325,23 +357,30 @@ ovs_barrier_destroy(struct ovs_barrier *barrier) void ovs_barrier_block(struct ovs_barrier *barrier) { - uint64_t seq = seq_read(barrier->seq); + struct ovs_barrier_impl *impl; uint32_t orig; + uint64_t seq; - orig = atomic_count_inc(&barrier->count); - if (orig + 1 == barrier->size) { - atomic_count_set(&barrier->count, 0); + impl = ovsrcu_get(struct ovs_barrier_impl *, &barrier->impl); + ovs_barrier_impl_ref(impl); + + seq = seq_read(impl->seq); + orig = atomic_count_inc(&impl->count); + if (orig + 1 == impl->size) { + atomic_count_set(&impl->count, 0); /* seq_change() serves as a release barrier against the other threads, * so the zeroed count is visible to them as they continue. */ - seq_change(barrier->seq); + seq_change(impl->seq); } else { /* To prevent thread from waking up by other event, * keeps waiting for the change of 'barrier->seq'. */ - while (seq == seq_read(barrier->seq)) { - seq_wait(barrier->seq, seq); + while (seq == seq_read(impl->seq)) { + seq_wait(impl->seq, seq); poll_block(); } } + + ovs_barrier_impl_unref(impl); } DEFINE_EXTERN_PER_THREAD_DATA(ovsthread_id, OVSTHREAD_ID_UNSET); diff --git a/lib/ovs-thread.h b/lib/ovs-thread.h index 7ee98bd4e..3b444ccdc 100644 --- a/lib/ovs-thread.h +++ b/lib/ovs-thread.h @@ -21,16 +21,16 @@ #include #include #include "ovs-atomic.h" +#include "ovs-rcu.h" #include "openvswitch/thread.h" #include "util.h" struct seq; /* Poll-block()-able barrier similar to pthread_barrier_t. */ +struct ovs_barrier_impl; struct ovs_barrier { - uint32_t size; /* Number of threads to wait. */ - atomic_count count; /* Number of threads already hit the barrier. */ - struct seq *seq; + OVSRCU_TYPE(struct ovs_barrier_impl *) impl; }; /* Wrappers for pthread_mutexattr_*() that abort the process on any error. */ From patchwork Mon Apr 12 15:20:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465347 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=oPOTCZ51; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=ooQCUMey; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJss7702Xz9sTD for ; Tue, 13 Apr 2021 01:21:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 64CA1419C6; Mon, 12 Apr 2021 15:21:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Oqt5woSyFchX; Mon, 12 Apr 2021 15:21:28 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTP id A06D6405AC; Mon, 12 Apr 2021 15:20:59 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 035E7C0033; Mon, 12 Apr 2021 15:20:51 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 24B51C0034 for ; Mon, 12 Apr 2021 15:20:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 2C5B040533 for ; Mon, 12 Apr 2021 15:20:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="oPOTCZ51"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="ooQCUMey" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y5O2w-fx7cFt for ; Mon, 12 Apr 2021 15:20:34 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by smtp4.osuosl.org (Postfix) with ESMTPS id 83C2340510 for ; Mon, 12 Apr 2021 15:20:34 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id CB90F181B; Mon, 12 Apr 2021 11:20:33 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Mon, 12 Apr 2021 11:20:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=wBUeCT9SVCIUv dZbiClcThq+0Qb5RNFeAUdqpc/Nhuc=; b=oPOTCZ51Ew/iBNtY0BZQtjcNWfaKH zZXFwap38eJ5YmqfgCqM3hessaN8hSHiTziUoe76rBneJnSkdpJTEMNgThJMHNpf O/QBYO7JfMbrIn5HHIBiiH4Oa3moLKvOX1JyHyMhIEn7o5CMjpCnQm1ob3AUs9rY O/TUXR27kfENjAwGE1poPxUmmghvpm97mA/UZiqlkFIGpEYI9LrujbHfKTdaGf58 HkGgoZe76NzWGtN0fo+k+uWIGDzHySZSqNJpyl6154Z4fUooWHJeWlxCjprCr1Qv UnkT3TV8xUTqRO6RRvt3wxkB84o8HThQMJNgjiohMPz47+u9CcKxv+V5w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=wBUeCT9SVCIUvdZbiClcThq+0Qb5RNFeAUdqpc/Nhuc=; b=ooQCUMey obko7QJWidcJzU8XMUD/gwGDjP568s9nNlUWKqUlIYIn6AffQAQfbYft9gE/xyJy irSiJr8HUkG9cwajFm3lhuHSwETYDblh4bV9ADHK37MdKl1So2bgvvSZ54/yos3s qb+1gd8EzxEavXVJ5Uz7fdsiQC8FU4S6xV6+xIoshfV6++zGv1rVGDX8Msf7hwYT fLh31KSwvh6Yg1ROwRoXzyh3THDeqq9rxLKgqLx7m+o9yFWGDgJAOn1Amx1+tdCO LXfEZa6OFoWgQ+WXSiWAtcjDHMUmQe9eeP27GZ0IpMH5XrDznQqiT9uQwM+X45JZ k3AsRh46RM92xg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhgggfestdekre dtredttdenucfhrhhomhepifgrvghtrghnucftihhvvghtuceoghhrihhvvgesuhdvheei rdhnvghtqeenucggtffrrghtthgvrhhnpeehgfevffekteehteefieefvdehleefjeefhe evudetjefhkeeutdekieeuvdetheenucfkphepkeeirddvheegrddvgeefrddugeeinecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id B9CF3240054; Mon, 12 Apr 2021 11:20:32 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:06 +0200 Message-Id: <16192985e173094abb46836901f5e811acb651e5.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 20/28] dpif-netdev: Execute flush from offload thread X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When a port is deleted, its offloads must be flushed. The operation runs in the thread that initiated it. Offload data is thus accessed jointly by the port deletion thread(s) and the offload thread, which complicates the data access model. To simplify this model, as a pre-step toward introducing parallel offloads, execute the flush operation in the offload thread. Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 126 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 122 insertions(+), 4 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index b8fccdb43..9d473c5c2 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -422,6 +422,7 @@ enum rxq_cycles_counter_type { enum dp_offload_type { DP_OFFLOAD_FLOW, + DP_OFFLOAD_FLUSH, }; enum { @@ -439,8 +440,15 @@ struct dp_offload_flow_item { size_t actions_len; }; +struct dp_offload_flush_item { + struct dp_netdev *dp; + struct netdev *netdev; + struct ovs_barrier *barrier; +}; + union dp_offload_thread_data { struct dp_offload_flow_item flow; + struct dp_offload_flush_item flush; }; struct dp_offload_thread_item { @@ -905,6 +913,9 @@ static void dp_netdev_del_bond_tx_from_pmd(struct dp_netdev_pmd_thread *pmd, uint32_t bond_id) OVS_EXCLUDED(pmd->bond_mutex); +static void dp_netdev_offload_flush(struct dp_netdev *dp, + struct dp_netdev_port *port); + static void reconfigure_datapath(struct dp_netdev *dp) OVS_REQUIRES(dp->port_mutex); static bool dp_netdev_pmd_try_ref(struct dp_netdev_pmd_thread *pmd); @@ -2305,7 +2316,7 @@ static void do_del_port(struct dp_netdev *dp, struct dp_netdev_port *port) OVS_REQUIRES(dp->port_mutex) { - netdev_flow_flush(port->netdev); + dp_netdev_offload_flush(dp, port); netdev_uninit_flow_api(port->netdev); hmap_remove(&dp->ports, &port->node); seq_change(dp->port_seq); @@ -2675,13 +2686,16 @@ dp_netdev_free_offload(struct dp_offload_thread_item *offload) case DP_OFFLOAD_FLOW: dp_netdev_free_flow_offload(offload); break; + case DP_OFFLOAD_FLUSH: + free(offload); + break; default: OVS_NOT_REACHED(); }; } static void -dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) +dp_netdev_append_offload(struct dp_offload_thread_item *offload) { ovs_mutex_lock(&dp_offload_thread.mutex); ovs_list_push_back(&dp_offload_thread.list, &offload->node); @@ -2814,6 +2828,23 @@ dp_offload_flow(struct dp_offload_thread_item *item) UUID_ARGS((struct uuid *) &flow_offload->flow->mega_ufid)); } +static void +dp_offload_flush(struct dp_offload_thread_item *item) +{ + struct dp_offload_flush_item *flush = &item->data->flush; + + ovs_mutex_lock(&flush->dp->port_mutex); + netdev_flow_flush(flush->netdev); + ovs_mutex_unlock(&flush->dp->port_mutex); + + ovs_barrier_block(flush->barrier); + + /* Allow the other thread to take again the port lock, before + * continuing offload operations in this thread. + */ + ovs_barrier_block(flush->barrier); +} + #define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ static void * @@ -2844,6 +2875,9 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) case DP_OFFLOAD_FLOW: dp_offload_flow(offload); break; + case DP_OFFLOAD_FLUSH: + dp_offload_flush(offload); + break; default: OVS_NOT_REACHED(); } @@ -2882,7 +2916,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, offload = dp_netdev_alloc_flow_offload(pmd, flow, DP_NETDEV_FLOW_OFFLOAD_OP_DEL); offload->timestamp = pmd->ctx.now; - dp_netdev_append_flow_offload(offload); + dp_netdev_append_offload(offload); } static void @@ -2917,7 +2951,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, flow_offload->actions_len = actions_len; item->timestamp = pmd->ctx.now; - dp_netdev_append_flow_offload(item); + dp_netdev_append_offload(item); } static void @@ -2941,6 +2975,90 @@ dp_netdev_pmd_remove_flow(struct dp_netdev_pmd_thread *pmd, dp_netdev_flow_unref(flow); } +static void +dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, + struct netdev *netdev, + struct ovs_barrier *barrier) +{ + struct dp_offload_thread_item *item; + struct dp_offload_flush_item *flush; + + if (ovsthread_once_start(&offload_thread_once)) { + xpthread_cond_init(&dp_offload_thread.cond, NULL); + ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); + ovsthread_once_done(&offload_thread_once); + } + + item = xmalloc(sizeof *item + sizeof *flush); + item->type = DP_OFFLOAD_FLUSH; + item->timestamp = time_usec(); + + flush = &item->data->flush; + flush->dp = dp; + flush->netdev = netdev; + flush->barrier = barrier; + + dp_netdev_append_offload(item); +} + +/* Blocking call that will wait on the offload thread to + * complete its work. As the flush order will only be + * enqueued after existing offload requests, those previous + * offload requests must be processed, which requires being + * able to lock the 'port_mutex' from the offload thread. + * + * Flow offload flush is done when a port is being deleted. + * Right after this call executes, the offload API is disabled + * for the port. This call must be made blocking until the + * offload provider completed its job. + */ +static void +dp_netdev_offload_flush(struct dp_netdev *dp, + struct dp_netdev_port *port) + OVS_REQUIRES(dp->port_mutex) +{ + /* The flush mutex only serves to protect the static memory barrier. + * The memory barrier needs to go beyond the function scope as + * the other thread can resume from blocking after this function + * already finished. + * As the barrier is made static, then it will be shared by + * calls to this function, and it needs to be protected from + * concurrent use. + */ + static struct ovs_mutex flush_mutex = OVS_MUTEX_INITIALIZER; + static struct ovs_barrier barrier OVS_GUARDED_BY(flush_mutex); + struct netdev *netdev; + + if (!netdev_is_flow_api_enabled()) { + return; + } + + ovs_mutex_unlock(&dp->port_mutex); + ovs_mutex_lock(&flush_mutex); + + /* This thread and the offload thread. */ + ovs_barrier_init(&barrier, 2); + + netdev = netdev_ref(port->netdev); + dp_netdev_offload_flush_enqueue(dp, netdev, &barrier); + ovs_barrier_block(&barrier); + netdev_close(netdev); + + /* Take back the datapath port lock before allowing the offload + * thread to proceed further. The port deletion must complete first, + * to ensure no further offloads are inserted after the flush. + * + * Some offload provider (e.g. DPDK) keeps a netdev reference with + * the offload data. If this reference is not closed, the netdev is + * kept indefinitely. */ + ovs_mutex_lock(&dp->port_mutex); + + ovs_barrier_block(&barrier); + ovs_barrier_destroy(&barrier); + + ovs_mutex_unlock(&flush_mutex); +} + static void dp_netdev_pmd_flow_flush(struct dp_netdev_pmd_thread *pmd) { From patchwork Mon Apr 12 15:20:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465352 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=kj2YhxAC; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=Oh+k8GGw; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssR4zDpz9rx6 for ; Tue, 13 Apr 2021 01:21:47 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 266C483CA0; Mon, 12 Apr 2021 15:21:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0iBQKDuF39ry; Mon, 12 Apr 2021 15:21:44 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 93A1883C1D; Mon, 12 Apr 2021 15:21:01 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C61EEC0019; Mon, 12 Apr 2021 15:20:52 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 16B0EC002B for ; Mon, 12 Apr 2021 15:20:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1883460AC2 for ; Mon, 12 Apr 2021 15:20:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="kj2YhxAC"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="Oh+k8GGw" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YmaP0V-Ba3Rz for ; Mon, 12 Apr 2021 15:20:35 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp3.osuosl.org (Postfix) with ESMTPS id 0193560AD6 for ; Mon, 12 Apr 2021 15:20:34 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 237D85C01CA; Mon, 12 Apr 2021 11:20:34 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=cMS1MnF/DQMm7 j93Dec2moyo3WaWLSRT2w6g0Bev1OI=; b=kj2YhxAC+ROKGFcw22edoGp1T/kHN i2VpgtDkja3U0Yyr1noReDhn8gc26+9KZgTwf2M1vjwn8BzvAbNqJqqHSPFzGnm8 uejKNdXbej5FetaySyGlYUyfZx66X/2xqncPrMIbNnYCoDXsg/JiuuaOyDLlTw5E tBPLvn0CvX2lGU+z9s0Y+SzzsA5ePtSUmoznWFY3U+h7zORtJDhvJ+tlKox4YBzs dBdoZGfRijZp3tIswhlAIdr3ZWe/OMBaxq35H8JLLoBlf/EY/045uskfOBEYCeSY hPEoF84yPIx2DNGESpZMTeuRqtvLYdxpJOd179N1Cz5KEBCadyrAsm3kw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=cMS1MnF/DQMm7j93Dec2moyo3WaWLSRT2w6g0Bev1OI=; b=Oh+k8GGw KGDul5vo+MVA3kJJFkfahgPohnd/dtvHhZbeaGWWjxvt2OGIvWOU+1DZQ7TIg61v B97BT1J5v9Xj6ZdQOl2Up7TpmH6T5YTCumKrArt9shrrPbuyblYRH8vmnPAivina 8rVpR5P+dppNN5yqcZ/V8TyYixtkL9HQcw1JuP5k1cLjbU5xq3ZBqr5HC8RR3j11 8ym5pG5eTY8Wiq2YY7nrh6WwQJkP+rspgr2Zzl+FUnCuvJNmoGNgCpCQy/cDjm+/ hFztRB1PLn9ULbV27nBYlJirsSvcJv37RihELs+r/IxwQcycI4YmNzx8MLTP3JPU pT+bc2n2wsXsfw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpeelnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 66A6E240066; Mon, 12 Apr 2021 11:20:33 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:07 +0200 Message-Id: <0351f2edc90f7a6f5ef09f6df1e7f7cbc6e322ac.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 21/28] netdev-offload-dpdk: Use per-thread HW offload stats X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The implementation of hardware offload counters in currently meant to be managed by a single thread. Use the offload thread pool API to manage one counter per thread. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-dpdk.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index c43e8b968..ecdc846e1 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -65,7 +65,7 @@ struct ufid_to_rte_flow_data { struct netdev_offload_dpdk_data { struct cmap ufid_to_rte_flow; - uint64_t rte_flow_counter; + uint64_t *rte_flow_counters; }; static int @@ -75,6 +75,8 @@ offload_data_init(struct netdev *netdev) data = xzalloc(sizeof *data); cmap_init(&data->ufid_to_rte_flow); + data->rte_flow_counters = xcalloc(netdev_offload_thread_nb(), + sizeof *data->rte_flow_counters); ovsrcu_set(&netdev->hw_info.offload_data, (void *) data); @@ -84,6 +86,7 @@ offload_data_init(struct netdev *netdev) static void offload_data_destroy__(struct netdev_offload_dpdk_data *data) { + free(data->rte_flow_counters); free(data); } @@ -646,10 +649,11 @@ netdev_offload_dpdk_flow_create(struct netdev *netdev, flow = netdev_dpdk_rte_flow_create(netdev, attr, items, actions, error); if (flow) { struct netdev_offload_dpdk_data *data; + unsigned int tid = netdev_offload_thread_id(); data = (struct netdev_offload_dpdk_data *) ovsrcu_get(void *, &netdev->hw_info.offload_data); - data->rte_flow_counter++; + data->rte_flow_counters[tid]++; if (!VLOG_DROP_DBG(&rl)) { dump_flow(&s, &s_extra, attr, items, actions); @@ -1532,10 +1536,11 @@ netdev_offload_dpdk_flow_destroy(struct ufid_to_rte_flow_data *rte_flow_data) if (ret == 0) { struct netdev_offload_dpdk_data *data; + unsigned int tid = netdev_offload_thread_id(); data = (struct netdev_offload_dpdk_data *) ovsrcu_get(void *, &netdev->hw_info.offload_data); - data->rte_flow_counter--; + data->rte_flow_counters[tid]--; ufid_to_rte_flow_disassociate(rte_flow_data); VLOG_DBG_RL(&rl, "%s: rte_flow 0x%"PRIxPTR @@ -1698,6 +1703,7 @@ netdev_offload_dpdk_get_n_flows(struct netdev *netdev, uint64_t *n_flows) { struct netdev_offload_dpdk_data *data; + unsigned int tid; data = (struct netdev_offload_dpdk_data *) ovsrcu_get(void *, &netdev->hw_info.offload_data); @@ -1705,7 +1711,9 @@ netdev_offload_dpdk_get_n_flows(struct netdev *netdev, return -1; } - *n_flows = data->rte_flow_counter; + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + n_flows[tid] = data->rte_flow_counters[tid]; + } return 0; } From patchwork Mon Apr 12 15:20:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465348 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=AYzkXoTb; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=SluLAWD/; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssB2KbPz9rx6 for ; Tue, 13 Apr 2021 01:21:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3540460B15; Mon, 12 Apr 2021 15:21:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wp-kt95IIvBP; Mon, 12 Apr 2021 15:21:27 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id 83B0F60BAD; Mon, 12 Apr 2021 15:21:14 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D63E1C001B; Mon, 12 Apr 2021 15:21:11 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1B5FBC0019 for ; Mon, 12 Apr 2021 15:21:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id E24BB83B58 for ; Mon, 12 Apr 2021 15:20:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="AYzkXoTb"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="SluLAWD/" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3itrXQtdsEv3 for ; Mon, 12 Apr 2021 15:20:37 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp1.osuosl.org (Postfix) with ESMTPS id 94A0E83BFC for ; Mon, 12 Apr 2021 15:20:35 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id C25565C01C8; Mon, 12 Apr 2021 11:20:34 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 12 Apr 2021 11:20:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=bdJOotF/1AlCF mXutZEt+BAW+l/b9mtV8/lIxFJyJ08=; b=AYzkXoTbqE98OuzwhS3/ZNqB1PuL/ 97bZsiJBDb8FgxDSyAJ4Jpr2YSW6J2hRNleBrTt6lljTv8noRTq7HziNIzXh6UaX u7RrmbhRfofm7GJrqaRwXbo/U/x5SosH/vLFUm6hSQtXMausiSNgsPR1tMcdQ3PZ motXFC6vdiHSwc5qrcbCVY78Swkebgs5Rxm69WFR08vhkJK+VnGdxZnB3PFY01nB JcuJ9Zt9WLMCzyfTaWgRVzBmK3ihaGe/J4Yi08tBxiKEpogDN/L2R3NTNaTLqEzD 45rKwIo6UsAFsPIp9DGnlenIBbEpgKyIFTJT9IIjK6YruQFiXCy8pVF+g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=bdJOotF/1AlCFmXutZEt+BAW+l/b9mtV8/lIxFJyJ08=; b=SluLAWD/ a0sxTMm4dI0IdCqyNdTv2XT0rOhJBW+/skMDOIIlXiMlbB28nPCLONDtqOLDwFO+ XyYl2WuynCl+7W3V5mH1m6i8f/aVl0EjTDwoMIrP2B0cnLdcjpPNiBDWre1ePIJ3 vFh77y8F2DJc6tuS67gR7fbyVv6dkMQH8YVLbnJ9ydRDWFYz4CWGaL++rhzN14AT DvBwroVZgySH5fevQqY7CC9gGUOXf2NiopYsrdHurM4/b30mPbE7NEcSnRtObs8T z6nXKZd0yXLHPtDw+IFuZHZFKo9bgL2yXk+V7JGeaOPQOjphscMOdZhMAXfLb7jW XY3MQB6WDcrNbw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 2C5F9240066; Mon, 12 Apr 2021 11:20:34 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:08 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 22/28] netdev-offload-dpdk: Lock rte_flow map access X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a lock to access the ufid to rte_flow map. This will protect it from concurrent write accesses when multiple threads attempt it. At this point, the reason for taking the lock is not to fullfill the needs of the DPDK offload implementation anymore. Rewrite the comments to reflect this change. The lock is still needed to protect against changes to netdev port mapping. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 8 ++--- lib/netdev-offload-dpdk.c | 61 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 61 insertions(+), 8 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 9d473c5c2..177d6a6dc 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2590,7 +2590,7 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, port = netdev_ports_get(in_port, dpif_type_str); if (port) { /* Taking a global 'port_mutex' to fulfill thread safety - * restrictions for the netdev-offload-dpdk module. */ + * restrictions regarding netdev port mapping. */ ovs_mutex_lock(&pmd->dp->port_mutex); ret = netdev_flow_del(port, &flow->mega_ufid, NULL); ovs_mutex_unlock(&pmd->dp->port_mutex); @@ -2770,8 +2770,8 @@ dp_netdev_flow_offload_put(struct dp_offload_flow_item *offload) netdev_close(port); goto err_free; } - /* Taking a global 'port_mutex' to fulfill thread safety restrictions for - * the netdev-offload-dpdk module. */ + /* Taking a global 'port_mutex' to fulfill thread safety + * restrictions regarding the netdev port mapping. */ ovs_mutex_lock(&pmd->dp->port_mutex); ret = netdev_flow_put(port, &offload->match, CONST_CAST(struct nlattr *, offload->actions), @@ -3574,7 +3574,7 @@ dpif_netdev_get_flow_offload_status(const struct dp_netdev *dp, } ofpbuf_use_stack(&buf, &act_buf, sizeof act_buf); /* Taking a global 'port_mutex' to fulfill thread safety - * restrictions for the netdev-offload-dpdk module. + * restrictions regarding netdev port mapping. * * XXX: Main thread will try to pause/stop all revalidators during datapath * reconfiguration via datapath purge callback (dp_purge_cb) while diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index ecdc846e1..4459a0aa1 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -38,9 +38,6 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(100, 5); * * Below API is NOT thread safe in following terms: * - * - The caller must be sure that none of these functions will be called - * simultaneously. Even for different 'netdev's. - * * - The caller must be sure that 'netdev' will not be destructed/deallocated. * * - The caller must be sure that 'netdev' configuration will not be changed. @@ -66,6 +63,7 @@ struct ufid_to_rte_flow_data { struct netdev_offload_dpdk_data { struct cmap ufid_to_rte_flow; uint64_t *rte_flow_counters; + struct ovs_mutex map_lock; }; static int @@ -74,6 +72,7 @@ offload_data_init(struct netdev *netdev) struct netdev_offload_dpdk_data *data; data = xzalloc(sizeof *data); + ovs_mutex_init(&data->map_lock); cmap_init(&data->ufid_to_rte_flow); data->rte_flow_counters = xcalloc(netdev_offload_thread_nb(), sizeof *data->rte_flow_counters); @@ -86,6 +85,7 @@ offload_data_init(struct netdev *netdev) static void offload_data_destroy__(struct netdev_offload_dpdk_data *data) { + ovs_mutex_destroy(&data->map_lock); free(data->rte_flow_counters); free(data); } @@ -117,6 +117,34 @@ offload_data_destroy(struct netdev *netdev) ovsrcu_set(&netdev->hw_info.offload_data, NULL); } +static void +offload_data_lock(struct netdev *netdev) + OVS_NO_THREAD_SAFETY_ANALYSIS +{ + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + if (!data) { + return; + } + ovs_mutex_lock(&data->map_lock); +} + +static void +offload_data_unlock(struct netdev *netdev) + OVS_NO_THREAD_SAFETY_ANALYSIS +{ + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + if (!data) { + return; + } + ovs_mutex_unlock(&data->map_lock); +} + static struct cmap * offload_data_map(struct netdev *netdev) { @@ -155,6 +183,24 @@ ufid_to_rte_flow_data_find(struct netdev *netdev, return NULL; } +/* Find rte_flow with @ufid, lock-protected. */ +static struct ufid_to_rte_flow_data * +ufid_to_rte_flow_data_find_protected(struct netdev *netdev, + const ovs_u128 *ufid) +{ + size_t hash = hash_bytes(ufid, sizeof *ufid, 0); + struct ufid_to_rte_flow_data *data; + struct cmap *map = offload_data_map(netdev); + + CMAP_FOR_EACH_WITH_HASH_PROTECTED (data, node, hash, map) { + if (ovs_u128_equals(*ufid, data->ufid)) { + return data; + } + } + + return NULL; +} + static inline struct ufid_to_rte_flow_data * ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, struct rte_flow *rte_flow, bool actions_offloaded) @@ -170,13 +216,15 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, data = xzalloc(sizeof *data); + offload_data_lock(netdev); + /* * We should not simply overwrite an existing rte flow. * We should have deleted it first before re-adding it. * Thus, if following assert triggers, something is wrong: * the rte_flow is not destroyed. */ - data_prev = ufid_to_rte_flow_data_find(netdev, ufid, false); + data_prev = ufid_to_rte_flow_data_find_protected(netdev, ufid); if (data_prev) { ovs_assert(data_prev->rte_flow == NULL); } @@ -187,6 +235,8 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, data->actions_offloaded = actions_offloaded; cmap_insert(map, CONST_CAST(struct cmap_node *, &data->node), hash); + + offload_data_unlock(netdev); return data; } @@ -200,7 +250,10 @@ ufid_to_rte_flow_disassociate(struct ufid_to_rte_flow_data *data) return; } + offload_data_lock(data->netdev); cmap_remove(map, CONST_CAST(struct cmap_node *, &data->node), hash); + offload_data_unlock(data->netdev); + netdev_close(data->netdev); ovsrcu_postpone(free, data); } From patchwork Mon Apr 12 15:20:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465354 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=PYiWtzTk; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=RFgBqUXj; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssV5Hdqz9rx6 for ; Tue, 13 Apr 2021 01:21:50 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 0145583E83; Mon, 12 Apr 2021 15:21:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M8vMPFEJ9Dj6; Mon, 12 Apr 2021 15:21:47 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 164FC83D5B; Mon, 12 Apr 2021 15:21:03 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D8B52C0015; Mon, 12 Apr 2021 15:20:57 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0EFC7C0015 for ; Mon, 12 Apr 2021 15:20:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id DA01F404D8 for ; Mon, 12 Apr 2021 15:20:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="PYiWtzTk"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="RFgBqUXj" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OgJnA0SBVQ2p for ; Mon, 12 Apr 2021 15:20:38 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp2.osuosl.org (Postfix) with ESMTPS id 76106404C3 for ; Mon, 12 Apr 2021 15:20:36 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 87AC15C01D2; Mon, 12 Apr 2021 11:20:35 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 12 Apr 2021 11:20:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=ZtF5wBhOsIN4z 4sFV9E5rL96q5g/WugMhY7B8OAla20=; b=PYiWtzTk0M4BUaPdLB6NCKXMJyNm1 GfbDk0CKAIGnNjnnSLPe5dmc6E+YDLEBwUVZnvB2xBPYdaelzYUb85Zukzx3fE6p v074eHrHLXgPIyvESZ+0nztSPEo88lcZVx8RBpjBH38d/wGcwaZzUecCY5Exnfso NV+lqZ/z7Q31l2xWkPq+pQ9yX7N6JlV8dDHilAFBqSkVX6twB2sil6kRfQPFfmHs Y2gEpETEFxiJPjZ+WeD6hg4CDCiKbLSXIn8wTmFxvtGM9oxDUZEKM2upLKQR5Ra1 Ehyr3KXI9guj9cLjMGLKjTxrzut6KwFEXPMO0ClWbay0QUSob4PhMDizA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=ZtF5wBhOsIN4z4sFV9E5rL96q5g/WugMhY7B8OAla20=; b=RFgBqUXj UDlv0eeDerZX8Cj86PtFCorPjI6Q6FVOUFi7gndaKrXo0tpQLR5p1CrDlCkSGoRC ojcw163kz0aJ2VhJi4+LalM7mOK6MSYavGDRUTir1AxB9TaOIxYwEDxcK4ZB18kP OUcfWhGfpar6807g3fhPZyRsXNWa61LJE0Rt0ZcyIYnvhv97jIT8zkeDhIXjA4VZ 8/cTElcVvm4JdRMpkaH46xxreRB52E0HhFL6tYyxLg8Z8mQSn7g0J1rg9kMrKqYy 2RJkR+6dVQ/g/TRnBoL9zaxmq6bzVRKfz5ODjBdke/wtnxyfZ4qvpFQJi50Q8TbX j9HcwdFoODuTNg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id E3732240054; Mon, 12 Apr 2021 11:20:34 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:09 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 23/28] netdev-offload-dpdk: Protect concurrent offload destroy/query X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The rte_flow API in DPDK is now thread safe for insertion and deletion. It is not however safe for concurrent query while the offload is being inserted or deleted. Insertion is not an issue as the rte_flow handle will be published to other threads only once it has been inserted in the hardware, so the query will only be able to proceed once it is already available. For the deletion path however, offload status queries can be made while an offload is being destroyed. This would create race conditions and use-after-free if not properly protected. As a pre-step before removing the OVS-level locks on the rte_flow API, mutually exclude offload query and deletion from concurrent execution. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-dpdk.c | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index 4459a0aa1..13e017ef8 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -58,6 +58,8 @@ struct ufid_to_rte_flow_data { struct rte_flow *rte_flow; bool actions_offloaded; struct dpif_flow_stats stats; + struct ovs_mutex lock; + bool dead; }; struct netdev_offload_dpdk_data { @@ -233,6 +235,7 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, data->netdev = netdev_ref(netdev); data->rte_flow = rte_flow; data->actions_offloaded = actions_offloaded; + ovs_mutex_init(&data->lock); cmap_insert(map, CONST_CAST(struct cmap_node *, &data->node), hash); @@ -240,8 +243,16 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, return data; } +static void +rte_flow_data_unref(struct ufid_to_rte_flow_data *data) +{ + ovs_mutex_destroy(&data->lock); + free(data); +} + static inline void ufid_to_rte_flow_disassociate(struct ufid_to_rte_flow_data *data) + OVS_REQUIRES(data->lock) { size_t hash = hash_bytes(&data->ufid, sizeof data->ufid, 0); struct cmap *map = offload_data_map(data->netdev); @@ -255,7 +266,7 @@ ufid_to_rte_flow_disassociate(struct ufid_to_rte_flow_data *data) offload_data_unlock(data->netdev); netdev_close(data->netdev); - ovsrcu_postpone(free, data); + ovsrcu_postpone(rte_flow_data_unref, data); } /* @@ -1581,6 +1592,15 @@ netdev_offload_dpdk_flow_destroy(struct ufid_to_rte_flow_data *rte_flow_data) ovs_u128 *ufid; int ret; + ovs_mutex_lock(&rte_flow_data->lock); + + if (rte_flow_data->dead) { + ovs_mutex_unlock(&rte_flow_data->lock); + return 0; + } + + rte_flow_data->dead = true; + rte_flow = rte_flow_data->rte_flow; netdev = rte_flow_data->netdev; ufid = &rte_flow_data->ufid; @@ -1607,6 +1627,8 @@ netdev_offload_dpdk_flow_destroy(struct ufid_to_rte_flow_data *rte_flow_data) UUID_ARGS((struct uuid *) ufid)); } + ovs_mutex_unlock(&rte_flow_data->lock); + return ret; } @@ -1702,8 +1724,19 @@ netdev_offload_dpdk_flow_get(struct netdev *netdev, struct rte_flow_error error; int ret = 0; + attrs->dp_extra_info = NULL; + rte_flow_data = ufid_to_rte_flow_data_find(netdev, ufid, false); - if (!rte_flow_data || !rte_flow_data->rte_flow) { + if (!rte_flow_data || !rte_flow_data->rte_flow || + rte_flow_data->dead || ovs_mutex_trylock(&rte_flow_data->lock)) { + return -1; + } + + /* Check again whether the data is dead, as it could have been + * updated while the lock was not yet taken. The first check above + * was only to avoid unnecessary locking if possible. + */ + if (rte_flow_data->dead) { ret = -1; goto out; } @@ -1730,7 +1763,7 @@ netdev_offload_dpdk_flow_get(struct netdev *netdev, } memcpy(stats, &rte_flow_data->stats, sizeof *stats); out: - attrs->dp_extra_info = NULL; + ovs_mutex_unlock(&rte_flow_data->lock); return ret; } From patchwork Mon Apr 12 15:20:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465356 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=k7J39ye5; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=ovoyoVjD; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssk4w9Vz9rx6 for ; Tue, 13 Apr 2021 01:22:02 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 20CDF83D90; Mon, 12 Apr 2021 15:22:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 996f0tCY5dTY; Mon, 12 Apr 2021 15:21:58 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTP id 97B5983C98; Mon, 12 Apr 2021 15:21:09 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4CC14C000A; Mon, 12 Apr 2021 15:21:09 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 156ADC000A for ; Mon, 12 Apr 2021 15:21:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id A7129404F9 for ; Mon, 12 Apr 2021 15:20:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="k7J39ye5"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="ovoyoVjD" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aY26EmkvJR_c for ; Mon, 12 Apr 2021 15:20:39 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1DC62404CA for ; Mon, 12 Apr 2021 15:20:37 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 4A9575C01AF; Mon, 12 Apr 2021 11:20:36 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 12 Apr 2021 11:20:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=o4YhGSw5RiOG9 tTXYBsoVhQ6qpB5KnulGVPSrNU1vWM=; b=k7J39ye5dIwdAaz1TvPLWSWU3r2IA dEbZJPlNLDSd4SeU6WfF0hW1qQrE2nBDJMIYsLJy4p72MUtZyAbEj6PEqBv9cR4k ACYSkw1+tNIaaIA4jMXsgC6ey7kdavdl9wCZ4uoBCZVkh9DdnXXDFXCPb6HlCWtP zjLbUG9WoFAvid1I5DgkMdJEObu+6Zi8yxaleT7n7mLzHSCkLtjZevmlqOxq3usl nED3XPMJlyTIaxzfSjDAUxSyB4aaTwH3bUqm950iuloNHSXMSYMDYrp1nCjhXSV7 2OWgi7oUZGFHgVpEGDvqzetRTCZqharuUAptxXOPIFGzUiw0TDVbkL8ng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=o4YhGSw5RiOG9tTXYBsoVhQ6qpB5KnulGVPSrNU1vWM=; b=ovoyoVjD dFSpagS5VRO1cHrwyap8YMFoj/ad+WxWRRpI0jsqsq4MBBxMGiGVIR1oIVn5e48X qUVWXKVEE2WGF0WorHcFDFmaPlOpn9e4v19GgAxX7QDKmAPtL/Iq7YkO9599Gt37 QxpXX8WzlgaVmGSWOm3a2gsI3g5ULptO2dKR3SLJMDpbLWBxYGRVjSIiLprVDAWC IxH4v0B+fcIlC5cbUDLG/9gXgW5kTkDK0AmU7FLER4Hv4LG9US/4yoY0qBR+QqzN ZPuHkPR7pPXxZZQa5M9akCcpmjcEOy95S/DjcMjGT9kslMTvJ0wiQOSYvJBBsVDY 5emOkqkHYjQOpg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpeefnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id A9FAB240054; Mon, 12 Apr 2021 11:20:35 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:10 +0200 Message-Id: <4524c29212a057777549dfa959edc7c984fe3407.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 24/28] dpif-netdev: Use lockless queue to manage offloads X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The dataplane threads (PMDs) send offloading commands to a dedicated offload management thread. The current implementation uses a lock and benchmarks show a high contention on the queue in some cases. With high-contention, the mutex will more often lead to the locking thread yielding in wait, using a syscall. This should be avoided in a userland dataplane. The mpsc-queue can be used instead. It uses less cycles and has lower latency. Benchmarks show better behavior as multiple revalidators and one or multiple PMDs writes to a single queue while another thread polls it. One trade-off with the new scheme however is to be forced to poll the queue from the offload thread. Without mutex, a cond_wait cannot be used for signaling. The offload thread is implementing an exponential backoff and will sleep in short increments when no data is available. This makes the thread yield, at the price of some latency to manage offloads after an inactivity period. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 109 ++++++++++++++++++++++++---------------------- 1 file changed, 57 insertions(+), 52 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 177d6a6dc..bc87b368b 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -52,6 +52,7 @@ #include "id-pool.h" #include "ipf.h" #include "mov-avg.h" +#include "mpsc-queue.h" #include "netdev.h" #include "netdev-offload.h" #include "netdev-provider.h" @@ -452,25 +453,22 @@ union dp_offload_thread_data { }; struct dp_offload_thread_item { - struct ovs_list node; + struct mpsc_queue_node node; enum dp_offload_type type; long long int timestamp; union dp_offload_thread_data data[0]; }; struct dp_offload_thread { - struct ovs_mutex mutex; - struct ovs_list list; - uint64_t enqueued_item; + struct mpsc_queue queue; + atomic_uint64_t enqueued_item; struct mov_avg_cma cma; struct mov_avg_ema ema; - pthread_cond_t cond; }; static struct dp_offload_thread dp_offload_thread = { - .mutex = OVS_MUTEX_INITIALIZER, - .list = OVS_LIST_INITIALIZER(&dp_offload_thread.list), - .enqueued_item = 0, + .queue = MPSC_QUEUE_INITIALIZER(&dp_offload_thread.queue), + .enqueued_item = ATOMIC_VAR_INIT(0), .cma = MOV_AVG_CMA_INITIALIZER, .ema = MOV_AVG_EMA_INITIALIZER(100), }; @@ -2697,11 +2695,8 @@ dp_netdev_free_offload(struct dp_offload_thread_item *offload) static void dp_netdev_append_offload(struct dp_offload_thread_item *offload) { - ovs_mutex_lock(&dp_offload_thread.mutex); - ovs_list_push_back(&dp_offload_thread.list, &offload->node); - dp_offload_thread.enqueued_item++; - xpthread_cond_signal(&dp_offload_thread.cond); - ovs_mutex_unlock(&dp_offload_thread.mutex); + mpsc_queue_insert(&dp_offload_thread.queue, &offload->node); + atomic_count_inc64(&dp_offload_thread.enqueued_item); } static int @@ -2845,59 +2840,69 @@ dp_offload_flush(struct dp_offload_thread_item *item) ovs_barrier_block(flush->barrier); } +#define DP_NETDEV_OFFLOAD_BACKOFF_MIN 1 +#define DP_NETDEV_OFFLOAD_BACKOFF_MAX 64 #define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ static void * dp_netdev_flow_offload_main(void *data OVS_UNUSED) { struct dp_offload_thread_item *offload; - struct ovs_list *list; + struct mpsc_queue_node *node; + struct mpsc_queue *queue; long long int latency_us; long long int next_rcu; long long int now; + uint64_t backoff; - next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; - for (;;) { - ovs_mutex_lock(&dp_offload_thread.mutex); - if (ovs_list_is_empty(&dp_offload_thread.list)) { - ovsrcu_quiesce_start(); - ovs_mutex_cond_wait(&dp_offload_thread.cond, - &dp_offload_thread.mutex); - ovsrcu_quiesce_end(); - next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; - } - list = ovs_list_pop_front(&dp_offload_thread.list); - dp_offload_thread.enqueued_item--; - offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); - ovs_mutex_unlock(&dp_offload_thread.mutex); - - switch (offload->type) { - case DP_OFFLOAD_FLOW: - dp_offload_flow(offload); - break; - case DP_OFFLOAD_FLUSH: - dp_offload_flush(offload); - break; - default: - OVS_NOT_REACHED(); + queue = &dp_offload_thread.queue; + mpsc_queue_acquire(queue); + + while (true) { + backoff = DP_NETDEV_OFFLOAD_BACKOFF_MIN; + while (mpsc_queue_tail(queue) == NULL) { + xnanosleep(backoff * 1E6); + if (backoff < DP_NETDEV_OFFLOAD_BACKOFF_MAX) { + backoff <<= 1; + } } - now = time_usec(); + next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; + MPSC_QUEUE_FOR_EACH_POP (node, queue) { + offload = CONTAINER_OF(node, struct dp_offload_thread_item, node); + atomic_count_dec64(&dp_offload_thread.enqueued_item); - latency_us = now - offload->timestamp; - mov_avg_cma_update(&dp_offload_thread.cma, latency_us); - mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + switch (offload->type) { + case DP_OFFLOAD_FLOW: + dp_offload_flow(offload); + break; + case DP_OFFLOAD_FLUSH: + dp_offload_flush(offload); + break; + default: + OVS_NOT_REACHED(); + } - dp_netdev_free_offload(offload); + now = time_usec(); - /* Do RCU synchronization at fixed interval. */ - if (now > next_rcu) { - if (!ovsrcu_try_quiesce()) { - next_rcu += DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; + latency_us = now - offload->timestamp; + mov_avg_cma_update(&dp_offload_thread.cma, latency_us); + mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + + dp_netdev_free_offload(offload); + + /* Do RCU synchronization at fixed interval. */ + if (now > next_rcu) { + if (!ovsrcu_try_quiesce()) { + next_rcu += DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; + } } } } + OVS_NOT_REACHED(); + mpsc_queue_release(queue); + return NULL; } @@ -2908,7 +2913,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, struct dp_offload_thread_item *offload; if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_offload_thread.cond, NULL); + mpsc_queue_init(&dp_offload_thread.queue); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -2933,7 +2938,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, } if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_offload_thread.cond, NULL); + mpsc_queue_init(&dp_offload_thread.queue); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -2984,7 +2989,7 @@ dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, struct dp_offload_flush_item *flush; if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_offload_thread.cond, NULL); + mpsc_queue_init(&dp_offload_thread.queue); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -4471,8 +4476,8 @@ dpif_netdev_offload_stats_get(struct dpif *dpif, } ovs_mutex_unlock(&dp->port_mutex); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value = - dp_offload_thread.enqueued_item; + atomic_read_relaxed(&dp_offload_thread.enqueued_item, + &stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value); stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED].value = nb_offloads; stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].value = mov_avg_cma(&dp_offload_thread.cma); From patchwork Mon Apr 12 15:20:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465355 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=B1TOlIkl; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=o02D80ai; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssW3pclz9sTD for ; Tue, 13 Apr 2021 01:21:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 81C5341A1C; Mon, 12 Apr 2021 15:21:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cquYs5S4pf6X; Mon, 12 Apr 2021 15:21:42 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTP id A6AD1405F0; Mon, 12 Apr 2021 15:21:12 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 24C09C001A; Mon, 12 Apr 2021 15:21:10 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2532FC000F for ; Mon, 12 Apr 2021 15:21:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 56DAA40515 for ; Mon, 12 Apr 2021 15:20:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="B1TOlIkl"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="o02D80ai" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vfAL_yghmbZ1 for ; Mon, 12 Apr 2021 15:20:39 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp4.osuosl.org (Postfix) with ESMTPS id 0D5774051B for ; Mon, 12 Apr 2021 15:20:37 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 0F7FF5C01C8; Mon, 12 Apr 2021 11:20:37 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=yHbgqj+Xwy+Wc PcX5QoTlHh04agiiQhi1dKVltTw52I=; b=B1TOlIklY8PMnHORRr0kdyOO6xCHE PUrhbeHe/elMP98NSApneDT8/rx9Skyh4nxDmYpTgiRoTzguCdHZ4P+Dx/gNFjf1 6wd6b30Aj+ure3nGDLaxeGr4iE9VhT3r/BWsBZl3kQv0uwk8q1D5dqMbcOjssAlu KcH7L/5A9i7+BjORe6UGCavYT0W1EBxtqfco7JpiT8eH6zk6/dbO4XIPbIzryjgu MhxhC5mCBduk8jhFt4vJduK7sNnVCwjGZfQ8zF4aMAcGLFIJozaIszn+AMVhLjfD fYTbOlsfR8lwd43DD6BaYiTHbZ+yK0qLB3X/5ryyEnO0tqViFcMTmDF8A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=yHbgqj+Xwy+WcPcX5QoTlHh04agiiQhi1dKVltTw52I=; b=o02D80ai 9oOVKevGKJzRGnq1+yiJPY58d/Q+MfJReiRx59qUPyPhE7LGTmKSjSehOj50XU6U GeYKcdo4GDBHt6I2rxzBjwQVWe/pMqy6+LpeKHNPbApyZwj1CM9/+s/y+U+aoO3R 7Uy/aikjevmuzfTyffPHnyjH9jeZmc5ODaJI/nu1K34rrNoiuuxqY7wjrWD1gPsd 9UfNgZVlJh2iNfUfXcse8EvMR98xIzPIEcyoQJNviIkMwNtBWr7/XLr4XKa+2vTp RMvGMmRP/DTbRyAZN+4/TwkvgCTSctB1jk3aFG9jPrh6MqpLId3o7d1rj+qK/WhX jiCz82aTTnKyvQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpeelnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 6DD30240054; Mon, 12 Apr 2021 11:20:36 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:11 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 25/28] dpif-netdev: Make megaflow and mark mappings thread objects X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In later commits hardware offloads are managed in several threads. Each offload is managed by a thread determined by its flow's 'mega_ufid'. As megaflow to mark and mark to flow mappings are 1:1 and 1:N respectively, then a single mark exists for a single 'mega_ufid', and multiple flows uses the same 'mega_ufid'. Because the managing thread will be choosen using the 'mega_ufid', then each mapping does not need to be shared with other offload threads. The mappings are kept as cmap as upcalls will sometimes query them before enqueuing orders to the offload threads. To prepare this change, move the mappings within the offload thread structure. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index bc87b368b..f0c33efca 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -462,12 +462,16 @@ struct dp_offload_thread_item { struct dp_offload_thread { struct mpsc_queue queue; atomic_uint64_t enqueued_item; + struct cmap megaflow_to_mark; + struct cmap mark_to_flow; struct mov_avg_cma cma; struct mov_avg_ema ema; }; static struct dp_offload_thread dp_offload_thread = { .queue = MPSC_QUEUE_INITIALIZER(&dp_offload_thread.queue), + .megaflow_to_mark = CMAP_INITIALIZER, + .mark_to_flow = CMAP_INITIALIZER, .enqueued_item = ATOMIC_VAR_INIT(0), .cma = MOV_AVG_CMA_INITIALIZER, .ema = MOV_AVG_EMA_INITIALIZER(100), @@ -2437,16 +2441,7 @@ struct megaflow_to_mark_data { uint32_t mark; }; -struct flow_mark { - struct cmap megaflow_to_mark; - struct cmap mark_to_flow; - struct seq_pool *pool; -}; - -static struct flow_mark flow_mark = { - .megaflow_to_mark = CMAP_INITIALIZER, - .mark_to_flow = CMAP_INITIALIZER, -}; +static struct seq_pool *flow_mark_pool; static uint32_t flow_mark_alloc(void) @@ -2457,12 +2452,12 @@ flow_mark_alloc(void) if (ovsthread_once_start(&pool_init)) { /* Haven't initiated yet, do it here */ - flow_mark.pool = seq_pool_create(netdev_offload_thread_nb(), + flow_mark_pool = seq_pool_create(netdev_offload_thread_nb(), 1, MAX_FLOW_MARK); ovsthread_once_done(&pool_init); } - if (seq_pool_new_id(flow_mark.pool, tid, &mark)) { + if (seq_pool_new_id(flow_mark_pool, tid, &mark)) { return mark; } @@ -2474,7 +2469,7 @@ flow_mark_free(uint32_t mark) { unsigned int tid = netdev_offload_thread_id(); - seq_pool_free_id(flow_mark.pool, tid, mark); + seq_pool_free_id(flow_mark_pool, tid, mark); } /* associate megaflow with a mark, which is a 1:1 mapping */ @@ -2487,7 +2482,7 @@ megaflow_to_mark_associate(const ovs_u128 *mega_ufid, uint32_t mark) data->mega_ufid = *mega_ufid; data->mark = mark; - cmap_insert(&flow_mark.megaflow_to_mark, + cmap_insert(&dp_offload_thread.megaflow_to_mark, CONST_CAST(struct cmap_node *, &data->node), hash); } @@ -2498,9 +2493,10 @@ megaflow_to_mark_disassociate(const ovs_u128 *mega_ufid) size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data; - CMAP_FOR_EACH_WITH_HASH (data, node, hash, &flow_mark.megaflow_to_mark) { + CMAP_FOR_EACH_WITH_HASH (data, node, hash, + &dp_offload_thread.megaflow_to_mark) { if (ovs_u128_equals(*mega_ufid, data->mega_ufid)) { - cmap_remove(&flow_mark.megaflow_to_mark, + cmap_remove(&dp_offload_thread.megaflow_to_mark, CONST_CAST(struct cmap_node *, &data->node), hash); ovsrcu_postpone(free, data); return; @@ -2517,7 +2513,8 @@ megaflow_to_mark_find(const ovs_u128 *mega_ufid) size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data; - CMAP_FOR_EACH_WITH_HASH (data, node, hash, &flow_mark.megaflow_to_mark) { + CMAP_FOR_EACH_WITH_HASH (data, node, hash, + &dp_offload_thread.megaflow_to_mark) { if (ovs_u128_equals(*mega_ufid, data->mega_ufid)) { return data->mark; } @@ -2534,7 +2531,7 @@ mark_to_flow_associate(const uint32_t mark, struct dp_netdev_flow *flow) { dp_netdev_flow_ref(flow); - cmap_insert(&flow_mark.mark_to_flow, + cmap_insert(&dp_offload_thread.mark_to_flow, CONST_CAST(struct cmap_node *, &flow->mark_node), hash_int(mark, 0)); flow->mark = mark; @@ -2549,7 +2546,7 @@ flow_mark_has_no_ref(uint32_t mark) struct dp_netdev_flow *flow; CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash_int(mark, 0), - &flow_mark.mark_to_flow) { + &dp_offload_thread.mark_to_flow) { if (flow->mark == mark) { return false; } @@ -2574,7 +2571,7 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, return EINVAL; } - cmap_remove(&flow_mark.mark_to_flow, mark_node, hash_int(mark, 0)); + cmap_remove(&dp_offload_thread.mark_to_flow, mark_node, hash_int(mark, 0)); flow->mark = INVALID_FLOW_MARK; /* @@ -2611,7 +2608,7 @@ flow_mark_flush(struct dp_netdev_pmd_thread *pmd) { struct dp_netdev_flow *flow; - CMAP_FOR_EACH (flow, mark_node, &flow_mark.mark_to_flow) { + CMAP_FOR_EACH (flow, mark_node, &dp_offload_thread.mark_to_flow) { if (flow->pmd_id == pmd->core_id) { queue_netdev_flow_del(pmd, flow); } @@ -2625,7 +2622,7 @@ mark_to_flow_find(const struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow; CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash_int(mark, 0), - &flow_mark.mark_to_flow) { + &dp_offload_thread.mark_to_flow) { if (flow->mark == mark && flow->pmd_id == pmd->core_id && flow->dead == false) { return flow; From patchwork Mon Apr 12 15:20:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465351 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=Nc5uAwpb; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=OSb9Yrwz; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssM0SbBz9rx6 for ; Tue, 13 Apr 2021 01:21:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 7B3EE60BD1; Mon, 12 Apr 2021 15:21:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BHk3uDhFhr2e; Mon, 12 Apr 2021 15:21:38 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTP id 3CD2D60ADD; Mon, 12 Apr 2021 15:21:23 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 04126C000C; Mon, 12 Apr 2021 15:21:23 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3C7EEC000C for ; Mon, 12 Apr 2021 15:21:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 6879D60AD6 for ; Mon, 12 Apr 2021 15:20:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yHO2fuOXVLIg for ; Mon, 12 Apr 2021 15:20:38 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp3.osuosl.org (Postfix) with ESMTPS id A1C7F60AE9 for ; Mon, 12 Apr 2021 15:20:38 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id CCEFA5C01CC; Mon, 12 Apr 2021 11:20:37 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=Q6tte0q5uItMr EZcE+xic3NYSj4x39G8tUzGRYJ4IcU=; b=Nc5uAwpbtOD87R+s6z613BD/Yu0KT 8xcCEJ6uRayAu5rQwvrYH3CpWkXXV9brhQLxkDvTGJ5wStMU9J0/FE2lEN8s4+Gq jdG4sREgLnO/ULyPCWku4p4BEIwqmwwlB5Q+0M74V69aeNVqVbdPJHrsSEHhkIMk yqSeWT7xsgJ3LjyhM17lQLJvCXGFv2rO41YdnX+qyTQ2DcTUJ0g2WohuvZjOlvBM jzyQ6M+2crwipzf0YMXc7V5GhNYrVksorXe8kcZo5Bi3NP0OjwHBRt1Garkl6NNZ RUr5KTmdzFjRw9/nU4O5xVGfo0T9/QBETJYYdmVK6en1MJk6Ee2E4fVrg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=Q6tte0q5uItMrEZcE+xic3NYSj4x39G8tUzGRYJ4IcU=; b=OSb9Yrwz TK9HgTNVWJIkWIo9evW3lT01LO1w3Q0KLGKXxph8RcaNLiZbXbCnJmZ8fHNGf392 PAa1qClNB2/MRLSH4lUUVWO+P1mesgWMDBgcfnKcAjO2U7doaixuRR+gHduoid8y fLnB1kaKorBYr/wRTG5/8myIiL/fnU2w9Ez91LiY+H5H3KC4gEcqgCW9GQa2HXzx 42nbauLRdvwZoqgkVf00WPY9TCs2ywpXpyWqsKqOdroIHvjgVrcsNBO6SFbON4m2 4RjxsZul8a2ge/O9DNc0I/dQ0142M431o529IkMRxEk2gmbWBYNqjIFpVOrAZAre jmIR/iJWMgxsZw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu kfhppeekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpeelnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id 32065240069; Mon, 12 Apr 2021 11:20:37 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:12 +0200 Message-Id: <8c94737b8c349d8402ed1fa932a0dd98a91a6193.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 26/28] dpif-netdev: Replace port mutex by rwlock X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The port mutex protects the netdev mapping, that can be changed by port addition or port deletion. HW offloads operations can be considered read operations on the port mapping itself. Use a rwlock to differentiate between read and write operations, allowing concurrent queries and offload insertions. Because offload queries, deletion, and reconfigure_datapath() calls are all rdlock, the deadlock fixed by [1] is still avoided, as the rdlock side is recursive as prescribed by the POSIX standard. Executing 'reconfigure_datapath()' only requires a rdlock taken, but it is sometimes executed in contexts where wrlock is taken ('do_add_port()' and 'do_del_port()'). This means that the deadlock described in [2] is still valid and should be mitigated. The rdlock is taken using 'tryrdlock()' during offload query, keeping the current behavior. [1]: 81e89d5c2645 ("dpif-netdev: Make datapath port mutex recursive.") [2]: 12d0edd75eba ("dpif-netdev: Avoid deadlock with offloading during PMD thread deletion."). Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 139 +++++++++++++++++++------------------- lib/netdev-offload-dpdk.c | 4 +- 2 files changed, 72 insertions(+), 71 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index f0c33efca..893963002 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -333,8 +333,8 @@ struct dp_netdev { /* Ports. * * Any lookup into 'ports' or any access to the dp_netdev_ports found - * through 'ports' requires taking 'port_mutex'. */ - struct ovs_mutex port_mutex; + * through 'ports' requires taking 'port_rwlock'. */ + struct ovs_rwlock port_rwlock; struct hmap ports; struct seq *port_seq; /* Incremented whenever a port changes. */ @@ -410,7 +410,7 @@ static void meter_unlock(const struct dp_netdev *dp, uint32_t meter_id) static struct dp_netdev_port *dp_netdev_lookup_port(const struct dp_netdev *dp, odp_port_t) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_RDLOCK(dp->port_rwlock); enum rxq_cycles_counter_type { RXQ_CYCLES_PROC_CURR, /* Cycles spent successfully polling and @@ -851,17 +851,17 @@ struct dpif_netdev { static int get_port_by_number(struct dp_netdev *dp, odp_port_t port_no, struct dp_netdev_port **portp) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_RDLOCK(dp->port_rwlock); static int get_port_by_name(struct dp_netdev *dp, const char *devname, struct dp_netdev_port **portp) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_RDLOCK(dp->port_rwlock); static void dp_netdev_free(struct dp_netdev *) OVS_REQUIRES(dp_netdev_mutex); static int do_add_port(struct dp_netdev *dp, const char *devname, const char *type, odp_port_t port_no) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_WRLOCK(dp->port_rwlock); static void do_del_port(struct dp_netdev *dp, struct dp_netdev_port *) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_WRLOCK(dp->port_rwlock); static int dpif_netdev_open(const struct dpif_class *, const char *name, bool create, struct dpif **); static void dp_netdev_execute_actions(struct dp_netdev_pmd_thread *pmd, @@ -882,7 +882,7 @@ static void dp_netdev_configure_pmd(struct dp_netdev_pmd_thread *pmd, int numa_id); static void dp_netdev_destroy_pmd(struct dp_netdev_pmd_thread *pmd); static void dp_netdev_set_nonpmd(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_WRLOCK(dp->port_rwlock); static void *pmd_thread_main(void *); static struct dp_netdev_pmd_thread *dp_netdev_get_pmd(struct dp_netdev *dp, @@ -919,7 +919,7 @@ static void dp_netdev_offload_flush(struct dp_netdev *dp, struct dp_netdev_port *port); static void reconfigure_datapath(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_RDLOCK(dp->port_rwlock); static bool dp_netdev_pmd_try_ref(struct dp_netdev_pmd_thread *pmd); static void dp_netdev_pmd_unref(struct dp_netdev_pmd_thread *pmd); static void dp_netdev_pmd_flow_flush(struct dp_netdev_pmd_thread *pmd); @@ -1425,8 +1425,8 @@ dpif_netdev_subtable_lookup_set(struct unixctl_conn *conn, int argc, struct dp_netdev_pmd_thread **pmd_list; sorted_poll_thread_list(dp, &pmd_list, &n); - /* take port mutex as HMAP iters over them. */ - ovs_mutex_lock(&dp->port_mutex); + /* take port rwlock as HMAP iters over them. */ + ovs_rwlock_rdlock(&dp->port_rwlock); for (size_t i = 0; i < n; i++) { struct dp_netdev_pmd_thread *pmd = pmd_list[i]; @@ -1449,8 +1449,8 @@ dpif_netdev_subtable_lookup_set(struct unixctl_conn *conn, int argc, } } - /* release port mutex before netdev mutex. */ - ovs_mutex_unlock(&dp->port_mutex); + /* release port rwlock before netdev mutex. */ + ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_unlock(&dp_netdev_mutex); struct ds reply = DS_EMPTY_INITIALIZER; @@ -1743,7 +1743,7 @@ create_dpif_netdev(struct dp_netdev *dp) * Return ODPP_NONE on failure. */ static odp_port_t choose_port(struct dp_netdev *dp, const char *name) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { uint32_t port_no; @@ -1806,7 +1806,7 @@ create_dp_netdev(const char *name, const struct dpif_class *class, ovs_refcount_init(&dp->ref_cnt); atomic_flag_clear(&dp->destroyed); - ovs_mutex_init_recursive(&dp->port_mutex); + ovs_rwlock_init(&dp->port_rwlock); hmap_init(&dp->ports); dp->port_seq = seq_create(); ovs_mutex_init(&dp->bond_mutex); @@ -1841,7 +1841,7 @@ create_dp_netdev(const char *name, const struct dpif_class *class, ovs_mutex_init_recursive(&dp->non_pmd_mutex); ovsthread_key_create(&dp->per_pmd_key, NULL); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); /* non-PMD will be created before all other threads and will * allocate static_tx_qid = 0. */ dp_netdev_set_nonpmd(dp); @@ -1849,7 +1849,7 @@ create_dp_netdev(const char *name, const struct dpif_class *class, error = do_add_port(dp, name, dpif_netdev_port_open_type(dp->class, "internal"), ODPP_LOCAL); - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); if (error) { dp_netdev_free(dp); return error; @@ -1935,11 +1935,11 @@ dp_netdev_free(struct dp_netdev *dp) shash_find_and_delete(&dp_netdevs, dp->name); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); HMAP_FOR_EACH_SAFE (port, next, node, &dp->ports) { do_del_port(dp, port); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_lock(&dp->bond_mutex); CMAP_FOR_EACH (bond, node, &dp->tx_bonds) { @@ -1964,7 +1964,7 @@ dp_netdev_free(struct dp_netdev *dp) seq_destroy(dp->port_seq); hmap_destroy(&dp->ports); - ovs_mutex_destroy(&dp->port_mutex); + ovs_rwlock_destroy(&dp->port_rwlock); cmap_destroy(&dp->tx_bonds); ovs_mutex_destroy(&dp->bond_mutex); @@ -2132,7 +2132,7 @@ out: static int do_add_port(struct dp_netdev *dp, const char *devname, const char *type, odp_port_t port_no) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_WRLOCK(dp->port_rwlock) { struct netdev_saved_flags *sf; struct dp_netdev_port *port; @@ -2184,7 +2184,7 @@ dpif_netdev_port_add(struct dpif *dpif, struct netdev *netdev, odp_port_t port_no; int error; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); dpif_port = netdev_vport_get_dpif_port(netdev, namebuf, sizeof namebuf); if (*port_nop != ODPP_NONE) { port_no = *port_nop; @@ -2197,7 +2197,7 @@ dpif_netdev_port_add(struct dpif *dpif, struct netdev *netdev, *port_nop = port_no; error = do_add_port(dp, dpif_port, netdev_get_type(netdev), port_no); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -2208,7 +2208,7 @@ dpif_netdev_port_del(struct dpif *dpif, odp_port_t port_no) struct dp_netdev *dp = get_dp_netdev(dpif); int error; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); if (port_no == ODPP_LOCAL) { error = EINVAL; } else { @@ -2219,7 +2219,7 @@ dpif_netdev_port_del(struct dpif *dpif, odp_port_t port_no) do_del_port(dp, port); } } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -2232,7 +2232,7 @@ is_valid_port_number(odp_port_t port_no) static struct dp_netdev_port * dp_netdev_lookup_port(const struct dp_netdev *dp, odp_port_t port_no) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; @@ -2247,7 +2247,7 @@ dp_netdev_lookup_port(const struct dp_netdev *dp, odp_port_t port_no) static int get_port_by_number(struct dp_netdev *dp, odp_port_t port_no, struct dp_netdev_port **portp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { if (!is_valid_port_number(port_no)) { *portp = NULL; @@ -2282,7 +2282,7 @@ port_destroy(struct dp_netdev_port *port) static int get_port_by_name(struct dp_netdev *dp, const char *devname, struct dp_netdev_port **portp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; @@ -2301,7 +2301,7 @@ get_port_by_name(struct dp_netdev *dp, /* Returns 'true' if there is a port with pmd netdev. */ static bool has_pmd_port(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; @@ -2316,7 +2316,7 @@ has_pmd_port(struct dp_netdev *dp) static void do_del_port(struct dp_netdev *dp, struct dp_netdev_port *port) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_WRLOCK(dp->port_rwlock) { dp_netdev_offload_flush(dp, port); netdev_uninit_flow_api(port->netdev); @@ -2345,12 +2345,12 @@ dpif_netdev_port_query_by_number(const struct dpif *dpif, odp_port_t port_no, struct dp_netdev_port *port; int error; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); error = get_port_by_number(dp, port_no, &port); if (!error && dpif_port) { answer_port_query(port, dpif_port); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -2363,12 +2363,12 @@ dpif_netdev_port_query_by_name(const struct dpif *dpif, const char *devname, struct dp_netdev_port *port; int error; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); error = get_port_by_name(dp, devname, &port); if (!error && dpif_port) { answer_port_query(port, dpif_port); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -2586,9 +2586,9 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, if (port) { /* Taking a global 'port_mutex' to fulfill thread safety * restrictions regarding netdev port mapping. */ - ovs_mutex_lock(&pmd->dp->port_mutex); + ovs_rwlock_rdlock(&pmd->dp->port_rwlock); ret = netdev_flow_del(port, &flow->mega_ufid, NULL); - ovs_mutex_unlock(&pmd->dp->port_mutex); + ovs_rwlock_unlock(&pmd->dp->port_rwlock); netdev_close(port); } @@ -2764,12 +2764,12 @@ dp_netdev_flow_offload_put(struct dp_offload_flow_item *offload) } /* Taking a global 'port_mutex' to fulfill thread safety * restrictions regarding the netdev port mapping. */ - ovs_mutex_lock(&pmd->dp->port_mutex); + ovs_rwlock_rdlock(&pmd->dp->port_rwlock); ret = netdev_flow_put(port, &offload->match, CONST_CAST(struct nlattr *, offload->actions), offload->actions_len, &flow->mega_ufid, &info, NULL); - ovs_mutex_unlock(&pmd->dp->port_mutex); + ovs_rwlock_unlock(&pmd->dp->port_rwlock); netdev_close(port); if (ret) { @@ -2825,9 +2825,9 @@ dp_offload_flush(struct dp_offload_thread_item *item) { struct dp_offload_flush_item *flush = &item->data->flush; - ovs_mutex_lock(&flush->dp->port_mutex); + ovs_rwlock_rdlock(&flush->dp->port_rwlock); netdev_flow_flush(flush->netdev); - ovs_mutex_unlock(&flush->dp->port_mutex); + ovs_rwlock_unlock(&flush->dp->port_rwlock); ovs_barrier_block(flush->barrier); @@ -3007,7 +3007,7 @@ dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, * complete its work. As the flush order will only be * enqueued after existing offload requests, those previous * offload requests must be processed, which requires being - * able to lock the 'port_mutex' from the offload thread. + * able to lock the 'port_rwlock' from the offload thread. * * Flow offload flush is done when a port is being deleted. * Right after this call executes, the offload API is disabled @@ -3017,7 +3017,7 @@ dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, static void dp_netdev_offload_flush(struct dp_netdev *dp, struct dp_netdev_port *port) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_WRLOCK(dp->port_rwlock) { /* The flush mutex only serves to protect the static memory barrier. * The memory barrier needs to go beyond the function scope as @@ -3035,7 +3035,7 @@ dp_netdev_offload_flush(struct dp_netdev *dp, return; } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_lock(&flush_mutex); /* This thread and the offload thread. */ @@ -3053,7 +3053,7 @@ dp_netdev_offload_flush(struct dp_netdev *dp, * Some offload provider (e.g. DPDK) keeps a netdev reference with * the offload data. If this reference is not closed, the netdev is * kept indefinitely. */ - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); ovs_barrier_block(&barrier); ovs_barrier_destroy(&barrier); @@ -3107,7 +3107,7 @@ dpif_netdev_port_dump_next(const struct dpif *dpif, void *state_, struct hmap_node *node; int retval; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); node = hmap_at_position(&dp->ports, &state->position); if (node) { struct dp_netdev_port *port; @@ -3124,7 +3124,7 @@ dpif_netdev_port_dump_next(const struct dpif *dpif, void *state_, } else { retval = EOF; } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return retval; } @@ -3575,24 +3575,24 @@ dpif_netdev_get_flow_offload_status(const struct dp_netdev *dp, return false; } ofpbuf_use_stack(&buf, &act_buf, sizeof act_buf); - /* Taking a global 'port_mutex' to fulfill thread safety + /* Taking a global 'port_rwlock' to fulfill thread safety * restrictions regarding netdev port mapping. * * XXX: Main thread will try to pause/stop all revalidators during datapath * reconfiguration via datapath purge callback (dp_purge_cb) while - * holding 'dp->port_mutex'. So we're not waiting for mutex here. - * Otherwise, deadlock is possible, bcause revalidators might sleep + * rw-holding 'dp->port_rwlock'. So we're not waiting for lock here. + * Otherwise, deadlock is possible, because revalidators might sleep * waiting for the main thread to release the lock and main thread * will wait for them to stop processing. * This workaround might make statistics less accurate. Especially * for flow deletion case, since there will be no other attempt. */ - if (!ovs_mutex_trylock(&dp->port_mutex)) { + if (!ovs_rwlock_tryrdlock(&dp->port_rwlock)) { ret = netdev_flow_get(netdev, &match, &actions, &netdev_flow->mega_ufid, stats, attrs, &buf); /* Storing statistics and attributes from the last request for * later use on mutex contention. */ dp_netdev_flow_set_last_stats_attrs(netdev_flow, stats, attrs, ret); - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); } else { dp_netdev_flow_get_last_stats_attrs(netdev_flow, stats, attrs, &ret); if (!ret && !attrs->dp_layer) { @@ -4462,7 +4462,7 @@ dpif_netdev_offload_stats_get(struct dpif *dpif, nb_offloads = 0; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); HMAP_FOR_EACH (port, node, &dp->ports) { uint64_t port_nb_offloads = 0; @@ -4471,7 +4471,7 @@ dpif_netdev_offload_stats_get(struct dpif *dpif, nb_offloads += port_nb_offloads; } } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); atomic_read_relaxed(&dp_offload_thread.enqueued_item, &stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value); @@ -4781,7 +4781,7 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, const char *affinity_list = smap_get(cfg, "pmd-rxq-affinity"); bool emc_enabled = smap_get_bool(cfg, "emc-enable", true); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); error = get_port_by_number(dp, port_no, &port); if (error) { goto unlock; @@ -4835,7 +4835,7 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, dp_netdev_request_reconfigure(dp); unlock: - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -5325,7 +5325,8 @@ compare_rxq_cycles(const void *a, const void *b) * The function doesn't touch the pmd threads, it just stores the assignment * in the 'pmd' member of each rxq. */ static void -rxq_scheduling(struct dp_netdev *dp, bool pinned) OVS_REQUIRES(dp->port_mutex) +rxq_scheduling(struct dp_netdev *dp, bool pinned) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; struct rr_numa_list rr; @@ -5469,7 +5470,7 @@ reload_affected_pmds(struct dp_netdev *dp) static void reconfigure_pmd_threads(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_pmd_thread *pmd; struct ovs_numa_dump *pmd_cores; @@ -5567,7 +5568,7 @@ static void pmd_remove_stale_ports(struct dp_netdev *dp, struct dp_netdev_pmd_thread *pmd) OVS_EXCLUDED(pmd->port_mutex) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct rxq_poll *poll, *poll_next; struct tx_port *tx, *tx_next; @@ -5597,7 +5598,7 @@ pmd_remove_stale_ports(struct dp_netdev *dp, * rxqs and assigns all rxqs/txqs to pmd threads. */ static void reconfigure_datapath(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct hmapx busy_threads = HMAPX_INITIALIZER(&busy_threads); struct dp_netdev_pmd_thread *pmd; @@ -5781,7 +5782,7 @@ reconfigure_datapath(struct dp_netdev *dp) /* Returns true if one of the netdevs in 'dp' requires a reconfiguration */ static bool ports_require_restart(const struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; @@ -5832,7 +5833,7 @@ variance(uint64_t a[], int n) static bool get_dry_run_variance(struct dp_netdev *dp, uint32_t *core_list, uint32_t num_pmds, uint64_t *predicted_variance) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; struct dp_netdev_pmd_thread *pmd; @@ -5955,7 +5956,7 @@ cleanup: * better distribution of load on PMDs. */ static bool pmd_rebalance_dry_run(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_pmd_thread *pmd; uint64_t *curr_pmd_usage; @@ -6050,7 +6051,7 @@ dpif_netdev_run(struct dpif *dpif) long long int now = time_msec(); struct dp_netdev_pmd_thread *pmd; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); non_pmd = dp_netdev_get_pmd(dp, NON_PMD_CORE_ID); if (non_pmd) { ovs_mutex_lock(&dp->non_pmd_mutex); @@ -6122,7 +6123,7 @@ dpif_netdev_run(struct dpif *dpif) if (dp_netdev_is_reconf_required(dp) || ports_require_restart(dp)) { reconfigure_datapath(dp); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); tnl_neigh_cache_run(); tnl_port_map_run(); @@ -6142,7 +6143,7 @@ dpif_netdev_wait(struct dpif *dpif) struct dp_netdev *dp = get_dp_netdev(dpif); ovs_mutex_lock(&dp_netdev_mutex); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); HMAP_FOR_EACH (port, node, &dp->ports) { netdev_wait_reconf_required(port->netdev); if (!netdev_is_pmd(port->netdev)) { @@ -6153,7 +6154,7 @@ dpif_netdev_wait(struct dpif *dpif) } } } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_unlock(&dp_netdev_mutex); seq_wait(tnl_conf_seq, dp->last_tnl_conf_seq); } @@ -6771,7 +6772,7 @@ dp_netdev_get_pmd(struct dp_netdev *dp, unsigned core_id) /* Sets the 'struct dp_netdev_pmd_thread' for non-pmd threads. */ static void dp_netdev_set_nonpmd(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_WRLOCK(dp->port_rwlock) { struct dp_netdev_pmd_thread *non_pmd; @@ -8834,7 +8835,7 @@ dpif_dummy_change_port_number(struct unixctl_conn *conn, int argc OVS_UNUSED, ovs_refcount_ref(&dp->ref_cnt); ovs_mutex_unlock(&dp_netdev_mutex); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); if (get_port_by_name(dp, argv[2], &port)) { unixctl_command_reply_error(conn, "unknown port"); goto exit; @@ -8863,7 +8864,7 @@ dpif_dummy_change_port_number(struct unixctl_conn *conn, int argc OVS_UNUSED, unixctl_command_reply(conn, NULL); exit: - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); dp_netdev_unref(dp); } diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index 13e017ef8..70e6843f5 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -44,8 +44,8 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(100, 5); * For example, simultaneous call of 'netdev_reconfigure()' for the same * 'netdev' is forbidden. * - * For current implementation all above restrictions could be fulfilled by - * taking the datapath 'port_mutex' in lib/dpif-netdev.c. */ + * For current implementation all above restrictions are fulfilled by + * read-locking the datapath 'port_rwlock' in lib/dpif-netdev.c. */ /* * A mapping from ufid to dpdk rte_flow. From patchwork Mon Apr 12 15:20:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465357 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=cZsCyfJ9; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=gnMKEy3x; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJsst3mSPz9sTD for ; Tue, 13 Apr 2021 01:22:10 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B434A61593; Mon, 12 Apr 2021 15:22:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id caihWPmNMRhF; Mon, 12 Apr 2021 15:22:05 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTP id 9C23E60AF0; Mon, 12 Apr 2021 15:21:48 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5F8DEC000C; Mon, 12 Apr 2021 15:21:48 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0E64BC000A for ; Mon, 12 Apr 2021 15:21:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C5E7183D04 for ; Mon, 12 Apr 2021 15:20:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="cZsCyfJ9"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="gnMKEy3x" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hFL7m8B5aRVp for ; Mon, 12 Apr 2021 15:20:45 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp1.osuosl.org (Postfix) with ESMTPS id 69AFB83C47 for ; Mon, 12 Apr 2021 15:20:39 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 925DB5C01C8; Mon, 12 Apr 2021 11:20:38 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 12 Apr 2021 11:20:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=hnVn9zflA7wMH 3tHhr7IMdKkPgfHbjDVeG7XXLEzxF4=; b=cZsCyfJ9x4fMk1gYn6iVFXp7M4xHu Yj6dd/VtiYq90j4ap0k2f7KcAprHXP2r58VhBadZFEbiOlD+5M6BqX9mqPtPl9pv SaSgTSD9B3TUY4hfyanHkxPSwtGGIyqQEqb6t3NnB8NdJ/ZCmG5uESsFaEcXlZf8 PpHDj74MOw3jBHxZ4irCtQuL02wOxSi4AVbMQbtIHLWPHpLZCzw3DrbwqkN70epF 2OKQrM7SxhkHKyhti4+mKMgCyaA6Qou8qxKu/WjJOsk0Vy7DKUZk3UcB3OmV+k9v JDKCW7RgPBJleqIbFsZC6Re4Dvqfpcw7/k+XHgMoarrbiEG/+cdYZacxA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=hnVn9zflA7wMH3tHhr7IMdKkPgfHbjDVeG7XXLEzxF4=; b=gnMKEy3x CzazJ7liRj3N6jTEDYEL08PWSfNMCAvwPIfP+Ltp44hW8ACGG9SEYvvHwZJwwEaC Sot9IhaZ7y+XLYomlna7Er+dNKvbO8BLZrAMGGUuKNn802YtPqtFG5qRUJ44sKfc pGKGpFmP9kLVf7WLAMNC+yuNH8/RsZPJQQGf8PbktQD9/H3+Izy7y81W2BZJjXXr ax5gTCelNQ+u+Iy+TNnkD4H8datC3eL28pq6MtVLcTX3AtY0dgXHrbSeNBdVGZeq dxheCs52xVCOsFUp5AkZBuuJo05pUMjAvTeSR2SzOz0xhyoyFAjgX6iSiobbCK9y 0x2Lep6oEyyfJA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epkeelhfffgfekudfggfduleeukefghefgtdfhvdekuefhffeutdetveevudeivdfgnecu ffhomhgrihhnpehmvggrnhdrthhothgrlhdpshhtugguvghvrdhtohhtrghlnecukfhppe ekiedrvdehgedrvdegfedrudegieenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id EC164240066; Mon, 12 Apr 2021 11:20:37 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:13 +0200 Message-Id: <6c85a4c0086a762c26ab074da9503608805e2a8b.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 27/28] dpif-netdev: Use one or more offload threads X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Read the user configuration in the netdev-offload module to modify the number of threads used to manage hardware offload requests. This allows processing insertion, deletion and modification concurrently. The offload thread structure was modified to contain all needed elements. This structure is multiplied by the number of requested threads and used separately. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 304 +++++++++++++++++++++++++------------- lib/netdev-offload-dpdk.c | 7 +- 2 files changed, 204 insertions(+), 107 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 893963002..ad13906f6 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -460,25 +460,47 @@ struct dp_offload_thread_item { }; struct dp_offload_thread { - struct mpsc_queue queue; - atomic_uint64_t enqueued_item; - struct cmap megaflow_to_mark; - struct cmap mark_to_flow; - struct mov_avg_cma cma; - struct mov_avg_ema ema; + PADDED_MEMBERS(CACHE_LINE_SIZE, + struct mpsc_queue queue; + atomic_uint64_t enqueued_item; + struct cmap megaflow_to_mark; + struct cmap mark_to_flow; + struct mov_avg_cma cma; + struct mov_avg_ema ema; + ); }; +static struct dp_offload_thread *dp_offload_threads; +static void *dp_netdev_flow_offload_main(void *arg); -static struct dp_offload_thread dp_offload_thread = { - .queue = MPSC_QUEUE_INITIALIZER(&dp_offload_thread.queue), - .megaflow_to_mark = CMAP_INITIALIZER, - .mark_to_flow = CMAP_INITIALIZER, - .enqueued_item = ATOMIC_VAR_INIT(0), - .cma = MOV_AVG_CMA_INITIALIZER, - .ema = MOV_AVG_EMA_INITIALIZER(100), -}; +static void +dp_netdev_offload_init(void) +{ + static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; + unsigned int nb_offload_thread = netdev_offload_thread_nb(); + unsigned int tid; + + if (!ovsthread_once_start(&once)) { + return; + } + + dp_offload_threads = xcalloc(nb_offload_thread, + sizeof *dp_offload_threads); -static struct ovsthread_once offload_thread_once - = OVSTHREAD_ONCE_INITIALIZER; + for (tid = 0; tid < nb_offload_thread; tid++) { + struct dp_offload_thread *thread; + + thread = &dp_offload_threads[tid]; + mpsc_queue_init(&thread->queue); + cmap_init(&thread->megaflow_to_mark); + cmap_init(&thread->mark_to_flow); + atomic_init(&thread->enqueued_item, 0); + mov_avg_cma_init(&thread->cma); + mov_avg_ema_init(&thread->ema, 100); + ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, thread); + } + + ovsthread_once_done(&once); +} #define XPS_TIMEOUT 500000LL /* In microseconds. */ @@ -2478,11 +2500,12 @@ megaflow_to_mark_associate(const ovs_u128 *mega_ufid, uint32_t mark) { size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data = xzalloc(sizeof(*data)); + unsigned int tid = netdev_offload_thread_id(); data->mega_ufid = *mega_ufid; data->mark = mark; - cmap_insert(&dp_offload_thread.megaflow_to_mark, + cmap_insert(&dp_offload_threads[tid].megaflow_to_mark, CONST_CAST(struct cmap_node *, &data->node), hash); } @@ -2492,11 +2515,12 @@ megaflow_to_mark_disassociate(const ovs_u128 *mega_ufid) { size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data; + unsigned int tid = netdev_offload_thread_id(); CMAP_FOR_EACH_WITH_HASH (data, node, hash, - &dp_offload_thread.megaflow_to_mark) { + &dp_offload_threads[tid].megaflow_to_mark) { if (ovs_u128_equals(*mega_ufid, data->mega_ufid)) { - cmap_remove(&dp_offload_thread.megaflow_to_mark, + cmap_remove(&dp_offload_threads[tid].megaflow_to_mark, CONST_CAST(struct cmap_node *, &data->node), hash); ovsrcu_postpone(free, data); return; @@ -2512,9 +2536,10 @@ megaflow_to_mark_find(const ovs_u128 *mega_ufid) { size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data; + unsigned int tid = netdev_offload_thread_id(); CMAP_FOR_EACH_WITH_HASH (data, node, hash, - &dp_offload_thread.megaflow_to_mark) { + &dp_offload_threads[tid].megaflow_to_mark) { if (ovs_u128_equals(*mega_ufid, data->mega_ufid)) { return data->mark; } @@ -2529,9 +2554,10 @@ megaflow_to_mark_find(const ovs_u128 *mega_ufid) static void mark_to_flow_associate(const uint32_t mark, struct dp_netdev_flow *flow) { + unsigned int tid = netdev_offload_thread_id(); dp_netdev_flow_ref(flow); - cmap_insert(&dp_offload_thread.mark_to_flow, + cmap_insert(&dp_offload_threads[tid].mark_to_flow, CONST_CAST(struct cmap_node *, &flow->mark_node), hash_int(mark, 0)); flow->mark = mark; @@ -2543,10 +2569,11 @@ mark_to_flow_associate(const uint32_t mark, struct dp_netdev_flow *flow) static bool flow_mark_has_no_ref(uint32_t mark) { + unsigned int tid = netdev_offload_thread_id(); struct dp_netdev_flow *flow; CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash_int(mark, 0), - &dp_offload_thread.mark_to_flow) { + &dp_offload_threads[tid].mark_to_flow) { if (flow->mark == mark) { return false; } @@ -2562,6 +2589,7 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, const char *dpif_type_str = dpif_normalize_type(pmd->dp->class->type); struct cmap_node *mark_node = CONST_CAST(struct cmap_node *, &flow->mark_node); + unsigned int tid = netdev_offload_thread_id(); uint32_t mark = flow->mark; int ret = 0; @@ -2571,7 +2599,8 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, return EINVAL; } - cmap_remove(&dp_offload_thread.mark_to_flow, mark_node, hash_int(mark, 0)); + cmap_remove(&dp_offload_threads[tid].mark_to_flow, + mark_node, hash_int(mark, 0)); flow->mark = INVALID_FLOW_MARK; /* @@ -2607,10 +2636,18 @@ static void flow_mark_flush(struct dp_netdev_pmd_thread *pmd) { struct dp_netdev_flow *flow; + unsigned int tid; - CMAP_FOR_EACH (flow, mark_node, &dp_offload_thread.mark_to_flow) { - if (flow->pmd_id == pmd->core_id) { - queue_netdev_flow_del(pmd, flow); + if (dp_offload_threads == NULL) { + return; + } + + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + CMAP_FOR_EACH (flow, mark_node, + &dp_offload_threads[tid].mark_to_flow) { + if (flow->pmd_id == pmd->core_id) { + queue_netdev_flow_del(pmd, flow); + } } } } @@ -2620,12 +2657,21 @@ mark_to_flow_find(const struct dp_netdev_pmd_thread *pmd, const uint32_t mark) { struct dp_netdev_flow *flow; + unsigned int tid; + size_t hash; - CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash_int(mark, 0), - &dp_offload_thread.mark_to_flow) { - if (flow->mark == mark && flow->pmd_id == pmd->core_id && - flow->dead == false) { - return flow; + if (dp_offload_threads == NULL) { + return NULL; + } + + hash = hash_int(mark, 0); + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash, + &dp_offload_threads[tid].mark_to_flow) { + if (flow->mark == mark && flow->pmd_id == pmd->core_id && + flow->dead == false) { + return flow; + } } } @@ -2690,10 +2736,25 @@ dp_netdev_free_offload(struct dp_offload_thread_item *offload) } static void -dp_netdev_append_offload(struct dp_offload_thread_item *offload) +dp_netdev_append_offload(struct dp_offload_thread_item *offload, + unsigned int tid) +{ + dp_netdev_offload_init(); + + mpsc_queue_insert(&dp_offload_threads[tid].queue, &offload->node); + atomic_count_inc64(&dp_offload_threads[tid].enqueued_item); +} + +static void +dp_netdev_offload_flow_enqueue(struct dp_offload_thread_item *item) { - mpsc_queue_insert(&dp_offload_thread.queue, &offload->node); - atomic_count_inc64(&dp_offload_thread.enqueued_item); + struct dp_offload_flow_item *flow_offload = &item->data->flow; + unsigned int tid; + + ovs_assert(item->type == DP_OFFLOAD_FLOW); + + tid = netdev_offload_ufid_to_thread_id(flow_offload->flow->mega_ufid); + dp_netdev_append_offload(item, tid); } static int @@ -2831,8 +2892,8 @@ dp_offload_flush(struct dp_offload_thread_item *item) ovs_barrier_block(flush->barrier); - /* Allow the other thread to take again the port lock, before - * continuing offload operations in this thread. + /* Allow the initiator thread to take again the port lock, + * before continuing offload operations in this thread. */ ovs_barrier_block(flush->barrier); } @@ -2842,8 +2903,9 @@ dp_offload_flush(struct dp_offload_thread_item *item) #define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ static void * -dp_netdev_flow_offload_main(void *data OVS_UNUSED) +dp_netdev_flow_offload_main(void *arg) { + struct dp_offload_thread *ofl_thread = arg; struct dp_offload_thread_item *offload; struct mpsc_queue_node *node; struct mpsc_queue *queue; @@ -2852,7 +2914,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) long long int now; uint64_t backoff; - queue = &dp_offload_thread.queue; + queue = &ofl_thread->queue; mpsc_queue_acquire(queue); while (true) { @@ -2867,7 +2929,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; MPSC_QUEUE_FOR_EACH_POP (node, queue) { offload = CONTAINER_OF(node, struct dp_offload_thread_item, node); - atomic_count_dec64(&dp_offload_thread.enqueued_item); + atomic_count_dec64(&ofl_thread->enqueued_item); switch (offload->type) { case DP_OFFLOAD_FLOW: @@ -2883,8 +2945,8 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) now = time_usec(); latency_us = now - offload->timestamp; - mov_avg_cma_update(&dp_offload_thread.cma, latency_us); - mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + mov_avg_cma_update(&ofl_thread->cma, latency_us); + mov_avg_ema_update(&ofl_thread->ema, latency_us); dp_netdev_free_offload(offload); @@ -2909,16 +2971,10 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, { struct dp_offload_thread_item *offload; - if (ovsthread_once_start(&offload_thread_once)) { - mpsc_queue_init(&dp_offload_thread.queue); - ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); - ovsthread_once_done(&offload_thread_once); - } - offload = dp_netdev_alloc_flow_offload(pmd, flow, DP_NETDEV_FLOW_OFFLOAD_OP_DEL); offload->timestamp = pmd->ctx.now; - dp_netdev_append_offload(offload); + dp_netdev_offload_flow_enqueue(offload); } static void @@ -2934,12 +2990,6 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, return; } - if (ovsthread_once_start(&offload_thread_once)) { - mpsc_queue_init(&dp_offload_thread.queue); - ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); - ovsthread_once_done(&offload_thread_once); - } - if (flow->mark != INVALID_FLOW_MARK) { op = DP_NETDEV_FLOW_OFFLOAD_OP_MOD; } else { @@ -2953,7 +3003,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, flow_offload->actions_len = actions_len; item->timestamp = pmd->ctx.now; - dp_netdev_append_offload(item); + dp_netdev_offload_flow_enqueue(item); } static void @@ -2982,25 +3032,24 @@ dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, struct netdev *netdev, struct ovs_barrier *barrier) { - struct dp_offload_thread_item *item; - struct dp_offload_flush_item *flush; + unsigned int tid; + long long int now_us = time_usec(); - if (ovsthread_once_start(&offload_thread_once)) { - mpsc_queue_init(&dp_offload_thread.queue); - ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); - ovsthread_once_done(&offload_thread_once); - } + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + struct dp_offload_thread_item *item; + struct dp_offload_flush_item *flush; - item = xmalloc(sizeof *item + sizeof *flush); - item->type = DP_OFFLOAD_FLUSH; - item->timestamp = time_usec(); + item = xmalloc(sizeof *item + sizeof *flush); + item->type = DP_OFFLOAD_FLUSH; + item->timestamp = now_us; - flush = &item->data->flush; - flush->dp = dp; - flush->netdev = netdev; - flush->barrier = barrier; + flush = &item->data->flush; + flush->dp = dp; + flush->netdev = netdev; + flush->barrier = barrier; - dp_netdev_append_offload(item); + dp_netdev_append_offload(item, tid); + } } /* Blocking call that will wait on the offload thread to @@ -3019,13 +3068,17 @@ dp_netdev_offload_flush(struct dp_netdev *dp, struct dp_netdev_port *port) OVS_REQ_WRLOCK(dp->port_rwlock) { - /* The flush mutex only serves to protect the static memory barrier. + /* The flush mutex serves to exclude mutual access to the static + * barrier, and to prevent multiple flush orders to several threads. + * * The memory barrier needs to go beyond the function scope as - * the other thread can resume from blocking after this function + * the other threads can resume from blocking after this function * already finished. - * As the barrier is made static, then it will be shared by - * calls to this function, and it needs to be protected from - * concurrent use. + * + * Additionally, because the flush operation is blocking, it would + * deadlock if multiple offload threads were blocking on several + * different barriers. Only allow a single flush order in the offload + * queue at a time. */ static struct ovs_mutex flush_mutex = OVS_MUTEX_INITIALIZER; static struct ovs_barrier barrier OVS_GUARDED_BY(flush_mutex); @@ -3038,8 +3091,8 @@ dp_netdev_offload_flush(struct dp_netdev *dp, ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_lock(&flush_mutex); - /* This thread and the offload thread. */ - ovs_barrier_init(&barrier, 2); + /* This thread and the offload threads. */ + ovs_barrier_init(&barrier, 1 + netdev_offload_thread_nb()); netdev = netdev_ref(port->netdev); dp_netdev_offload_flush_enqueue(dp, netdev, &barrier); @@ -3047,7 +3100,7 @@ dp_netdev_offload_flush(struct dp_netdev *dp, netdev_close(netdev); /* Take back the datapath port lock before allowing the offload - * thread to proceed further. The port deletion must complete first, + * threads to proceed further. The port deletion must complete first, * to ensure no further offloads are inserted after the flush. * * Some offload provider (e.g. DPDK) keeps a netdev reference with @@ -4434,60 +4487,99 @@ dpif_netdev_offload_stats_get(struct dpif *dpif, DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN, DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV, }; - const char *names[] = { + struct { + const char *name; + uint64_t total; + } hwol_stats[] = { [DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED] = - " Enqueued offloads", + { " Enqueued offloads", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_INSERTED] = - " Inserted offloads", + { " Inserted offloads", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN] = - " Cumulative Average latency (us)", + { " Cumulative Average latency (us)", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV] = - " Cumulative Latency stddev (us)", + { " Cumulative Latency stddev (us)", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN] = - " Exponential Average latency (us)", + { " Exponential Average latency (us)", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV] = - " Exponential Latency stddev (us)", + { " Exponential Latency stddev (us)", 0 }, }; struct dp_netdev *dp = get_dp_netdev(dpif); struct dp_netdev_port *port; - uint64_t nb_offloads; + unsigned int nb_thread; + uint64_t *port_nb_offloads; + uint64_t *nb_offloads; + unsigned int tid; size_t i; if (!netdev_is_flow_api_enabled()) { return EINVAL; } - stats->size = ARRAY_SIZE(names); + nb_thread = netdev_offload_thread_nb(); + /* nb_thread counters for the overall total as well. */ + stats->size = ARRAY_SIZE(hwol_stats) * (nb_thread + 1); stats->counters = xcalloc(stats->size, sizeof *stats->counters); - nb_offloads = 0; + nb_offloads = xcalloc(nb_thread, sizeof *nb_offloads); + port_nb_offloads = xcalloc(nb_thread, sizeof *port_nb_offloads); ovs_rwlock_rdlock(&dp->port_rwlock); HMAP_FOR_EACH (port, node, &dp->ports) { - uint64_t port_nb_offloads = 0; - + memset(port_nb_offloads, 0, nb_thread * sizeof *port_nb_offloads); /* Do not abort on read error from a port, just report 0. */ - if (!netdev_flow_get_n_flows(port->netdev, &port_nb_offloads)) { - nb_offloads += port_nb_offloads; + if (!netdev_flow_get_n_flows(port->netdev, port_nb_offloads)) { + for (i = 0; i < nb_thread; i++) { + nb_offloads[i] += port_nb_offloads[i]; + } } } ovs_rwlock_unlock(&dp->port_rwlock); - atomic_read_relaxed(&dp_offload_thread.enqueued_item, - &stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED].value = nb_offloads; - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].value = - mov_avg_cma(&dp_offload_thread.cma); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV].value = - mov_avg_cma_std_dev(&dp_offload_thread.cma); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN].value = - mov_avg_ema(&dp_offload_thread.ema); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV].value = - mov_avg_ema_std_dev(&dp_offload_thread.ema); - - for (i = 0; i < ARRAY_SIZE(names); i++) { + free(port_nb_offloads); + + for (tid = 0; tid < nb_thread; tid++) { + uint64_t counts[ARRAY_SIZE(hwol_stats)]; + size_t idx = ((tid + 1) * ARRAY_SIZE(hwol_stats)); + + memset(counts, 0, sizeof counts); + counts[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED] = nb_offloads[tid]; + if (dp_offload_threads != NULL) { + atomic_read_relaxed(&dp_offload_threads[tid].enqueued_item, + &counts[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED]); + + counts[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN] = + mov_avg_cma(&dp_offload_threads[tid].cma); + counts[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV] = + mov_avg_cma_std_dev(&dp_offload_threads[tid].cma); + + counts[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN] = + mov_avg_ema(&dp_offload_threads[tid].ema); + counts[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV] = + mov_avg_ema_std_dev(&dp_offload_threads[tid].ema); + } + + for (i = 0; i < ARRAY_SIZE(hwol_stats); i++) { + snprintf(stats->counters[idx + i].name, + sizeof(stats->counters[idx + i].name), + " [%3u] %s", tid, hwol_stats[i].name); + stats->counters[idx + i].value = counts[i]; + hwol_stats[i].total += counts[i]; + } + } + + free(nb_offloads); + + /* Do an average of the average for the aggregate. */ + hwol_stats[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].total /= nb_thread; + hwol_stats[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV].total /= nb_thread; + hwol_stats[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN].total /= nb_thread; + hwol_stats[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV].total /= nb_thread; + + for (i = 0; i < ARRAY_SIZE(hwol_stats); i++) { snprintf(stats->counters[i].name, sizeof(stats->counters[i].name), - "%s", names[i]); + " Total %s", hwol_stats[i].name); + stats->counters[i].value = hwol_stats[i].total; } return 0; diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index 70e6843f5..f4452553b 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -59,6 +59,7 @@ struct ufid_to_rte_flow_data { bool actions_offloaded; struct dpif_flow_stats stats; struct ovs_mutex lock; + unsigned int creation_tid; bool dead; }; @@ -235,6 +236,7 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, data->netdev = netdev_ref(netdev); data->rte_flow = rte_flow; data->actions_offloaded = actions_offloaded; + data->creation_tid = netdev_offload_thread_id(); ovs_mutex_init(&data->lock); cmap_insert(map, CONST_CAST(struct cmap_node *, &data->node), hash); @@ -1772,13 +1774,16 @@ netdev_offload_dpdk_flow_flush(struct netdev *netdev) { struct cmap *map = offload_data_map(netdev); struct ufid_to_rte_flow_data *data; + unsigned int tid = netdev_offload_thread_id(); if (!map) { return -1; } CMAP_FOR_EACH (data, node, map) { - netdev_offload_dpdk_flow_destroy(data); + if (data->creation_tid == tid) { + netdev_offload_dpdk_flow_destroy(data); + } } return 0; From patchwork Mon Apr 12 15:20:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1465353 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm1 header.b=j5AQu76K; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm2 header.b=TinZLSYF; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FJssR6kcxz9sTD for ; Tue, 13 Apr 2021 01:21:47 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 517FE40E6B; Mon, 12 Apr 2021 15:21:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4b17hpwqfJ-d; Mon, 12 Apr 2021 15:21:45 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTP id 2B71940597; Mon, 12 Apr 2021 15:21:16 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 05522C001D; Mon, 12 Apr 2021 15:21:13 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 27429C001D for ; Mon, 12 Apr 2021 15:21:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1E4F460B03 for ; Mon, 12 Apr 2021 15:20:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="j5AQu76K"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="TinZLSYF" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id APA7WImvCT14 for ; Mon, 12 Apr 2021 15:20:40 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4768160AE7 for ; Mon, 12 Apr 2021 15:20:40 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 54D1F5C01AF; Mon, 12 Apr 2021 11:20:39 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 12 Apr 2021 11:20:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm1; bh=UJ5umZKVetgPE sH4/a4K0jrjSCdXkrRzRMQ386/R/yU=; b=j5AQu76KW2i8zs5sUZyGqnzJdwo3H ySWSKW30TWP4icq3bJBAmx02xx7zlN260kGtkMcf2UhslZ691+ceUr30WoTbb+s6 UNbj1Z9jlQqeshG+P7h5BWgwQJ9ha5E/6cmXiU5E2Ny0mxjRQsBZsf4ejs0nkzQ1 VilsqbQtYfOZ3FNTMUhTfuaahia+iQ00EoIC+c4QIKRQ6jSKC9cKFeCG8TR0HOiq hnoGbbowgOjzsC8djBpGCEFJu840gZtlr0Dr96riNpJp62P/zpJur9jTaRI5ouiu wKsteVDk38SBzjye7op+g0qot46HFhovv4h4O74a141F80PPPDxFiN3rA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=UJ5umZKVetgPEsH4/a4K0jrjSCdXkrRzRMQ386/R/yU=; b=TinZLSYF 8PlupBY1QixCXcHhSkQwjF7trYmehhJmrPEocLDTctnMIUl7FqCDPD6ZviBqDVLX 8FGEqMDNNjRfhPZnr7WEPgfUj+DqGQMyr0eHTLU/1xgKS2KhOKCrNtAHy8YEABEy Jx4qIfmzb2wsVSx6udu+GLWmVnJb2VjunOarJ+6HkNtyXgeWCbAwILGQ4iU7ggMs gGb7znzcWQCjrBQtGiLYJpG3O647Srnn+N2dmhW95V6UctAtyyAhtYeFB5/YZpzZ Sku4Re1/pnvrDiSsxnFo1oY/Sd5WUF/ykKiEUozBt75y5iuWpNjPKy5qmjo2WzYO ecDijmtPBfyZ3w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudekjedgkeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epgedtueehiefgieefffevhfehtedvuddtvdfhuefgueeffffhkeeigfejudfhgeeunecu ffhomhgrihhnpeguphgukhdrohhrghenucfkphepkeeirddvheegrddvgeefrddugeeine cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhv vgesuhdvheeirdhnvght X-ME-Proxy: Received: from inocybe.home (lfbn-poi-1-930-146.w86-254.abo.wanadoo.fr [86.254.243.146]) by mail.messagingengine.com (Postfix) with ESMTPA id AF613240066; Mon, 12 Apr 2021 11:20:38 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Mon, 12 Apr 2021 17:20:14 +0200 Message-Id: <3d36b8b31e1e0261ff265b5e87e641b2c6494689.1618236421.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , maxime.coquelin@redhat.com, david.marchand@redhat.com Subject: [ovs-dev] [PATCH v2 28/28] netdev-dpdk: Remove rte-flow API access locks X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The rte_flow DPDK API was made thread-safe [1] in release 20.11. Now that the DPDK offload provider in OVS is thread safe, remove the locks. [1]: http://mails.dpdk.org/archives/dev/2020-October/184251.html Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-dpdk.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 9d8096668..c7ebeb4d5 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -5239,9 +5239,7 @@ netdev_dpdk_rte_flow_destroy(struct netdev *netdev, struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); int ret; - ovs_mutex_lock(&dev->mutex); ret = rte_flow_destroy(dev->port_id, rte_flow, error); - ovs_mutex_unlock(&dev->mutex); return ret; } @@ -5255,9 +5253,7 @@ netdev_dpdk_rte_flow_create(struct netdev *netdev, struct rte_flow *flow; struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); - ovs_mutex_lock(&dev->mutex); flow = rte_flow_create(dev->port_id, attr, items, actions, error); - ovs_mutex_unlock(&dev->mutex); return flow; } @@ -5285,9 +5281,7 @@ netdev_dpdk_rte_flow_query_count(struct netdev *netdev, } dev = netdev_dpdk_cast(netdev); - ovs_mutex_lock(&dev->mutex); ret = rte_flow_query(dev->port_id, rte_flow, actions, query, error); - ovs_mutex_unlock(&dev->mutex); return ret; }