From patchwork Wed Jun 9 13:09:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489857 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=wuTw4uID; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=A9M/+zfs; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SBg0NMDz9sT6 for ; Wed, 9 Jun 2021 23:10:01 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 09DCC608F2; Wed, 9 Jun 2021 13:09:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ENVZvwL67xLM; Wed, 9 Jun 2021 13:09:55 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id CCC15608FB; Wed, 9 Jun 2021 13:09:54 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F0C41C0026; Wed, 9 Jun 2021 13:09:52 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id A07B8C000B for ; Wed, 9 Jun 2021 13:09:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 8FB4660637 for ; Wed, 9 Jun 2021 13:09:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M5vJ8c3JwewH for ; Wed, 9 Jun 2021 13:09:46 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp3.osuosl.org (Postfix) with ESMTPS id 9AFA360633 for ; Wed, 9 Jun 2021 13:09:46 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 6A2BF265E; Wed, 9 Jun 2021 09:09:45 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 09 Jun 2021 09:09:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=RfJR2N9ZtfI8g JnzZTpO+Rw+u0FSWJMgJE2t73bPgu8=; b=wuTw4uIDQDwwKthU7vx0UmFMjcCUG qP3ozzxXJVvywCZ0bpwYyEXnGdAFN+EQ4hcHRYZvK/sJr/yQhjZOdVM2eV8aKxwG qqTfvojABgdqh0tREGBJE7o0d70TeM4mqAFRtiMZZkwNrVhVtqJlKthFuUyz86Xd GmRIcGNJBQwPKumEVicOvGB7CvzFurI8G4aCxKqd0brbAKWdbYVvZyVggJMZLU4z EJcC3w5E/K6tHPwK3gFLOMTnm5yJ+MezU+wEUe47UD+mlu0Tz/RZLyaZc1dTb8J8 DrCuEC4F3O/8KL1Yy/4C3kitCTNtJnBTs0qhlyI/Cfo2Y1pa4l/GDEJNA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=RfJR2N9ZtfI8gJnzZTpO+Rw+u0FSWJMgJE2t73bPgu8=; b=A9M/+zfs WYBtjU6+uouHkE27F91cS2OO0xMXRkYt7FmZSSDpjwSf1aZglmUNDgHaQzElHdUS 0kQ9RkZ5k/w72tjeVEovosMgH1lVzdlOLXsTTUb/Z+KfG6FGT1V3UwRlWeBbX6el lJVh3aPExxJQUSvJYtQW0dOPCdwTj6v+i05Os0hJc8/C/V9hmIvxBxAiP0ii4ZO/ AlcioKkUx8fZinCm5sYEyCTZtL0ttmldacYSDoU1qloKEKytDjyXXlWdXd5cDaWZ 7WceptjTN2gXBcGkYRmwgmjvaKMSDsxpT2BIhtm9qkKHddaDkJujHzzxVlZqwyhj /GRS0cZ25IYaTw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:44 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:09 +0200 Message-Id: <5edc113c6f8e2c0b8ac47a8c7ddef382e5fadde5.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Maxime Coquelin Subject: [ovs-dev] [PATCH v4 01/27] ovs-thread: Fix barrier use-after-free X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When a thread is blocked on a barrier, there is no guarantee regarding the moment it will resume, only that it will at some point in the future. One thread can resume first then proceed to destroy the barrier while another thread has not yet awoken. When it finally happens, the second thread will attempt a seq_read() on the barrier seq, while the first thread have already destroyed it, triggering a use-after-free. Introduce an additional indirection layer within the barrier. A internal barrier implementation holds all the necessary elements for a thread to safely block and destroy. Whenever a barrier is destroyed, the internal implementation is left available to still blocking threads if necessary. A reference counter is used to track threads still using the implementation. Note that current uses of ovs-barrier are not affected: RCU and revalidators will not destroy their barrier immediately after blocking on it. Fixes: d8043da7182a ("ovs-thread: Implement OVS specific barrier.") Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/ovs-thread.c | 61 +++++++++++++++++++++++++++++++++++++++--------- lib/ovs-thread.h | 6 ++--- 2 files changed, 53 insertions(+), 14 deletions(-) diff --git a/lib/ovs-thread.c b/lib/ovs-thread.c index b686e4548..805cba622 100644 --- a/lib/ovs-thread.c +++ b/lib/ovs-thread.c @@ -299,21 +299,53 @@ ovs_spin_init(const struct ovs_spin *spin) } #endif +struct ovs_barrier_impl { + uint32_t size; /* Number of threads to wait. */ + atomic_count count; /* Number of threads already hit the barrier. */ + struct seq *seq; + struct ovs_refcount refcnt; +}; + +static void +ovs_barrier_impl_ref(struct ovs_barrier_impl *impl) +{ + ovs_refcount_ref(&impl->refcnt); +} + +static void +ovs_barrier_impl_unref(struct ovs_barrier_impl *impl) +{ + if (ovs_refcount_unref(&impl->refcnt) == 1) { + seq_destroy(impl->seq); + free(impl); + } +} + /* Initializes the 'barrier'. 'size' is the number of threads * expected to hit the barrier. */ void ovs_barrier_init(struct ovs_barrier *barrier, uint32_t size) { - barrier->size = size; - atomic_count_init(&barrier->count, 0); - barrier->seq = seq_create(); + struct ovs_barrier_impl *impl; + + impl = xmalloc(sizeof *impl); + impl->size = size; + atomic_count_init(&impl->count, 0); + impl->seq = seq_create(); + ovs_refcount_init(&impl->refcnt); + + ovsrcu_set(&barrier->impl, impl); } /* Destroys the 'barrier'. */ void ovs_barrier_destroy(struct ovs_barrier *barrier) { - seq_destroy(barrier->seq); + struct ovs_barrier_impl *impl; + + impl = ovsrcu_get(struct ovs_barrier_impl *, &barrier->impl); + ovsrcu_set(&barrier->impl, NULL); + ovs_barrier_impl_unref(impl); } /* Makes the calling thread block on the 'barrier' until all @@ -325,23 +357,30 @@ ovs_barrier_destroy(struct ovs_barrier *barrier) void ovs_barrier_block(struct ovs_barrier *barrier) { - uint64_t seq = seq_read(barrier->seq); + struct ovs_barrier_impl *impl; uint32_t orig; + uint64_t seq; - orig = atomic_count_inc(&barrier->count); - if (orig + 1 == barrier->size) { - atomic_count_set(&barrier->count, 0); + impl = ovsrcu_get(struct ovs_barrier_impl *, &barrier->impl); + ovs_barrier_impl_ref(impl); + + seq = seq_read(impl->seq); + orig = atomic_count_inc(&impl->count); + if (orig + 1 == impl->size) { + atomic_count_set(&impl->count, 0); /* seq_change() serves as a release barrier against the other threads, * so the zeroed count is visible to them as they continue. */ - seq_change(barrier->seq); + seq_change(impl->seq); } else { /* To prevent thread from waking up by other event, * keeps waiting for the change of 'barrier->seq'. */ - while (seq == seq_read(barrier->seq)) { - seq_wait(barrier->seq, seq); + while (seq == seq_read(impl->seq)) { + seq_wait(impl->seq, seq); poll_block(); } } + + ovs_barrier_impl_unref(impl); } DEFINE_EXTERN_PER_THREAD_DATA(ovsthread_id, OVSTHREAD_ID_UNSET); diff --git a/lib/ovs-thread.h b/lib/ovs-thread.h index 7ee98bd4e..3b444ccdc 100644 --- a/lib/ovs-thread.h +++ b/lib/ovs-thread.h @@ -21,16 +21,16 @@ #include #include #include "ovs-atomic.h" +#include "ovs-rcu.h" #include "openvswitch/thread.h" #include "util.h" struct seq; /* Poll-block()-able barrier similar to pthread_barrier_t. */ +struct ovs_barrier_impl; struct ovs_barrier { - uint32_t size; /* Number of threads to wait. */ - atomic_count count; /* Number of threads already hit the barrier. */ - struct seq *seq; + OVSRCU_TYPE(struct ovs_barrier_impl *) impl; }; /* Wrappers for pthread_mutexattr_*() that abort the process on any error. */ From patchwork Wed Jun 9 13:09:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489861 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=I1F7iueC; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=cxtZJPlD; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SBm6SrCz9sWD for ; Wed, 9 Jun 2021 23:10:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 4658A83CEE; Wed, 9 Jun 2021 13:10:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kGIwnh43vsw6; Wed, 9 Jun 2021 13:10:02 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3879483CA8; Wed, 9 Jun 2021 13:09:58 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CBDECC000B; Wed, 9 Jun 2021 13:09:57 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 53664C0030 for ; Wed, 9 Jun 2021 13:09:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 2253B40528 for ; Wed, 9 Jun 2021 13:09:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="I1F7iueC"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="cxtZJPlD" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZbU6b1wT_vUe for ; Wed, 9 Jun 2021 13:09:51 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp4.osuosl.org (Postfix) with ESMTPS id 58BCD40482 for ; Wed, 9 Jun 2021 13:09:51 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id EB7452669; Wed, 9 Jun 2021 09:09:47 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 09 Jun 2021 09:09:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=CM82Dsb1GM774 PR/sdFGrlh9U82rbxB17dbarNRxQMY=; b=I1F7iueCh8PJ9ruQYP73/k7bUZjHA s9Igi/24dI25BSp+UATOnmdNM5a7cuK8XSpM4j0uvqK93i4Pqo378XzXSzG1F5eQ 8G+kgLp5oOo2sIRLpVqNtscoCCeBZVJB6QgolwC+alS53o1jlhPPlJ3UhD9o/7wQ OkNJaa3T6LpWkR9iBHECDb6ouSgw2JofnrKzk3QAft1oBfBaSYcA9W8XBDmlHsFJ B7MpWWYBeoZyMHhnFOLYrJz7AWc+WdEVz/yNMPUUlZwjsCB1KDKfwUrUMUWxYNa1 JtiqM3GlLNmpB5WajipOOEFAGu891YXTiuotbYD/v0/wb5nqWgV5ouQ7A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=CM82Dsb1GM774PR/sdFGrlh9U82rbxB17dbarNRxQMY=; b=cxtZJPlD obC+UDC7hC/KYe6lT/GhpzJxr135aPdSurw9D4TXStgiH/OzNwWCVlOcad9HfbWx DQFnYAT2M/PL4GKpSkyjbCod3bD16JuuMeLHGZXL41qrKACnZ5Dv1Nbft7dYa3b6 K27NHjgDN2WTQ7dUGDGgXvUCkKKIK+yNFTgKQloCURkpFDNbUbvjGIKopBYGN6l3 ogMQt5eW1ZPTd3fj0A/mzDHNi60yPKrhZQ1mn+CXovBLZBWNhMSopOW918kQwMCd SPp0cXXD2lKEgCQycR2bqAs3Vats+X9wsfrEro5l8j0/UtKriE8suMf3d7V0Vvzj HmOv4iHPl6/j+Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:45 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:10 +0200 Message-Id: <1666dbfbf8eef11826a3dfff5f6cab2e4fe48cb7.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Maxime Coquelin Subject: [ovs-dev] [PATCH v4 02/27] dpif-netdev: Rename flow offload thread X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" ovs_strlcpy silently fails to copy the thread name if it is too long. Rename the flow offload thread to differentiate it from the main thread. Fixes: 02bb2824e51d ("dpif-netdev: do hw flow offload in a thread") Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 650e67ab3..d2c480529 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2786,8 +2786,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, if (ovsthread_once_start(&offload_thread_once)) { xpthread_cond_init(&dp_flow_offload.cond, NULL); - ovs_thread_create("dp_netdev_flow_offload", - dp_netdev_flow_offload_main, NULL); + ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -2810,8 +2809,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, if (ovsthread_once_start(&offload_thread_once)) { xpthread_cond_init(&dp_flow_offload.cond, NULL); - ovs_thread_create("dp_netdev_flow_offload", - dp_netdev_flow_offload_main, NULL); + ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } From patchwork Wed Jun 9 13:09:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489860 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=mM/7J85T; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=tSBFjXh3; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SBl4tglz9sW8 for ; Wed, 9 Jun 2021 23:10:07 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 68A9160A6D; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rTDnmUi_zoiR; Wed, 9 Jun 2021 13:09:57 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6F0E4608EF; Wed, 9 Jun 2021 13:09:56 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0BE35C000B; Wed, 9 Jun 2021 13:09:55 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id D80BDC002A for ; Wed, 9 Jun 2021 13:09:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id B402C404FC for ; Wed, 9 Jun 2021 13:09:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="mM/7J85T"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="tSBFjXh3" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MNdLPWkFzQPm for ; Wed, 9 Jun 2021 13:09:50 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2822440137 for ; Wed, 9 Jun 2021 13:09:50 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 5679C2663; Wed, 9 Jun 2021 09:09:49 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 09 Jun 2021 09:09:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=cb9+vyzif3gFc Hl40kdd94+ajBmkdCFd/ecwuCDXrLM=; b=mM/7J85T8e9ijEwEo+olhQGJVIeoE 5SzAEqOlP3cPH1qdYdilPS8rK7aTdM4ZL4qU2L3gltWUUDkkUQTLSE6JlPRNZcFl NtSZ8m6nsuxK+TgivJ2osFAVQj+VBdVn7a/10mKJv1UiiiljgtCezkGOuFIW0vxB JqU0KBLRpQKHSi0PFRrNrB14VpeP9cs2AFuaIEEg7wAKxL+XUslZt2FgRcW6HGz9 2VRBKit+w1UbtoxBUzogsjCaxaWkwWElcvZ5M7aGwIOQVbbpPBx1d/y74JUOYID4 +5Znvx1OSvLWlT1Nm97PQQ6JmaKyXIvOck6FBQGWXAEFhUIsbj2MY3yYg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=cb9+vyzif3gFcHl40kdd94+ajBmkdCFd/ecwuCDXrLM=; b=tSBFjXh3 1RbuqUy64sNybh44Wn9EbOePbf78astg/LLeV2nBwnFmMmXrJaS/UloLPwv87f3T hlSG3ia+pclLkcBI3lPVRqxAeFCREJixMccJVPMchHpu7ZzsWvFRJyjEszhNz6rx 9kyNpwNMaL+vyhFwYgpqe+NVMHcceYyFesJpfSZxgRgwZB0RVcHKM/5IpkGVACwG AE3lXibz2oNMlF2uHWLqZMB4KJQlJhMa4Nuy/qotMu9HSpPHuItkudkSFIbpvolH 1bltR66Inw+oBnGiMPOknObafu8IrwB0PlRQf2dwZ8ghdgWfv8XZv+yxxnDnK8g0 4Nbi2+QQFoZg1Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgiedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epudfhfeeikeeiiedutefgheekledukedtkeejheduheffteeiudehiefgkedvleefnecu ffhomhgrihhnpegrphgrtghhvgdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurf grrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:48 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:11 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Maxime Coquelin Subject: [ovs-dev] [PATCH v4 03/27] tests: Add ovs-barrier unit test X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" No unit test exist currently for the ovs-barrier type. It is however crucial as a building block and should be verified to work as expected. Create a simple test verifying the basic function of ovs-barrier. Integrate the test as part of the test suite. Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- tests/automake.mk | 1 + tests/library.at | 5 + tests/test-barrier.c | 264 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 270 insertions(+) create mode 100644 tests/test-barrier.c diff --git a/tests/automake.mk b/tests/automake.mk index 1a528aa39..a32abd41c 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -448,6 +448,7 @@ tests_ovstest_SOURCES = \ tests/ovstest.h \ tests/test-aes128.c \ tests/test-atomic.c \ + tests/test-barrier.c \ tests/test-bundle.c \ tests/test-byte-order.c \ tests/test-classifier.c \ diff --git a/tests/library.at b/tests/library.at index 1702b7556..e572c22e3 100644 --- a/tests/library.at +++ b/tests/library.at @@ -246,6 +246,11 @@ AT_SETUP([ofpbuf module]) AT_CHECK([ovstest test-ofpbuf], [0], []) AT_CLEANUP +AT_SETUP([barrier module]) +AT_KEYWORDS([barrier]) +AT_CHECK([ovstest test-barrier], [0], []) +AT_CLEANUP + AT_SETUP([rcu]) AT_CHECK([ovstest test-rcu-quiesce], [0], []) AT_CLEANUP diff --git a/tests/test-barrier.c b/tests/test-barrier.c new file mode 100644 index 000000000..3bc5291cc --- /dev/null +++ b/tests/test-barrier.c @@ -0,0 +1,264 @@ +/* + * Copyright (c) 2021 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include + +#include "ovs-thread.h" +#include "ovs-rcu.h" +#include "ovstest.h" +#include "random.h" +#include "util.h" + +#define DEFAULT_N_THREADS 4 +#define NB_STEPS 4 + +static bool verbose; +static struct ovs_barrier barrier; + +struct blocker_aux { + unsigned int tid; + bool leader; + int step; +}; + +static void * +basic_blocker_main(void *aux_) +{ + struct blocker_aux *aux = aux_; + size_t i; + + aux->step = 0; + for (i = 0; i < NB_STEPS; i++) { + ovs_barrier_block(&barrier); + aux->step++; + ovs_barrier_block(&barrier); + } + + return NULL; +} + +static void +basic_block_check(struct blocker_aux *aux, size_t n, int expected) +{ + size_t i; + + for (i = 0; i < n; i++) { + if (verbose) { + printf("aux[%" PRIuSIZE "]=%d == %d", i, aux[i].step, expected); + if (aux[i].step != expected) { + printf(" <--- X"); + } + printf("\n"); + } else { + ovs_assert(aux[i].step == expected); + } + } + ovs_barrier_block(&barrier); + ovs_barrier_block(&barrier); +} + +/* + * Basic barrier test. + * + * N writers and 1 reader participate in the test. + * Each thread goes through M steps (=NB_STEPS). + * The main thread participates as the reader. + * + * A Step is divided in three parts: + * 1. before + * (barrier) + * 2. during + * (barrier) + * 3. after + * + * Each writer updates a thread-local variable with the + * current step number within part 2 and waits. + * + * The reader checks all variables during part 3, expecting + * all variables to be equal. If any variable differs, it means + * its thread was not properly blocked by the barrier. + */ +static void +test_barrier_basic(size_t n_threads) +{ + struct blocker_aux *aux; + pthread_t *threads; + size_t i; + + ovs_barrier_init(&barrier, n_threads + 1); + + aux = xcalloc(n_threads, sizeof *aux); + threads = xmalloc(n_threads * sizeof *threads); + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("ovs-barrier", + basic_blocker_main, &aux[i]); + } + + for (i = 0; i < NB_STEPS; i++) { + basic_block_check(aux, n_threads, i); + } + ovs_barrier_destroy(&barrier); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + free(threads); + free(aux); +} + +static unsigned int *shared_mem; + +static void * +lead_blocker_main(void *aux_) +{ + struct blocker_aux *aux = aux_; + size_t i; + + aux->step = 0; + for (i = 0; i < NB_STEPS; i++) { + if (aux->leader) { + shared_mem = xmalloc(sizeof *shared_mem); + if (verbose) { + printf("*T1: allocated shmem\n"); + } + } + xnanosleep(random_range(100) * 1000); + + ovs_barrier_block(&barrier); + + if (verbose) { + printf("%cT%u: ENTER, writing\n", + (aux->leader ? '*' : ' '), aux->tid); + } + + shared_mem[0] = 42; + + ovs_barrier_block(&barrier); + + if (verbose) { + printf("%cT%u: EXIT\n", + (aux->leader ? '*' : ' '), aux->tid); + } + + if (aux->leader) { + free(shared_mem); + if (verbose) { + printf("*T1: freed shmem\n"); + } + } + xnanosleep(random_range(100) * 1000); + } + + return NULL; +} + +/* + * Leader barrier test. + * + * N threads participates, one of which is marked as + * the leader (thread 0). The main thread does not + * participate. + * + * The test is divided in M steps (=NB_STEPS). + * A Step is divided in three parts: + * 1. before + * (barrier) + * 2. during + * (barrier) + * 3. after + * + * Part 1, the leader allocates a block of shared memory. + * Part 2, all threads write to the shared memory. + * Part 3: the leader frees the shared memory. + * + * If any thread is improperly blocked by the barrier, + * the shared memory accesses will trigger a segfault + * or a use-after-free if ASAN is enabled. + */ +static void +test_barrier_lead(size_t n_threads) +{ + struct blocker_aux *aux; + pthread_t *threads; + size_t i; + + ovs_barrier_init(&barrier, n_threads); + + aux = xcalloc(n_threads, sizeof *aux); + threads = xmalloc(n_threads * sizeof *threads); + + aux[0].leader = true; + + for (i = 0; i < n_threads; i++) { + aux[i].tid = i + 1; + threads[i] = ovs_thread_create("ovs-barrier", + lead_blocker_main, &aux[i]); + } + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + /* If the main thread does not participate in the barrier, + * it must wait for all threads to join before destroying it. + */ + ovs_barrier_destroy(&barrier); + + free(threads); + free(aux); +} + +static void +usage(char *test_name) +{ + fprintf(stderr, "Usage: %s [n_threads=%d] [-v]\n", + test_name, DEFAULT_N_THREADS); +} + +static void +test_barrier(int argc, char *argv[]) +{ + size_t n_threads = DEFAULT_N_THREADS; + char **args = argv + optind - 1; + + set_program_name(argv[0]); + + argc -= optind; + if (argc > 2) { + usage(args[0]); + return; + } + + while (argc-- > 0) { + args++; + if (!strcmp(args[0], "-v")) { + verbose = true; + } else { + n_threads = strtol(args[0], NULL, 10); + if (n_threads > 20) { + n_threads = 20; + } + } + } + + test_barrier_basic(n_threads); + test_barrier_lead(n_threads); +} + +OVSTEST_REGISTER("test-barrier", test_barrier); From patchwork Wed Jun 9 13:09:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489859 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=Cuw+BKU8; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=Zj7UKQaU; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SBk6pmPz9sW6 for ; Wed, 9 Jun 2021 23:10:06 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 58BB14051B; Wed, 9 Jun 2021 13:10:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XcmTqN7Dv5Lv; Wed, 9 Jun 2021 13:10:00 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id AE2FD40545; Wed, 9 Jun 2021 13:09:59 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id A5BD8C0026; Wed, 9 Jun 2021 13:09:58 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8B2D4C0024 for ; Wed, 9 Jun 2021 13:09:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 73CEA60906 for ; Wed, 9 Jun 2021 13:09:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="Cuw+BKU8"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="Zj7UKQaU" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NEGwxbDn6BB4 for ; Wed, 9 Jun 2021 13:09:51 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp3.osuosl.org (Postfix) with ESMTPS id AB20360633 for ; Wed, 9 Jun 2021 13:09:51 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 1CEB7266B; Wed, 9 Jun 2021 09:09:51 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:09:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=1CVhXqjqN8Cyl m3rEXU6/5cS2caLddEeIxwf9wf5A+g=; b=Cuw+BKU8Y9/odFY8oFLL3CuIjYNPx 5T412RKPlv8+eROVjGQKnSAbhfhnXF3+5R42iAKmXV1vL9hxsoVbwvBo4qkUwRui W3U9deG+YI+Pl6vnJKi31ZDJ8xUjkKIGuqYrXqqxJQlXVbtGQOU/+2mvHq0GJofG fdyAZrtNuWwrofAZPUsZRnGQPtXw9ny9Na2ZZeyNDH3Pm46KXaMgl0atljb+BWQc NAp4dMH/G52KOK/eoFbqoP69EODBv1LV1gm2FC2dqnVCkQKiSPbBGKkAqw3MeoBP 54eyQtZTLsgyjnz5+PYT0RBxhFs4hrl2EeO6fiONchHn28JHA0R62w9GA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=1CVhXqjqN8Cylm3rEXU6/5cS2caLddEeIxwf9wf5A+g=; b=Zj7UKQaU OJ2FJ+UFu27mLJ2VDdVMOcleF5DkGoC+P3mgOxirpQxJYiSOOeO5CRjwxOcgdaU9 sBlGb4EDfVbv7gNljmPldM7eXw7THVu8kHldmz2eNHXhr85J5w25bGaphAjj6HkP Pj2Guz+IHx4ABAVRr2Y83Y+sL5W9EtbV8ZE/nVuaUONkX9Z4PmvP7MqwWZN4k8hW WoZUZhjTn9mr5Bq5wlx41Yxt6fZigzzPHYsjdog/Zc9dKZDuClUL8HeTOU4xiDCZ Xa3yP6DCISecT5bQXOrbmt/d5UOui9HGVfzaiRYiGf9TMeMPw+vqrRB9D/RQYu7i 2AqVJfRKfQHr8w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:50 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:12 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 04/27] netdev: Add flow API uninit function X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a new operation for flow API providers to uninitialize when the API is disassociated from a netdev. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-provider.h | 3 +++ lib/netdev-offload.c | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/lib/netdev-offload-provider.h b/lib/netdev-offload-provider.h index cf859d1b4..2127599d3 100644 --- a/lib/netdev-offload-provider.h +++ b/lib/netdev-offload-provider.h @@ -90,6 +90,9 @@ struct netdev_flow_api { /* Initializies the netdev flow api. * Return 0 if successful, otherwise returns a positive errno value. */ int (*init_flow_api)(struct netdev *); + + /* Uninitializes the netdev flow api. */ + void (*uninit_flow_api)(struct netdev *); }; int netdev_register_flow_api_provider(const struct netdev_flow_api *); diff --git a/lib/netdev-offload.c b/lib/netdev-offload.c index 6237667c3..deefefd63 100644 --- a/lib/netdev-offload.c +++ b/lib/netdev-offload.c @@ -320,6 +320,10 @@ netdev_uninit_flow_api(struct netdev *netdev) return; } + if (flow_api->uninit_flow_api) { + flow_api->uninit_flow_api(netdev); + } + ovsrcu_set(&netdev->flow_api, NULL); rfa = netdev_lookup_flow_api(flow_api->type); ovs_refcount_unref(&rfa->refcnt); From patchwork Wed Jun 9 13:09:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489862 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=UTn2MCrR; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=X/borEAa; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SBp1C4gz9sT6 for ; Wed, 9 Jun 2021 23:10:09 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 84D9C40613; Wed, 9 Jun 2021 13:10:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dOgjYiJxCaap; Wed, 9 Jun 2021 13:10:04 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 62E8C4055C; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9208DC002F; Wed, 9 Jun 2021 13:10:00 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 45574C000D for ; Wed, 9 Jun 2021 13:09:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 1B466404A5 for ; Wed, 9 Jun 2021 13:09:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="UTn2MCrR"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="X/borEAa" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8HfVhc6yHaeX for ; Wed, 9 Jun 2021 13:09:53 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp4.osuosl.org (Postfix) with ESMTPS id 54C4340499 for ; Wed, 9 Jun 2021 13:09:53 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id A2FF72657; Wed, 9 Jun 2021 09:09:52 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:09:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=SSwfmotX/2fWx 9jC3zvujrV3ktD7u3MopPzHSSwK6v4=; b=UTn2MCrRIDbBysoqdBTKcdQUu47wO wimw3NME7/L2pYEgRXDOPDHACdIi6wYLHAjxG1pUnsK7ApdP2TNkN6pCOomoUV2U rF8hzwqSBLYiXL80eHxivR7ht2hPwxN6yoKhsMn0sWOXt/zDmwspzIb6FJt7aJsg HIwI/lPZUqFB685NASciI8duDFQKtX99sSA6DAv0RunprrKW9m2Vr/eEPiKHpih+ 8sKMjlGh893gEiom/MXNbWR0HKCwlHUDOu2IM0oA2JCLq92pSuBaRmvQWPooXyZM TyHlb5MruUJ5H1K6wJY7KPgy7Fn7QSjDDWgZb4hIFwXTK87hz47hulg+w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=SSwfmotX/2fWx9jC3zvujrV3ktD7u3MopPzHSSwK6v4=; b=X/borEAa 3TEAeCM+bp2Yn1sdUPwQ6MgovxhmdP/lDkWw+7HclAivDbkHZ/wJgJlKJo6L0We1 reUiK3l6mfn37Po+r8ex4lTkO/u+r2u+WqH+5hWanyjlhtgulIxdWJiFLmAzRKO4 KfCTMb4ydHWn2mOGXVuJ1MUwViDF8JUNTll3vd3dG5dlPbTpRNZgWQvNDshTnnx9 APfRiz4RNU12WMq1c4oG4jReR4514O4TZsc1AdiH03nebsimK1vhl1nWs18qwDhX SZxqNI7YD9RoBYMLKl0dalNdAH9lTrNaPbB1lD9WO4vqYXNh6k5bJuFssKWUvT4q eUm7N/0bfas1EQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:51 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:13 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 05/27] netdev-offload-dpdk: Use per-netdev offload metadata X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a per-netdev offload data field as part of netdev hw_info structure. Use this field in netdev-offload-dpdk to map offload metadata (ufid to rte_flow). Use flow API deinit ops to destroy the per-netdev metadata when deallocating a netdev. Use RCU primitives to ensure coherency during port deletion. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-dpdk.c | 131 ++++++++++++++++++++++++++++++++------ lib/netdev-offload.h | 2 + 2 files changed, 114 insertions(+), 19 deletions(-) diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index f2413f5be..01d09aca7 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -26,6 +26,7 @@ #include "netdev-provider.h" #include "openvswitch/match.h" #include "openvswitch/vlog.h" +#include "ovs-rcu.h" #include "packets.h" #include "uuid.h" @@ -52,7 +53,6 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(100, 5); /* * A mapping from ufid to dpdk rte_flow. */ -static struct cmap ufid_to_rte_flow = CMAP_INITIALIZER; struct ufid_to_rte_flow_data { struct cmap_node node; @@ -63,14 +63,81 @@ struct ufid_to_rte_flow_data { struct dpif_flow_stats stats; }; +struct netdev_offload_dpdk_data { + struct cmap ufid_to_rte_flow; +}; + +static int +offload_data_init(struct netdev *netdev) +{ + struct netdev_offload_dpdk_data *data; + + data = xzalloc(sizeof *data); + cmap_init(&data->ufid_to_rte_flow); + + ovsrcu_set(&netdev->hw_info.offload_data, (void *) data); + + return 0; +} + +static void +offload_data_destroy__(struct netdev_offload_dpdk_data *data) +{ + free(data); +} + +static void +offload_data_destroy(struct netdev *netdev) +{ + struct netdev_offload_dpdk_data *data; + struct ufid_to_rte_flow_data *node; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + if (data == NULL) { + return; + } + + if (!cmap_is_empty(&data->ufid_to_rte_flow)) { + VLOG_ERR("Incomplete flush: %s contains rte_flow elements", + netdev_get_name(netdev)); + } + + CMAP_FOR_EACH (node, node, &data->ufid_to_rte_flow) { + ovsrcu_postpone(free, node); + } + + cmap_destroy(&data->ufid_to_rte_flow); + ovsrcu_postpone(offload_data_destroy__, data); + + ovsrcu_set(&netdev->hw_info.offload_data, NULL); +} + +static struct cmap * +offload_data_map(struct netdev *netdev) +{ + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + + return data ? &data->ufid_to_rte_flow : NULL; +} + /* Find rte_flow with @ufid. */ static struct ufid_to_rte_flow_data * -ufid_to_rte_flow_data_find(const ovs_u128 *ufid, bool warn) +ufid_to_rte_flow_data_find(struct netdev *netdev, + const ovs_u128 *ufid, bool warn) { size_t hash = hash_bytes(ufid, sizeof *ufid, 0); struct ufid_to_rte_flow_data *data; + struct cmap *map = offload_data_map(netdev); + + if (!map) { + return NULL; + } - CMAP_FOR_EACH_WITH_HASH (data, node, hash, &ufid_to_rte_flow) { + CMAP_FOR_EACH_WITH_HASH (data, node, hash, map) { if (ovs_u128_equals(*ufid, data->ufid)) { return data; } @@ -85,12 +152,19 @@ ufid_to_rte_flow_data_find(const ovs_u128 *ufid, bool warn) } static inline struct ufid_to_rte_flow_data * -ufid_to_rte_flow_associate(const ovs_u128 *ufid, struct netdev *netdev, +ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, struct rte_flow *rte_flow, bool actions_offloaded) { size_t hash = hash_bytes(ufid, sizeof *ufid, 0); - struct ufid_to_rte_flow_data *data = xzalloc(sizeof *data); + struct cmap *map = offload_data_map(netdev); struct ufid_to_rte_flow_data *data_prev; + struct ufid_to_rte_flow_data *data; + + if (!map) { + return NULL; + } + + data = xzalloc(sizeof *data); /* * We should not simply overwrite an existing rte flow. @@ -98,7 +172,7 @@ ufid_to_rte_flow_associate(const ovs_u128 *ufid, struct netdev *netdev, * Thus, if following assert triggers, something is wrong: * the rte_flow is not destroyed. */ - data_prev = ufid_to_rte_flow_data_find(ufid, false); + data_prev = ufid_to_rte_flow_data_find(netdev, ufid, false); if (data_prev) { ovs_assert(data_prev->rte_flow == NULL); } @@ -108,8 +182,7 @@ ufid_to_rte_flow_associate(const ovs_u128 *ufid, struct netdev *netdev, data->rte_flow = rte_flow; data->actions_offloaded = actions_offloaded; - cmap_insert(&ufid_to_rte_flow, - CONST_CAST(struct cmap_node *, &data->node), hash); + cmap_insert(map, CONST_CAST(struct cmap_node *, &data->node), hash); return data; } @@ -117,9 +190,13 @@ static inline void ufid_to_rte_flow_disassociate(struct ufid_to_rte_flow_data *data) { size_t hash = hash_bytes(&data->ufid, sizeof data->ufid, 0); + struct cmap *map = offload_data_map(data->netdev); + + if (!map) { + return; + } - cmap_remove(&ufid_to_rte_flow, - CONST_CAST(struct cmap_node *, &data->node), hash); + cmap_remove(map, CONST_CAST(struct cmap_node *, &data->node), hash); netdev_close(data->netdev); ovsrcu_postpone(free, data); } @@ -1421,7 +1498,7 @@ netdev_offload_dpdk_add_flow(struct netdev *netdev, if (!flow) { goto out; } - flows_data = ufid_to_rte_flow_associate(ufid, netdev, flow, + flows_data = ufid_to_rte_flow_associate(netdev, ufid, flow, actions_offloaded); VLOG_DBG("%s: installed flow %p by ufid "UUID_FMT, netdev_get_name(netdev), flow, UUID_ARGS((struct uuid *)ufid)); @@ -1478,7 +1555,7 @@ netdev_offload_dpdk_flow_put(struct netdev *netdev, struct match *match, * Here destroy the old rte flow first before adding a new one. * Keep the stats for the newly created rule. */ - rte_flow_data = ufid_to_rte_flow_data_find(ufid, false); + rte_flow_data = ufid_to_rte_flow_data_find(netdev, ufid, false); if (rte_flow_data && rte_flow_data->rte_flow) { old_stats = rte_flow_data->stats; modification = true; @@ -1509,7 +1586,7 @@ netdev_offload_dpdk_flow_del(struct netdev *netdev OVS_UNUSED, { struct ufid_to_rte_flow_data *rte_flow_data; - rte_flow_data = ufid_to_rte_flow_data_find(ufid, true); + rte_flow_data = ufid_to_rte_flow_data_find(netdev, ufid, true); if (!rte_flow_data || !rte_flow_data->rte_flow) { return -1; } @@ -1523,7 +1600,21 @@ netdev_offload_dpdk_flow_del(struct netdev *netdev OVS_UNUSED, static int netdev_offload_dpdk_init_flow_api(struct netdev *netdev) { - return netdev_dpdk_flow_api_supported(netdev) ? 0 : EOPNOTSUPP; + int ret = EOPNOTSUPP; + + if (netdev_dpdk_flow_api_supported(netdev)) { + ret = offload_data_init(netdev); + } + + return ret; +} + +static void +netdev_offload_dpdk_uninit_flow_api(struct netdev *netdev) +{ + if (netdev_dpdk_flow_api_supported(netdev)) { + offload_data_destroy(netdev); + } } static int @@ -1540,7 +1631,7 @@ netdev_offload_dpdk_flow_get(struct netdev *netdev, struct rte_flow_error error; int ret = 0; - rte_flow_data = ufid_to_rte_flow_data_find(ufid, false); + rte_flow_data = ufid_to_rte_flow_data_find(netdev, ufid, false); if (!rte_flow_data || !rte_flow_data->rte_flow) { ret = -1; goto out; @@ -1575,13 +1666,14 @@ out: static int netdev_offload_dpdk_flow_flush(struct netdev *netdev) { + struct cmap *map = offload_data_map(netdev); struct ufid_to_rte_flow_data *data; - CMAP_FOR_EACH (data, node, &ufid_to_rte_flow) { - if (data->netdev != netdev) { - continue; - } + if (!map) { + return -1; + } + CMAP_FOR_EACH (data, node, map) { netdev_offload_dpdk_flow_destroy(data); } @@ -1593,6 +1685,7 @@ const struct netdev_flow_api netdev_offload_dpdk = { .flow_put = netdev_offload_dpdk_flow_put, .flow_del = netdev_offload_dpdk_flow_del, .init_flow_api = netdev_offload_dpdk_init_flow_api, + .uninit_flow_api = netdev_offload_dpdk_uninit_flow_api, .flow_get = netdev_offload_dpdk_flow_get, .flow_flush = netdev_offload_dpdk_flow_flush, }; diff --git a/lib/netdev-offload.h b/lib/netdev-offload.h index 18b48790f..d820e23ed 100644 --- a/lib/netdev-offload.h +++ b/lib/netdev-offload.h @@ -20,6 +20,7 @@ #include "openvswitch/netdev.h" #include "openvswitch/types.h" +#include "ovs-rcu.h" #include "packets.h" #include "flow.h" @@ -45,6 +46,7 @@ struct netdev_hw_info { bool oor; /* Out of Offload Resources ? */ int offload_count; /* Pending (non-offloaded) flow count */ int pending_count; /* Offloaded flow count */ + OVSRCU_TYPE(void *) offload_data; /* Offload metadata. */ }; enum hw_info_type { From patchwork Wed Jun 9 13:09:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489864 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=nf4C25xF; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=eq0T1v5L; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SBt3zGlz9sW6 for ; Wed, 9 Jun 2021 23:10:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 8A4FA83D3E; Wed, 9 Jun 2021 13:10:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3-rvNEbxE7yY; Wed, 9 Jun 2021 13:10:09 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id E3A3683CD0; Wed, 9 Jun 2021 13:10:02 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4A60EC000F; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 84EFEC002B for ; Wed, 9 Jun 2021 13:09:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6F6AE40542 for ; Wed, 9 Jun 2021 13:09:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="nf4C25xF"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="eq0T1v5L" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ShBRe7tuK3zp for ; Wed, 9 Jun 2021 13:09:55 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id EB81D40527 for ; Wed, 9 Jun 2021 13:09:54 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 5E3C6266B; Wed, 9 Jun 2021 09:09:54 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 09 Jun 2021 09:09:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=KHleDa3kFDitl 1JcBnpZIGp7yasGqnzp4zTjuKO3XVo=; b=nf4C25xFwA8vEimvC6HyNObAAoKxK zjYtsVnl+ax7NyA2gjDPpgH2SXAVSshwgfteBVIR/H+dpFyzuk+lmU2CA/HndK1P X/19dib1LPmlPmEcPXFwSPX2XZNf+n7lUuOloSm0h4Nv6TgYXWYwdmE6a7L3tL/C dWVH9QUA6/gJdTytMNLwVDV/hSZHsCmdhvhJKFM+FmWgZjZtEX2OH5MY2f6gJJyt 0XRY8sx4Pgd32EfwtGiSU4EIsJIcJ5CdVz/OP74LNrOgODzdqakpDTK9bUqtEZuu vCTCaaW06N++LXXtSQY0Hz1PGgLqNsewN+vIYy2m0KSlJQujmV2RJswqA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=KHleDa3kFDitl1JcBnpZIGp7yasGqnzp4zTjuKO3XVo=; b=eq0T1v5L DR0D/W/uyNY41eOUiptQg2OfdVXs1BcxXHZiaBVFFinA1QpIazFu0V3L6A+5s3KK vEExrX4Ij/dyChjjeoXhilhSn0y3t+QSXXvSCmCDz/oAmITie1GV5QxhQICt7oyY e8Md69xncaXbT/XAl3BNlS6mYlclb0ehWecw2q5TycUh4zTGsNxJL4U0H837jaZ4 QX2K84yo2aSFZZi6jfNiHSd6jDYRfL1KFy623BHCduAcNWdJCjir0OGPq/DfC6Mi LdgJKh7K503XbJv82NrsPA7CLm/mlBWcphODCYwyc+FlNurWsSqwKoczzJaDphgE G7kmcaU5u953RA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:53 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:14 +0200 Message-Id: <473d475731fac962f68193042c2a6469fc00bb90.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 06/27] netdev-offload-dpdk: Implement hw-offload statistics read X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In the DPDK offload provider, keep track of inserted rte_flow and report it when queried. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-dpdk.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index 01d09aca7..c43e8b968 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -65,6 +65,7 @@ struct ufid_to_rte_flow_data { struct netdev_offload_dpdk_data { struct cmap ufid_to_rte_flow; + uint64_t rte_flow_counter; }; static int @@ -644,6 +645,12 @@ netdev_offload_dpdk_flow_create(struct netdev *netdev, flow = netdev_dpdk_rte_flow_create(netdev, attr, items, actions, error); if (flow) { + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + data->rte_flow_counter++; + if (!VLOG_DROP_DBG(&rl)) { dump_flow(&s, &s_extra, attr, items, actions); extra_str = ds_cstr(&s_extra); @@ -1524,6 +1531,12 @@ netdev_offload_dpdk_flow_destroy(struct ufid_to_rte_flow_data *rte_flow_data) ret = netdev_dpdk_rte_flow_destroy(netdev, rte_flow, &error); if (ret == 0) { + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + data->rte_flow_counter--; + ufid_to_rte_flow_disassociate(rte_flow_data); VLOG_DBG_RL(&rl, "%s: rte_flow 0x%"PRIxPTR " flow destroy %d ufid " UUID_FMT, @@ -1680,6 +1693,23 @@ netdev_offload_dpdk_flow_flush(struct netdev *netdev) return 0; } +static int +netdev_offload_dpdk_get_n_flows(struct netdev *netdev, + uint64_t *n_flows) +{ + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + if (!data) { + return -1; + } + + *n_flows = data->rte_flow_counter; + + return 0; +} + const struct netdev_flow_api netdev_offload_dpdk = { .type = "dpdk_flow_api", .flow_put = netdev_offload_dpdk_flow_put, @@ -1688,4 +1718,5 @@ const struct netdev_flow_api netdev_offload_dpdk = { .uninit_flow_api = netdev_offload_dpdk_uninit_flow_api, .flow_get = netdev_offload_dpdk_flow_get, .flow_flush = netdev_offload_dpdk_flow_flush, + .flow_get_n_flows = netdev_offload_dpdk_get_n_flows, }; From patchwork Wed Jun 9 13:09:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489863 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=ci8/hLEd; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=wVUREbZS; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SBt3yrZz9sVt for ; Wed, 9 Jun 2021 23:10:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 8FD1E60A5C; Wed, 9 Jun 2021 13:10:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SfD9YeVb53t9; Wed, 9 Jun 2021 13:10:07 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 7830E60A93; Wed, 9 Jun 2021 13:10:04 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4FE2AC0028; Wed, 9 Jun 2021 13:10:03 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 91A28C0025 for ; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 63EF283CC4 for ; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="ci8/hLEd"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="wVUREbZS" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VbHiXOipNOwk for ; Wed, 9 Jun 2021 13:09:57 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp1.osuosl.org (Postfix) with ESMTPS id AC22A83C91 for ; Wed, 9 Jun 2021 13:09:56 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id E84412664; Wed, 9 Jun 2021 09:09:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 09 Jun 2021 09:09:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=vq3jfoq87n6pi 0/fwEiv5RvQsukWEP85N0MeJ1GNQ1o=; b=ci8/hLEdxIYqEoqPJB7Wfw+y6lfxc U/1bciWhg1Ny92q9j3DdOp70WXRsd2o+EdY4l3Rvi27KI2l2MuqT0Yb02UucarZn 3fwXysEUuLPaM8kyFbfLGZv/zNmSL4gV6vM4N+IggGlyY0HbHy9Lt8uQQL7vvbXc oAPK3/sXYqo0CWGff8LeNZhaPbc6gfrWa2XRPCygxFk0wAkO/O65ESgqQuBCK7f3 BcMWP0xX11/6mOh6Kx50Ar+UsMCpu/C7qfdtZm8/9ql7DwgPBV8XnSFtrwx2r7xc sN4eaH5tyXoKgRmnJNxjn8FlR/vAp2AobXqZHZAJdEfP4RlQ7cXoF6q+Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=vq3jfoq87n6pi0/fwEiv5RvQsukWEP85N0MeJ1GNQ1o=; b=wVUREbZS NW5jXVt1V/5xr3xJe/30R06dKq14xYVraWL9gz1taLYmsL4hk84YchCtQ+lhFHfo 71CS6RLd1ApCgjtKYHPxNOVl1TdKcFNVS/LTcSVLjwueUQNijWx8LbgVM2FPz2e8 HseUVpPZHMSUHadO33lkMHvrHqRgZaFDeZAcVxe2KDGWd+ozG2d4ZRQTDmg0F+uF JjN1kfUVxLrbT3kt9/jV61juXaRKiLCXW+P3PVSE8vSa0j3OkaC/sU4DiMeDcR5p ouB/AXJ2t8/7bpAYJBdWF2uMY20b4gcYsq4yZQnd6kIqFe8Upg3qvznM3dYqekaP 63RcXSG66HYw2g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgiedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:54 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:15 +0200 Message-Id: <02d1679c41101ad1b431e26ba86caca345e013b0.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 07/27] dpctl: Add function to read hardware offload statistics X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Expose a function to query datapath offload statistics. This function is separate from the current one in netdev-offload as it exposes more detailed statistics from the datapath, instead of only from the netdev-offload provider. Each datapath is meant to use the custom counters as it sees fit for its handling of hardware offloads. Call the new API from dpctl. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpctl.c | 36 ++++++++++++++++++++++++++++++++++++ lib/dpif-netdev.c | 1 + lib/dpif-netlink.c | 1 + lib/dpif-provider.h | 7 +++++++ lib/dpif.c | 8 ++++++++ lib/dpif.h | 9 +++++++++ 6 files changed, 62 insertions(+) diff --git a/lib/dpctl.c b/lib/dpctl.c index ef8ae7402..6ff73e2d9 100644 --- a/lib/dpctl.c +++ b/lib/dpctl.c @@ -1541,6 +1541,40 @@ dpctl_del_flows(int argc, const char *argv[], struct dpctl_params *dpctl_p) return error; } +static int +dpctl_offload_stats_show(int argc, const char *argv[], + struct dpctl_params *dpctl_p) +{ + struct netdev_custom_stats stats; + struct dpif *dpif; + int error; + size_t i; + + error = opt_dpif_open(argc, argv, dpctl_p, 2, &dpif); + if (error) { + return error; + } + + memset(&stats, 0, sizeof(stats)); + error = dpif_offload_stats_get(dpif, &stats); + if (error) { + dpctl_error(dpctl_p, error, "retrieving offload statistics"); + goto close_dpif; + } + + dpctl_print(dpctl_p, "HW Offload stats:\n"); + for (i = 0; i < stats.size; i++) { + dpctl_print(dpctl_p, " %s: %6" PRIu64 "\n", + stats.counters[i].name, stats.counters[i].value); + } + + netdev_free_custom_stats_counters(&stats); + +close_dpif: + dpif_close(dpif); + return error; +} + static int dpctl_help(int argc OVS_UNUSED, const char *argv[] OVS_UNUSED, struct dpctl_params *dpctl_p) @@ -2697,6 +2731,8 @@ static const struct dpctl_command all_commands[] = { { "add-flows", "[dp] file", 1, 2, dpctl_process_flows, DP_RW }, { "mod-flows", "[dp] file", 1, 2, dpctl_process_flows, DP_RW }, { "del-flows", "[dp] [file]", 0, 2, dpctl_del_flows, DP_RW }, + { "offload-stats-show", "[dp]", + 0, 1, dpctl_offload_stats_show, DP_RO }, { "dump-conntrack", "[-m] [-s] [dp] [zone=N]", 0, 4, dpctl_dump_conntrack, DP_RO }, { "flush-conntrack", "[dp] [zone=N] [ct-tuple]", 0, 3, diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index d2c480529..7710417a7 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -8483,6 +8483,7 @@ const struct dpif_class dpif_netdev_class = { dpif_netdev_flow_dump_thread_destroy, dpif_netdev_flow_dump_next, dpif_netdev_operate, + NULL, /* offload_stats_get */ NULL, /* recv_set */ NULL, /* handlers_set */ dpif_netdev_set_config, diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index 50520f8c0..9dd580f63 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -3972,6 +3972,7 @@ const struct dpif_class dpif_netlink_class = { dpif_netlink_flow_dump_thread_destroy, dpif_netlink_flow_dump_next, dpif_netlink_operate, + NULL, /* offload_stats_get */ dpif_netlink_recv_set, dpif_netlink_handlers_set, NULL, /* set_config */ diff --git a/lib/dpif-provider.h b/lib/dpif-provider.h index b817fceac..36dfa8e71 100644 --- a/lib/dpif-provider.h +++ b/lib/dpif-provider.h @@ -330,6 +330,13 @@ struct dpif_class { void (*operate)(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, enum dpif_offload_type offload_type); + /* Get hardware-offloads activity counters from a dataplane. + * Those counters are not offload statistics (which are accessible through + * netdev statistics), but a status of hardware offload management: + * how many offloads are currently waiting, inserted, etc. */ + int (*offload_stats_get)(struct dpif *dpif, + struct netdev_custom_stats *stats); + /* Enables or disables receiving packets with dpif_recv() for 'dpif'. * Turning packet receive off and then back on is allowed to change Netlink * PID assignments (see ->port_get_pid()). The client is responsible for diff --git a/lib/dpif.c b/lib/dpif.c index 26e8bfb7d..7d3e09d78 100644 --- a/lib/dpif.c +++ b/lib/dpif.c @@ -1427,6 +1427,14 @@ dpif_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, } } +int dpif_offload_stats_get(struct dpif *dpif, + struct netdev_custom_stats *stats) +{ + return (dpif->dpif_class->offload_stats_get + ? dpif->dpif_class->offload_stats_get(dpif, stats) + : EOPNOTSUPP); +} + /* Returns a string that represents 'type', for use in log messages. */ const char * dpif_upcall_type_to_string(enum dpif_upcall_type type) diff --git a/lib/dpif.h b/lib/dpif.h index f9728e673..8c18b3626 100644 --- a/lib/dpif.h +++ b/lib/dpif.h @@ -787,6 +787,15 @@ struct dpif_op { void dpif_operate(struct dpif *, struct dpif_op **ops, size_t n_ops, enum dpif_offload_type); + +/* Queries the datapath for hardware offloads stats. + * + * Statistics are written in 'stats' following the 'netdev_custom_stats' + * format. They are allocated on the heap and must be freed by the caller, + * using 'netdev_free_custom_stats_counters'. + */ +int dpif_offload_stats_get(struct dpif *dpif, + struct netdev_custom_stats *stats); /* Upcalls. */ From patchwork Wed Jun 9 13:09:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489865 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=1ulESCoh; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=irvptEI/; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SBy1gsGz9sT6 for ; Wed, 9 Jun 2021 23:10:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 93A3383D59; Wed, 9 Jun 2021 13:10:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4--xQrULoAHh; Wed, 9 Jun 2021 13:10:12 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id BDE8D83C79; Wed, 9 Jun 2021 13:10:05 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4D1E0C002E; Wed, 9 Jun 2021 13:10:04 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id DC6A5C0025 for ; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C4E614052D for ; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="1ulESCoh"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="irvptEI/" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QCC_S-ybp_02 for ; Wed, 9 Jun 2021 13:09:58 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1EA6D40538 for ; Wed, 9 Jun 2021 13:09:57 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 7CE412662; Wed, 9 Jun 2021 09:09:57 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 09 Jun 2021 09:09:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=ok25YBlhDQSja JLyfZqjAGYkDjEBJIh0u76Alu5JFhU=; b=1ulESCohc1F7NyZ1VMOunkSV7k+xo yuqBNSClaC7tqhqrYtxgH6T7iXlwiATqqVi7OPbbJoAlrMC4J8n2tyU2e4FvykL5 /BcAl36Yqe7pisDXzqBxYMcFPCG0PiHhr2xfxHbLYwWV2dT+y7ZcXV/fVYov849n OiuwQ5snWpApP4FR1V+16aC75j5ve/c/itbNrrbxLFiobCO8k8tiGdoJh3fQGAbw WwFno/Al318utKDtBfo9OuJOvrr+BqlwwIMZtYreB4mwilmsVkMs162z5SKTpOwb xmnwZrXUIFe+Cpk0C6zTqTTngTLR31OAqSu7gTJ7ZO0QRueOvbyuj1Izw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=ok25YBlhDQSjaJLyfZqjAGYkDjEBJIh0u76Alu5JFhU=; b=irvptEI/ 3eXabt+r98L2w+hpIvjlpaTxwTMBS1O7cWR/2KAfgbftDtuvdQV7uAvBFmo0TLEx NXrDrQ8DAunVOCt8njGKM33vEToWuuEm9msIvYX2WB6C+68aBe0bfx4LqVK4sKzc xBvgmdnPtBNf8c+Y89TpyJGSsSNTQz54PvyvpJtJf9R5jbgZDbE/LfQmH9kHo+gl QpKnFAdkBEC0DTywDnxCD2A3H+PSDDLs5VoTJeD1TB9ER3kjfgOHS/XnUZTCuYu9 Lj4rLQ9ooAiEtF2qmPbUkg5yPLlQo0EfkXeyPfSU9QmTRkybQtXR5JwTOraDFtzW 6wAJADOg5WUq0w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:56 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:16 +0200 Message-Id: <7f24d293f04a80ac99d4b83bd495deef18f7e2a2.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 08/27] dpif-netdev: Rename offload thread structure X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The offload management in userspace is done through a separate thread. The naming of the structure holding the objects used for synchronization with the dataplane is generic and nondescript. Clarify the object function by renaming it. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 52 +++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 7710417a7..b666bc405 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -424,7 +424,7 @@ enum { DP_NETDEV_FLOW_OFFLOAD_OP_DEL, }; -struct dp_flow_offload_item { +struct dp_offload_thread_item { struct dp_netdev_pmd_thread *pmd; struct dp_netdev_flow *flow; int op; @@ -435,15 +435,15 @@ struct dp_flow_offload_item { struct ovs_list node; }; -struct dp_flow_offload { +struct dp_offload_thread { struct ovs_mutex mutex; struct ovs_list list; pthread_cond_t cond; }; -static struct dp_flow_offload dp_flow_offload = { +static struct dp_offload_thread dp_offload_thread = { .mutex = OVS_MUTEX_INITIALIZER, - .list = OVS_LIST_INITIALIZER(&dp_flow_offload.list), + .list = OVS_LIST_INITIALIZER(&dp_offload_thread.list), }; static struct ovsthread_once offload_thread_once @@ -2599,12 +2599,12 @@ mark_to_flow_find(const struct dp_netdev_pmd_thread *pmd, return NULL; } -static struct dp_flow_offload_item * +static struct dp_offload_thread_item * dp_netdev_alloc_flow_offload(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow, int op) { - struct dp_flow_offload_item *offload; + struct dp_offload_thread_item *offload; offload = xzalloc(sizeof(*offload)); offload->pmd = pmd; @@ -2618,7 +2618,7 @@ dp_netdev_alloc_flow_offload(struct dp_netdev_pmd_thread *pmd, } static void -dp_netdev_free_flow_offload(struct dp_flow_offload_item *offload) +dp_netdev_free_flow_offload(struct dp_offload_thread_item *offload) { dp_netdev_pmd_unref(offload->pmd); dp_netdev_flow_unref(offload->flow); @@ -2628,16 +2628,16 @@ dp_netdev_free_flow_offload(struct dp_flow_offload_item *offload) } static void -dp_netdev_append_flow_offload(struct dp_flow_offload_item *offload) +dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) { - ovs_mutex_lock(&dp_flow_offload.mutex); - ovs_list_push_back(&dp_flow_offload.list, &offload->node); - xpthread_cond_signal(&dp_flow_offload.cond); - ovs_mutex_unlock(&dp_flow_offload.mutex); + ovs_mutex_lock(&dp_offload_thread.mutex); + ovs_list_push_back(&dp_offload_thread.list, &offload->node); + xpthread_cond_signal(&dp_offload_thread.cond); + ovs_mutex_unlock(&dp_offload_thread.mutex); } static int -dp_netdev_flow_offload_del(struct dp_flow_offload_item *offload) +dp_netdev_flow_offload_del(struct dp_offload_thread_item *offload) { return mark_to_flow_disassociate(offload->pmd, offload->flow); } @@ -2654,7 +2654,7 @@ dp_netdev_flow_offload_del(struct dp_flow_offload_item *offload) * valid, thus only item 2 needed. */ static int -dp_netdev_flow_offload_put(struct dp_flow_offload_item *offload) +dp_netdev_flow_offload_put(struct dp_offload_thread_item *offload) { struct dp_netdev_pmd_thread *pmd = offload->pmd; struct dp_netdev_flow *flow = offload->flow; @@ -2734,22 +2734,22 @@ err_free: static void * dp_netdev_flow_offload_main(void *data OVS_UNUSED) { - struct dp_flow_offload_item *offload; + struct dp_offload_thread_item *offload; struct ovs_list *list; const char *op; int ret; for (;;) { - ovs_mutex_lock(&dp_flow_offload.mutex); - if (ovs_list_is_empty(&dp_flow_offload.list)) { + ovs_mutex_lock(&dp_offload_thread.mutex); + if (ovs_list_is_empty(&dp_offload_thread.list)) { ovsrcu_quiesce_start(); - ovs_mutex_cond_wait(&dp_flow_offload.cond, - &dp_flow_offload.mutex); + ovs_mutex_cond_wait(&dp_offload_thread.cond, + &dp_offload_thread.mutex); ovsrcu_quiesce_end(); } - list = ovs_list_pop_front(&dp_flow_offload.list); - offload = CONTAINER_OF(list, struct dp_flow_offload_item, node); - ovs_mutex_unlock(&dp_flow_offload.mutex); + list = ovs_list_pop_front(&dp_offload_thread.list); + offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); + ovs_mutex_unlock(&dp_offload_thread.mutex); switch (offload->op) { case DP_NETDEV_FLOW_OFFLOAD_OP_ADD: @@ -2782,10 +2782,10 @@ static void queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow) { - struct dp_flow_offload_item *offload; + struct dp_offload_thread_item *offload; if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_flow_offload.cond, NULL); + xpthread_cond_init(&dp_offload_thread.cond, NULL); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -2800,7 +2800,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow, struct match *match, const struct nlattr *actions, size_t actions_len) { - struct dp_flow_offload_item *offload; + struct dp_offload_thread_item *offload; int op; if (!netdev_is_flow_api_enabled()) { @@ -2808,7 +2808,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, } if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_flow_offload.cond, NULL); + xpthread_cond_init(&dp_offload_thread.cond, NULL); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } From patchwork Wed Jun 9 13:09:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489866 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=RARor6ap; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=aY5+FclE; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SC14Hvxz9sW8 for ; Wed, 9 Jun 2021 23:10:21 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id B5F9283D01; Wed, 9 Jun 2021 13:10:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gXrK4rKwuiSS; Wed, 9 Jun 2021 13:10:16 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id BB6E483CBF; Wed, 9 Jun 2021 13:10:07 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E3133C002A; Wed, 9 Jun 2021 13:10:05 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8EC11C002A for ; Wed, 9 Jun 2021 13:10:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7D5DF40599 for ; Wed, 9 Jun 2021 13:10:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="RARor6ap"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="aY5+FclE" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qc4rAEOnB7WT for ; Wed, 9 Jun 2021 13:09:59 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id B2CDE40547 for ; Wed, 9 Jun 2021 13:09:59 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 11362267A; Wed, 9 Jun 2021 09:09:59 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:09:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=89/Jzw1TTwXDg OKODSL9/dchLO9iMLblH8hh5p+BEOM=; b=RARor6ap5xRzhalQqF4x5+N2EaaKG MMsEkdVb/EaQmXebCdC7KQDrXIPOV/LwUnmRJKTD/ER99qSTy2OB6yzUp29+RBBa XgRviVpg686b08y78yoyq5CVi54oli835KiKX0a/6XNgbSNzBdkgnKPJXMAa56ax 0IMp36JIRGFLXX2J6zHn1xNnLKSARE/Woh5HlwhNzxxvZqQPsFgzUrOTjjwMfBIA 97tYGkMcEgNCU+8UaauJvY0W01TzTqf7C6KL5yRTLGQraovcMGD4UInpmcmG15kS 7hHiT+sQQIUp3zLeCvN32DoMC1pORRQ/VXwoMSqho17Wq5fUv87kT+ilQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=89/Jzw1TTwXDgOKODSL9/dchLO9iMLblH8hh5p+BEOM=; b=aY5+FclE v01C1nOgUQySaU5M7VcXZljctIYLWr47YbI4MlCAd/OzNOD/7cw3yo22386hnABV kYqvYjKRHkQuU0GTyx9OlfGn2u7hMiHuYD3JMCaQBqSYLGnrlDispKjmXh5XkUs6 6tl+js8JL3tK1/6oqOK8NLryA4beEqBWzAA8toplwh3gPU+BalCBEUURpxjSDNzZ 5sdBVHy4RA50+3lJ76VT8Kvk+GTKuiwQdR/ezlfLlKNgygkXipVzAHyifDAogPZ5 Of3Rqwc72iGazw5bzZzKFH4s5H5ZVVARl43w6oKLvYvbwgubLDj4b6IlWoQIsrqQ mjnLNuq998LbpA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epleegffdttdeftdevhfeikeegveeugeehleeltdduieegvddvveelgeeugeekhffgnecu ffhomhgrihhnpegrphgrtghhvgdrohhrghdpfihikhhiphgvughirgdrohhrghenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehgrhhivhgvsehu vdehiedrnhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:57 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:17 +0200 Message-Id: <64ffd431f40bc9c563c3d92652bb473644fb5cf6.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 09/27] mov-avg: Add a moving average helper structure X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a new module offering a helper to compute the Cumulative Moving Average (CMA) and the Exponential Moving Average (EMA) of a series of values. Use the new helpers to add latency metrics in dpif-netdev. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/automake.mk | 1 + lib/mov-avg.h | 171 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 172 insertions(+) create mode 100644 lib/mov-avg.h diff --git a/lib/automake.mk b/lib/automake.mk index 39901bd6d..111179736 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -166,6 +166,7 @@ lib_libopenvswitch_la_SOURCES = \ lib/memory.c \ lib/memory.h \ lib/meta-flow.c \ + lib/mov-avg.h \ lib/multipath.c \ lib/multipath.h \ lib/namemap.c \ diff --git a/lib/mov-avg.h b/lib/mov-avg.h new file mode 100644 index 000000000..4a7e62c18 --- /dev/null +++ b/lib/mov-avg.h @@ -0,0 +1,171 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef _MOV_AVG_H +#define _MOV_AVG_H 1 + +#include + +/* Moving average helpers. */ + +/* Cumulative Moving Average. + * + * Computes the arithmetic mean over a whole series of value. + * Online equivalent of sum(V) / len(V). + * + * As all values have equal weight, this average will + * be slow to show recent changes in the series. + * + */ + +struct mov_avg_cma { + unsigned long long int count; + double mean; + double sum_dsquared; +}; + +#define MOV_AVG_CMA_INITIALIZER \ + { .count = 0, .mean = .0, .sum_dsquared = .0 } + +static inline void +mov_avg_cma_init(struct mov_avg_cma *cma) +{ + *cma = (struct mov_avg_cma) MOV_AVG_CMA_INITIALIZER; +} + +static inline void +mov_avg_cma_update(struct mov_avg_cma *cma, double new_val) +{ + double new_mean; + + cma->count++; + new_mean = cma->mean + (new_val - cma->mean) / cma->count; + + cma->sum_dsquared += (new_val - new_mean) * (new_val - cma->mean); + cma->mean = new_mean; +} + +static inline double +mov_avg_cma(struct mov_avg_cma *cma) +{ + return cma->mean; +} + +static inline double +mov_avg_cma_std_dev(struct mov_avg_cma *cma) +{ + double variance = 0.0; + + if (cma->count > 1) { + variance = cma->sum_dsquared / (cma->count - 1); + } + + return sqrt(variance); +} + +/* Exponential Moving Average. + * + * Each value in the series has an exponentially decreasing weight, + * the older they get the less weight they have. + * + * The smoothing factor 'alpha' must be within 0 < alpha < 1. + * The closer this factor to zero, the more equal the weight between + * recent and older values. As it approaches one, the more recent values + * will have more weight. + * + * The EMA can be thought of as an estimator for the next value when measures + * are dependent. In this case, it can make sense to consider the mean square + * error of the prediction. An 'alpha' minimizing this error would be the + * better choice to improve the estimation. + * + * A common way to choose 'alpha' is to use the following formula: + * + * a = 2 / (N + 1) + * + * With this 'alpha', the EMA will have the same 'center of mass' as an + * equivalent N-values Simple Moving Average. + * + * When using this factor, the N last values of the EMA will have a sum weight + * converging toward 0.8647, meaning that those values will account for 86% of + * the average[1]. + * + * [1] https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average + */ + +struct mov_avg_ema { + double alpha; /* 'Smoothing' factor. */ + double mean; + double variance; + bool initialized; +}; + +/* Choose alpha explicitly. */ +#define MOV_AVG_EMA_INITIALIZER_ALPHA(a) { \ + .initialized = false, \ + .alpha = (a), .variance = 0.0, .mean = 0.0 \ +} + +/* Choose alpha to consider 'N' past periods as 86% of the EMA. */ +#define MOV_AVG_EMA_INITIALIZER(n_elem) \ + MOV_AVG_EMA_INITIALIZER_ALPHA(2.0 / ((double)(n_elem) + 1.0)) + +static inline void +mov_avg_ema_init_alpha(struct mov_avg_ema *ema, + double alpha) +{ + *ema = (struct mov_avg_ema) MOV_AVG_EMA_INITIALIZER_ALPHA(alpha); +} + +static inline void +mov_avg_ema_init(struct mov_avg_ema *ema, + unsigned long long int n_elem) +{ + *ema = (struct mov_avg_ema) MOV_AVG_EMA_INITIALIZER(n_elem); +} + +static inline void +mov_avg_ema_update(struct mov_avg_ema *ema, double new_val) +{ + const double alpha = ema->alpha; + double alpha_diff; + double diff; + + if (!ema->initialized) { + ema->initialized = true; + ema->mean = new_val; + return; + } + + diff = new_val - ema->mean; + alpha_diff = alpha * diff; + + ema->variance = (1.0 - alpha) * (ema->variance + alpha_diff * diff); + ema->mean = ema->mean + alpha_diff; +} + +static inline double +mov_avg_ema(struct mov_avg_ema *ema) +{ + return ema->mean; +} + +static inline double +mov_avg_ema_std_dev(struct mov_avg_ema *ema) +{ + return sqrt(ema->variance); +} + +#endif /* _MOV_AVG_H */ From patchwork Wed Jun 9 13:09:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489867 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=LLzM58Ib; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=D8fjFZd3; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SC736nCz9sT6 for ; Wed, 9 Jun 2021 23:10:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B2A1560B01; Wed, 9 Jun 2021 13:10:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Io0d_HBeswpt; Wed, 9 Jun 2021 13:10:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 2680E60A4B; Wed, 9 Jun 2021 13:10:14 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CBD96C000D; Wed, 9 Jun 2021 13:10:13 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 09E7AC000B for ; Wed, 9 Jun 2021 13:10:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D6708405F6 for ; Wed, 9 Jun 2021 13:10:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="LLzM58Ib"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="D8fjFZd3" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GkSabXzdCv1h for ; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2BC7C40560 for ; Wed, 9 Jun 2021 13:10:01 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 9805D2664; Wed, 9 Jun 2021 09:10:00 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=VvFaWPqls6Qd5 RnxckPORK0/ZbeCpqf2hLip2IBzXUk=; b=LLzM58Ib9Aziyx4lF5V9sg4rh0k8R n1R+z7iR5SxhseT537KL2DJy558RHzcQ8i9KnZeGB70yZRQ7U5CI3E/kdX5iovkh bgCgR9+E3TfGC1+I3BzXcgcPAhlycvHzD+6A6mUN5Vb1JeNWsjOqcBhIKt4EUGWa G37pigtQW3mqFgroA5hEHP9N7FyIByjP2U67HWAqD5+lqosDzgkruOHRJTW6zLQI 5YKIkPwZ6OjjB5TdCv4ZXfXQdse526A6BAzaUd6fqRb+LFm7Duz2p5NlSBIJ140N Hhg/wmwHmrWO9tgqloMROIZIbK7D5glyzERkzgJ4Ez1Wf3rTEzV+ltcGA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=VvFaWPqls6Qd5RnxckPORK0/ZbeCpqf2hLip2IBzXUk=; b=D8fjFZd3 fXXEA0+EyFv+/gHT8wiCcjJh2bt1ASospMKBzIP//C0qX+2wwPaGsmAEeaNlKh8p +HqkiMtY40YmGYyqNIsRRZdoVv8YfY4C6QhuNpwONF29fKVfTf9jdePDE/1TW4DC ZgMJWH8Qx21BtkQd7B5MLu/ylOszIpve7x9gse5nsr1x7bEqe5eQT/GhDaNeO5LM H8JovItMkagGbmZ4GegGj4OyZdFnjcOU6i0Tfs3Quff3csqX6yOINOl53HjgVIlV EJ9CUVdCO+Ly85xchV4RNt79rbRdDg/EfVG6RmlpfCVlnqdR2f2jDKRoUapXBgPb yUnYLo13fVLV/w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:09:59 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:18 +0200 Message-Id: <49a53d7be3f11f399733b922a7549c81c281a664.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 10/27] dpif-netdev: Implement hardware offloads stats query X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In the netdev datapath, keep track of the enqueued offloads between the PMDs and the offload thread. Additionally, query each netdev for their hardware offload counters. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 89 insertions(+), 1 deletion(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index b666bc405..a20eeda4d 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -51,6 +51,7 @@ #include "hmapx.h" #include "id-pool.h" #include "ipf.h" +#include "mov-avg.h" #include "netdev.h" #include "netdev-offload.h" #include "netdev-provider.h" @@ -431,6 +432,7 @@ struct dp_offload_thread_item { struct match match; struct nlattr *actions; size_t actions_len; + long long int timestamp; struct ovs_list node; }; @@ -438,12 +440,18 @@ struct dp_offload_thread_item { struct dp_offload_thread { struct ovs_mutex mutex; struct ovs_list list; + uint64_t enqueued_item; + struct mov_avg_cma cma; + struct mov_avg_ema ema; pthread_cond_t cond; }; static struct dp_offload_thread dp_offload_thread = { .mutex = OVS_MUTEX_INITIALIZER, .list = OVS_LIST_INITIALIZER(&dp_offload_thread.list), + .enqueued_item = 0, + .cma = MOV_AVG_CMA_INITIALIZER, + .ema = MOV_AVG_EMA_INITIALIZER(100), }; static struct ovsthread_once offload_thread_once @@ -2632,6 +2640,7 @@ dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) { ovs_mutex_lock(&dp_offload_thread.mutex); ovs_list_push_back(&dp_offload_thread.list, &offload->node); + dp_offload_thread.enqueued_item++; xpthread_cond_signal(&dp_offload_thread.cond); ovs_mutex_unlock(&dp_offload_thread.mutex); } @@ -2736,6 +2745,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) { struct dp_offload_thread_item *offload; struct ovs_list *list; + long long int latency_us; const char *op; int ret; @@ -2748,6 +2758,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) ovsrcu_quiesce_end(); } list = ovs_list_pop_front(&dp_offload_thread.list); + dp_offload_thread.enqueued_item--; offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); ovs_mutex_unlock(&dp_offload_thread.mutex); @@ -2768,6 +2779,10 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) OVS_NOT_REACHED(); } + latency_us = time_usec() - offload->timestamp; + mov_avg_cma_update(&dp_offload_thread.cma, latency_us); + mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + VLOG_DBG("%s to %s netdev flow "UUID_FMT, ret == 0 ? "succeed" : "failed", op, UUID_ARGS((struct uuid *) &offload->flow->mega_ufid)); @@ -2792,6 +2807,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, offload = dp_netdev_alloc_flow_offload(pmd, flow, DP_NETDEV_FLOW_OFFLOAD_OP_DEL); + offload->timestamp = pmd->ctx.now; dp_netdev_append_flow_offload(offload); } @@ -2824,6 +2840,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, memcpy(offload->actions, actions, actions_len); offload->actions_len = actions_len; + offload->timestamp = pmd->ctx.now; dp_netdev_append_flow_offload(offload); } @@ -4209,6 +4226,77 @@ dpif_netdev_operate(struct dpif *dpif, struct dpif_op **ops, size_t n_ops, } } +static int +dpif_netdev_offload_stats_get(struct dpif *dpif, + struct netdev_custom_stats *stats) +{ + enum { + DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED, + DP_NETDEV_HW_OFFLOADS_STATS_INSERTED, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN, + DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV, + }; + const char *names[] = { + [DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED] = + " Enqueued offloads", + [DP_NETDEV_HW_OFFLOADS_STATS_INSERTED] = + " Inserted offloads", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN] = + " Cumulative Average latency (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV] = + " Cumulative Latency stddev (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN] = + " Exponential Average latency (us)", + [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV] = + " Exponential Latency stddev (us)", + }; + struct dp_netdev *dp = get_dp_netdev(dpif); + struct dp_netdev_port *port; + uint64_t nb_offloads; + size_t i; + + if (!netdev_is_flow_api_enabled()) { + return EINVAL; + } + + stats->size = ARRAY_SIZE(names); + stats->counters = xcalloc(stats->size, sizeof *stats->counters); + + nb_offloads = 0; + + ovs_mutex_lock(&dp->port_mutex); + HMAP_FOR_EACH (port, node, &dp->ports) { + uint64_t port_nb_offloads = 0; + + /* Do not abort on read error from a port, just report 0. */ + if (!netdev_flow_get_n_flows(port->netdev, &port_nb_offloads)) { + nb_offloads += port_nb_offloads; + } + } + ovs_mutex_unlock(&dp->port_mutex); + + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value = + dp_offload_thread.enqueued_item; + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED].value = nb_offloads; + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].value = + mov_avg_cma(&dp_offload_thread.cma); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV].value = + mov_avg_cma_std_dev(&dp_offload_thread.cma); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN].value = + mov_avg_ema(&dp_offload_thread.ema); + stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV].value = + mov_avg_ema_std_dev(&dp_offload_thread.ema); + + for (i = 0; i < ARRAY_SIZE(names); i++) { + snprintf(stats->counters[i].name, sizeof(stats->counters[i].name), + "%s", names[i]); + } + + return 0; +} + /* Enable or Disable PMD auto load balancing. */ static void set_pmd_auto_lb(struct dp_netdev *dp, bool always_log) @@ -8483,7 +8571,7 @@ const struct dpif_class dpif_netdev_class = { dpif_netdev_flow_dump_thread_destroy, dpif_netdev_flow_dump_next, dpif_netdev_operate, - NULL, /* offload_stats_get */ + dpif_netdev_offload_stats_get, NULL, /* recv_set */ NULL, /* handlers_set */ dpif_netdev_set_config, From patchwork Wed Jun 9 13:09:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489868 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=fSwJildz; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=RMnsxOb3; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SCB6FDSz9sW8 for ; Wed, 9 Jun 2021 23:10:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 5572B4059A; Wed, 9 Jun 2021 13:10:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RybYVkQhM2xj; Wed, 9 Jun 2021 13:10:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 9D68B4058B; Wed, 9 Jun 2021 13:10:17 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 44F9AC0029; Wed, 9 Jun 2021 13:10:17 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0A4D9C002B for ; Wed, 9 Jun 2021 13:10:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 7FA4783CD3 for ; Wed, 9 Jun 2021 13:10:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="fSwJildz"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="RMnsxOb3" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EJcI4Ncl86jx for ; Wed, 9 Jun 2021 13:10:08 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp1.osuosl.org (Postfix) with ESMTPS id DE60E83CCF for ; Wed, 9 Jun 2021 13:10:02 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 445DF2657; Wed, 9 Jun 2021 09:10:02 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 09 Jun 2021 09:10:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=Yvl4r9kQh64HU ASVvgT1bAiJYkfZycRowrVksd8jWC8=; b=fSwJildzxBAYjxoOFgqyp/2jAP/s4 WsR6ql89v9ym2Tcfu+UIb2WR4e4/eslUYRzGNzdgYDsMzRlpPwPOO3YBY6Lc1vHB N2VSHGm2b5xtSlG1DcgTe3meD5kRyy58qI2WG/p8MnX1KyNNxUyjGdsuPc8+VlZl i+Heu1tSYI01Ijo2brIXVtRJoIfVI/gWu/H7OVGf/2BI3syMBWRADn1+Xqm7PiK3 nhTUOFafoET7X8Y8gLrsIq1YUOlX1R79AmU/9UiJzm+ZE365XDXRFu9HWI/rWfYg ND0bj+pVl1TegMyWnh8RXmtSFrajIK2G26oz26okp0/rmwHr2b8zN6dqA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=Yvl4r9kQh64HUASVvgT1bAiJYkfZycRowrVksd8jWC8=; b=RMnsxOb3 VggMq/MVoOiEsxe1zjbaSkhbRyphVri1x/U6CJ16zKvgVPAvJGw+qFwaL6zgTqsD R1JlAgNMUgp1RH2odVXtrZAMMfuFxr09rWvStTTLSUl6LSh4Nf7VOiux/jHzDt3A N3/MAQ5LeVTRgLqvRr6RDz7lRDo3Supa3uZAV1nQTxUYiRHllrED4EOv008Rhurm 13YUyPdYS6nMW3x79UJ/FK1kb/1HM/+wK2OsLEIGoQrXM9zRb52BsvIRmUf5jjuW BCjkjWtqMmOPYyc99Q9o7yu7FNFVx1Gs23Wx59i5dGZTM6UID1w64iQKi4hQRiN6 AzTRw8woNlG1mQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:01 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:19 +0200 Message-Id: <3ab29aa015e2c114a826a104d3ac9d6af5dd5b3e.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein Subject: [ovs-dev] [PATCH v4 11/27] ovs-atomic: Expose atomic exchange operation X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The atomic exchange operation is a useful primitive that should be available as well. Most compiler already expose or offer a way to use it, but a single symbol needs to be defined. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/ovs-atomic-c++.h | 3 +++ lib/ovs-atomic-clang.h | 5 +++++ lib/ovs-atomic-gcc4+.h | 5 +++++ lib/ovs-atomic-gcc4.7+.h | 5 +++++ lib/ovs-atomic-i586.h | 5 +++++ lib/ovs-atomic-locked.h | 9 +++++++++ lib/ovs-atomic-msvc.h | 22 ++++++++++++++++++++++ lib/ovs-atomic-pthreads.h | 5 +++++ lib/ovs-atomic-x86_64.h | 5 +++++ lib/ovs-atomic.h | 8 +++++++- 10 files changed, 71 insertions(+), 1 deletion(-) diff --git a/lib/ovs-atomic-c++.h b/lib/ovs-atomic-c++.h index d47b8dd39..8605fa9d3 100644 --- a/lib/ovs-atomic-c++.h +++ b/lib/ovs-atomic-c++.h @@ -29,6 +29,9 @@ using std::atomic_compare_exchange_strong_explicit; using std::atomic_compare_exchange_weak; using std::atomic_compare_exchange_weak_explicit; +using std::atomic_exchange; +using std::atomic_exchange_explicit; + #define atomic_read(SRC, DST) \ atomic_read_explicit(SRC, DST, memory_order_seq_cst) #define atomic_read_explicit(SRC, DST, ORDER) \ diff --git a/lib/ovs-atomic-clang.h b/lib/ovs-atomic-clang.h index 34cc2faa7..cdf02a512 100644 --- a/lib/ovs-atomic-clang.h +++ b/lib/ovs-atomic-clang.h @@ -67,6 +67,11 @@ typedef enum { #define atomic_compare_exchange_weak_explicit(DST, EXP, SRC, ORD1, ORD2) \ __c11_atomic_compare_exchange_weak(DST, EXP, SRC, ORD1, ORD2) +#define atomic_exchange(RMW, ARG) \ + atomic_exchange_explicit(RMW, ARG, memory_order_seq_cst) +#define atomic_exchange_explicit(RMW, ARG, ORDER) \ + __c11_atomic_exchange(RMW, ARG, ORDER) + #define atomic_add(RMW, ARG, ORIG) \ atomic_add_explicit(RMW, ARG, ORIG, memory_order_seq_cst) #define atomic_sub(RMW, ARG, ORIG) \ diff --git a/lib/ovs-atomic-gcc4+.h b/lib/ovs-atomic-gcc4+.h index 25bcf20a0..f9accde1a 100644 --- a/lib/ovs-atomic-gcc4+.h +++ b/lib/ovs-atomic-gcc4+.h @@ -128,6 +128,11 @@ atomic_signal_fence(memory_order order) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +#define atomic_exchange_explicit(DST, SRC, ORDER) \ + __sync_lock_test_and_set(DST, SRC) +#define atomic_exchange(DST, SRC) \ + atomic_exchange_explicit(DST, SRC, memory_order_seq_cst) + #define atomic_op__(RMW, OP, ARG, ORIG) \ ({ \ typeof(RMW) rmw__ = (RMW); \ diff --git a/lib/ovs-atomic-gcc4.7+.h b/lib/ovs-atomic-gcc4.7+.h index 4c197ebe0..846e05775 100644 --- a/lib/ovs-atomic-gcc4.7+.h +++ b/lib/ovs-atomic-gcc4.7+.h @@ -61,6 +61,11 @@ typedef enum { #define atomic_compare_exchange_weak_explicit(DST, EXP, SRC, ORD1, ORD2) \ __atomic_compare_exchange_n(DST, EXP, SRC, true, ORD1, ORD2) +#define atomic_exchange_explicit(DST, SRC, ORDER) \ + __atomic_exchange_n(DST, SRC, ORDER) +#define atomic_exchange(DST, SRC) \ + atomic_exchange_explicit(DST, SRC, memory_order_seq_cst) + #define atomic_add(RMW, OPERAND, ORIG) \ atomic_add_explicit(RMW, OPERAND, ORIG, memory_order_seq_cst) #define atomic_sub(RMW, OPERAND, ORIG) \ diff --git a/lib/ovs-atomic-i586.h b/lib/ovs-atomic-i586.h index 9a385ce84..35a0959ff 100644 --- a/lib/ovs-atomic-i586.h +++ b/lib/ovs-atomic-i586.h @@ -400,6 +400,11 @@ atomic_signal_fence(memory_order order) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +#define atomic_exchange_explicit(RMW, ARG, ORDER) \ + atomic_exchange__(RMW, ARG, ORDER) +#define atomic_exchange(RMW, ARG) \ + atomic_exchange_explicit(RMW, ARG, memory_order_seq_cst) + #define atomic_add__(RMW, ARG, CLOB) \ asm volatile("lock; xadd %0,%1 ; " \ "# atomic_add__ " \ diff --git a/lib/ovs-atomic-locked.h b/lib/ovs-atomic-locked.h index f8f0ba2a5..bf38c4a43 100644 --- a/lib/ovs-atomic-locked.h +++ b/lib/ovs-atomic-locked.h @@ -31,6 +31,15 @@ void atomic_unlock__(void *); atomic_unlock__(DST), \ false))) +#define atomic_exchange_locked(DST, SRC) \ + ({ \ + atomic_lock__(DST); \ + typeof(*(DST)) __tmp = *(DST); \ + *(DST) = SRC; \ + atomic_unlock__(DST); \ + __tmp; \ + }) + #define atomic_op_locked_add += #define atomic_op_locked_sub -= #define atomic_op_locked_or |= diff --git a/lib/ovs-atomic-msvc.h b/lib/ovs-atomic-msvc.h index 9def887d3..ef8310269 100644 --- a/lib/ovs-atomic-msvc.h +++ b/lib/ovs-atomic-msvc.h @@ -345,6 +345,28 @@ atomic_signal_fence(memory_order order) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +/* While intrinsics offering different memory ordering + * are available in MSVC C compiler, they are not defined + * in the C++ compiler. Ignore for compatibility. + * + * Use nested ternary operators as the GNU extension ({}) + * is not available. + */ + +#define atomic_exchange_explicit(DST, SRC, ORDER) \ + ((sizeof *(DST) == 1) ? \ + _InterlockedExchange8((char volatile *) DST, SRC) \ + : (sizeof *(DST) == 2) ? \ + _InterlockedExchange16((short volatile *) DST, SRC) \ + : (sizeof *(DST) == 4) ? \ + _InterlockedExchange((long int volatile *) DST, SRC) \ + : (sizeof *(DST) == 8) ? \ + _InterlockedExchange64((__int64 volatile *) DST, SRC) \ + : (ovs_abort(), 0)) + +#define atomic_exchange(DST, SRC) \ + atomic_exchange_explicit(DST, SRC, memory_order_seq_cst) + /* MSVCs c++ compiler implements c11 atomics and looking through its * implementation (in xatomic.h), orders are ignored for x86 platform. * Do the same here. */ diff --git a/lib/ovs-atomic-pthreads.h b/lib/ovs-atomic-pthreads.h index 12234e79e..570a67fe4 100644 --- a/lib/ovs-atomic-pthreads.h +++ b/lib/ovs-atomic-pthreads.h @@ -77,6 +77,11 @@ atomic_signal_fence(memory_order order OVS_UNUSED) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +#define atomic_exchange(DST, SRC) \ + atomic_exchange_locked(DST, SRC) +#define atomic_exchange_explicit(DST, SRC, ORDER) \ + ((void) (ORDER), atomic_exchange(DST, SRC)) + #define atomic_add(RMW, ARG, ORIG) atomic_op_locked(RMW, add, ARG, ORIG) #define atomic_sub(RMW, ARG, ORIG) atomic_op_locked(RMW, sub, ARG, ORIG) #define atomic_or( RMW, ARG, ORIG) atomic_op_locked(RMW, or, ARG, ORIG) diff --git a/lib/ovs-atomic-x86_64.h b/lib/ovs-atomic-x86_64.h index 1e7d42707..3bdaf2f08 100644 --- a/lib/ovs-atomic-x86_64.h +++ b/lib/ovs-atomic-x86_64.h @@ -274,6 +274,11 @@ atomic_signal_fence(memory_order order) #define atomic_compare_exchange_weak_explicit \ atomic_compare_exchange_strong_explicit +#define atomic_exchange_explicit(RMW, ARG, ORDER) \ + atomic_exchange__(RMW, ARG, ORDER) +#define atomic_exchange(RMW, ARG) \ + atomic_exchange_explicit(RMW, ARG, memory_order_seq_cst) + #define atomic_add__(RMW, ARG, CLOB) \ asm volatile("lock; xadd %0,%1 ; " \ "# atomic_add__ " \ diff --git a/lib/ovs-atomic.h b/lib/ovs-atomic.h index 11fa19268..8fdce0cf8 100644 --- a/lib/ovs-atomic.h +++ b/lib/ovs-atomic.h @@ -210,7 +210,7 @@ * In this section, A is an atomic type and C is the corresponding non-atomic * type. * - * The "store" and "compare_exchange" primitives match C11: + * The "store", "exchange", and "compare_exchange" primitives match C11: * * void atomic_store(A *object, C value); * void atomic_store_explicit(A *object, C value, memory_order); @@ -244,6 +244,12 @@ * efficiently, so it should be used if the application will need to * loop anyway. * + * C atomic_exchange(A *object, C desired); + * C atomic_exchange_explicit(A *object, C desired, memory_order); + * + * Atomically stores 'desired' into '*object', returning the value + * previously held. + * * The following primitives differ from the C11 ones (and have different names) * because there does not appear to be a way to implement the standard * primitives in standard C: From patchwork Wed Jun 9 13:09:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489869 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=ObZHj/tx; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=iU8UtqZZ; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SCK2wPvz9sT6 for ; Wed, 9 Jun 2021 23:10:37 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 0EF6A41CC6; Wed, 9 Jun 2021 13:10:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OuV91SYosyFZ; Wed, 9 Jun 2021 13:10:30 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 62851414B5; Wed, 9 Jun 2021 13:10:19 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 21408C002B; Wed, 9 Jun 2021 13:10:19 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 02A20C0026 for ; Wed, 9 Jun 2021 13:10:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3DBFF60AD9 for ; Wed, 9 Jun 2021 13:10:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="ObZHj/tx"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="iU8UtqZZ" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RJ53ggm_QwqZ for ; Wed, 9 Jun 2021 13:10:05 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp3.osuosl.org (Postfix) with ESMTPS id 8591860A99 for ; Wed, 9 Jun 2021 13:10:04 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id DFD812673; Wed, 9 Jun 2021 09:10:03 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 09 Jun 2021 09:10:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=at82edyVtBmqV bPo31d34Zxp8rSrSHYHR4CtedYrKjo=; b=ObZHj/txwI1B7rlbG/V+2VoPs4Nt7 1SatYkd0qHOsVhDt+Hi53grVYqVBpIExE/PitSJrQ9EnlZ1+e0I9ozbm6F2MPs2C XjGhQTfzegAF/5Ev1x0NAFVO5517yKD4R/LBW0u1H7YFWPgT45J16it35A6oplzh XNmIzZ4hnufrBd4v39z8GqFYuoOvs+uINWlX/YNGDFnmKXBCBt+ovewoGKIW0lX/ pt+Dip25dz01JiheoLg+9bMo/0QjUghVGBn8jVtUg2pIhyZDv/w1SpeF/kjnwxFS 6irwo0Jogb2a3fa3VrQ2jjHTJOdgtMNMSmyQy+MRc30gLJKmnInHQ+lwQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=at82edyVtBmqVbPo31d34Zxp8rSrSHYHR4CtedYrKjo=; b=iU8UtqZZ JE9N+6cAadFiEbdw3Sg34VYiPGnkdfTDRasWQBpWt8yDChLLnqiSDNXAxmqDxvxz BCC50ebhMVt9PWLyPh5RQNv3/NXCmRbEE4gpGBhFw786NcMTMubalxMWIQUhhFrn QXgKwX9NdCDG/+99V45E8GghWyPUpRrdfqXt1gFi9M/XQahU5w9Z14B4dU4PyDdx u7AO8+LtuBIVz4LG4pnqQF8vb14Rp+eQ+zDekh5KbdwGmokA9EojwEFIUQuVNiM6 EoxZpk0rlg0rn4pIRZKKXKUsYdVn80PNzgNyMYjg5Ga6dMixk+t/+9DJ1+VslAhm RLu5yGGwfi5b3A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgiedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh eptedthefhudeugffgffelhffffedtkeefudefjedvgffhgfdttdfgueeuhfffgfejnecu ffhomhgrihhnpegrphgrtghhvgdrohhrghdpuddtvdegtghorhgvshdrnhgvthdprhhotg hhvghsthgvrhdrvgguuhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgr ihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:02 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:20 +0200 Message-Id: <250d65b4122e1a2188034297295366cfea81684a.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein Subject: [ovs-dev] [PATCH v4 12/27] mpsc-queue: Module for lock-free message passing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a lockless multi-producer/single-consumer (MPSC), linked-list based, intrusive, unbounded queue that does not require deferred memory management. The queue is designed to improve the specific MPSC setup. A benchmark accompanies the unit tests to measure the difference in this configuration. A single reader thread polls the queue while N writers enqueue elements as fast as possible. The mpsc-queue is compared against the regular ovs-list as well as the guarded list. The latter usually offers a slight improvement by batching the element removal, however the mpsc-queue is faster. The average is of each producer threads time: $ ./tests/ovstest test-mpsc-queue benchmark 3000000 1 Benchmarking n=3000000 on 1 + 1 threads. type\thread: Reader 1 Avg mpsc-queue: 167 167 167 ms list(spin): 89 80 80 ms list(mutex): 745 745 745 ms guarded list: 788 788 788 ms $ ./tests/ovstest test-mpsc-queue benchmark 3000000 2 Benchmarking n=3000000 on 1 + 2 threads. type\thread: Reader 1 2 Avg mpsc-queue: 98 97 94 95 ms list(spin): 185 171 173 172 ms list(mutex): 203 199 203 201 ms guarded list: 269 269 188 228 ms $ ./tests/ovstest test-mpsc-queue benchmark 3000000 3 Benchmarking n=3000000 on 1 + 3 threads. type\thread: Reader 1 2 3 Avg mpsc-queue: 76 76 65 76 72 ms list(spin): 246 110 240 238 196 ms list(mutex): 542 541 541 539 540 ms guarded list: 535 535 507 511 517 ms $ ./tests/ovstest test-mpsc-queue benchmark 3000000 4 Benchmarking n=3000000 on 1 + 4 threads. type\thread: Reader 1 2 3 4 Avg mpsc-queue: 73 68 68 68 68 68 ms list(spin): 294 275 279 277 282 278 ms list(mutex): 346 309 287 345 302 310 ms guarded list: 378 319 334 378 351 345 ms Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/automake.mk | 2 + lib/mpsc-queue.c | 251 +++++++++++++ lib/mpsc-queue.h | 190 ++++++++++ tests/automake.mk | 1 + tests/library.at | 5 + tests/test-mpsc-queue.c | 772 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 1221 insertions(+) create mode 100644 lib/mpsc-queue.c create mode 100644 lib/mpsc-queue.h create mode 100644 tests/test-mpsc-queue.c diff --git a/lib/automake.mk b/lib/automake.mk index 111179736..b45801852 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -167,6 +167,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/memory.h \ lib/meta-flow.c \ lib/mov-avg.h \ + lib/mpsc-queue.c \ + lib/mpsc-queue.h \ lib/multipath.c \ lib/multipath.h \ lib/namemap.c \ diff --git a/lib/mpsc-queue.c b/lib/mpsc-queue.c new file mode 100644 index 000000000..ee762e1dc --- /dev/null +++ b/lib/mpsc-queue.c @@ -0,0 +1,251 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "ovs-atomic.h" + +#include "mpsc-queue.h" + +/* Multi-producer, single-consumer queue + * ===================================== + * + * This an implementation of the MPSC queue described by Dmitri Vyukov [1]. + * + * One atomic exchange operation is done per insertion. Removal in most cases + * will not require atomic operation and will use one atomic exchange to close + * the queue chain. + * + * Insertion + * ========= + * + * The queue is implemented using a linked-list. Insertion is done at the + * back of the queue, by swapping the current end with the new node atomically, + * then pointing the previous end toward the new node. To follow Vyukov + * nomenclature, the end-node of the chain is called head. A producer will + * only manipulate the head. + * + * The head swap is atomic, however the link from the previous head to the new + * one is done in a separate operation. This means that the chain is + * momentarily broken, when the previous head still points to NULL and the + * current head has been inserted. + * + * Considering a series of insertions, the queue state will remain consistent + * and the insertions order is compatible with their precedence, thus the + * queue is serializable. However, because an insertion consists in two + * separate memory transactions, it is not linearizable. + * + * Removal + * ======= + * + * The consumer must deal with the queue inconsistency. It will manipulate + * the tail of the queue and move it along the latest consumed elements. + * When an end of the chain of elements is found (the next pointer is NULL), + * the tail is compared with the head. + * + * If both points to different addresses, then the queue is in an inconsistent + * state: the tail cannot move forward as the next is NULL, but the head is not + * the last element in the chain: this can only happen if the chain is broken. + * + * In this case, the consumer must wait for the producer to finish writing the + * next pointer of its current tail: 'MPSC_QUEUE_RETRY' is returned. + * + * Removal is thus in most cases (when there are elements in the queue) + * accomplished without using atomics, until the last element of the queue. + * There, the head is atomically loaded. If the queue is in a consistent state, + * the head is moved back to the queue stub by inserting the stub in the queue: + * ending the queue is the same as an insertion, which is one atomic XCHG. + * + * Forward guarantees + * ================== + * + * Insertion and peeking are wait-free: they will execute in a known bounded + * number of instructions, regardless of the state of the queue. + * + * However, while removal consists in peeking and a constant write to + * update the tail, it can repeatedly fail until the queue become consistent. + * It is thus dependent on other threads progressing. This means that the + * queue forward progress is obstruction-free only. It has a potential for + * livelocking. + * + * The chain will remain broken as long as a producer is not finished writing + * its next pointer. If a producer is cancelled for example, the queue could + * remain broken for any future readings. This queue should either be used + * with cooperative threads or insertion must only be done outside cancellable + * sections. + * + * Performances + * ============ + * + * In benchmarks this structure was better than alternatives such as: + * + * * A reversed Treiber stack [2], using 1 CAS per operations + * and requiring reversal of the node list on removal. + * + * * Michael-Scott lock-free queue [3], using 2 CAS per operations. + * + * While it is not linearizable, this queue is well-suited for message passing. + * If a proper hardware XCHG operation is used, it scales better than + * CAS-based implementations. + * + * References + * ========== + * + * [1]: http://www.1024cores.net/home/lock-free-algorithms/queues/intrusive-mpsc-node-based-queue + * + * [2]: R. K. Treiber. Systems programming: Coping with parallelism. + * Technical Report RJ 5118, IBM Almaden Research Center, April 1986. + * + * [3]: M. M. Michael, Simple, Fast, and Practical Non-Blocking and + * Blocking Concurrent Queue Algorithms + * [3]: https://www.cs.rochester.edu/research/synchronization/pseudocode/queues.html + * + */ + +void +mpsc_queue_init(struct mpsc_queue *queue) +{ + atomic_store_relaxed(&queue->head, &queue->stub); + atomic_store_relaxed(&queue->tail, &queue->stub); + atomic_store_relaxed(&queue->stub.next, NULL); + + ovs_mutex_init(&queue->read_lock); +} + +void +mpsc_queue_destroy(struct mpsc_queue *queue) + OVS_EXCLUDED(queue->read_lock) +{ + ovs_mutex_destroy(&queue->read_lock); +} + +enum mpsc_queue_poll_result +mpsc_queue_poll(struct mpsc_queue *queue, struct mpsc_queue_node **node) + OVS_REQUIRES(queue->read_lock) +{ + struct mpsc_queue_node *tail; + struct mpsc_queue_node *next; + struct mpsc_queue_node *head; + + atomic_read_relaxed(&queue->tail, &tail); + atomic_read_explicit(&tail->next, &next, memory_order_acquire); + + if (tail == &queue->stub) { + if (next == NULL) { + return MPSC_QUEUE_EMPTY; + } + + atomic_store_relaxed(&queue->tail, next); + tail = next; + atomic_read_explicit(&tail->next, &next, memory_order_acquire); + } + + if (next != NULL) { + atomic_store_relaxed(&queue->tail, next); + *node = tail; + return MPSC_QUEUE_ITEM; + } + + atomic_read_explicit(&queue->head, &head, memory_order_acquire); + if (tail != head) { + return MPSC_QUEUE_RETRY; + } + + mpsc_queue_insert(queue, &queue->stub); + + atomic_read_explicit(&tail->next, &next, memory_order_acquire); + if (next != NULL) { + atomic_store_relaxed(&queue->tail, next); + *node = tail; + return MPSC_QUEUE_ITEM; + } + + return MPSC_QUEUE_EMPTY; +} + +struct mpsc_queue_node * +mpsc_queue_pop(struct mpsc_queue *queue) + OVS_REQUIRES(queue->read_lock) +{ + enum mpsc_queue_poll_result result; + struct mpsc_queue_node *node; + + do { + result = mpsc_queue_poll(queue, &node); + if (result == MPSC_QUEUE_EMPTY) { + return NULL; + } + } while (result == MPSC_QUEUE_RETRY); + + return node; +} + +void +mpsc_queue_push_front(struct mpsc_queue *queue, struct mpsc_queue_node *node) + OVS_REQUIRES(queue->read_lock) +{ + struct mpsc_queue_node *tail; + + atomic_read_relaxed(&queue->tail, &tail); + atomic_store_relaxed(&node->next, tail); + atomic_store_relaxed(&queue->tail, node); +} + +struct mpsc_queue_node * +mpsc_queue_tail(struct mpsc_queue *queue) + OVS_REQUIRES(queue->read_lock) +{ + struct mpsc_queue_node *tail; + struct mpsc_queue_node *next; + + atomic_read_relaxed(&queue->tail, &tail); + atomic_read_explicit(&tail->next, &next, memory_order_acquire); + + if (tail == &queue->stub) { + if (next == NULL) { + return NULL; + } + + atomic_store_relaxed(&queue->tail, next); + tail = next; + } + + return tail; +} + +/* Get the next element of a node. */ +struct mpsc_queue_node *mpsc_queue_next(struct mpsc_queue *queue, + struct mpsc_queue_node *prev) + OVS_REQUIRES(queue->read_lock) +{ + struct mpsc_queue_node *next; + + atomic_read_explicit(&prev->next, &next, memory_order_acquire); + if (next == &queue->stub) { + atomic_read_explicit(&next->next, &next, memory_order_acquire); + } + return next; +} + +void +mpsc_queue_insert(struct mpsc_queue *queue, struct mpsc_queue_node *node) +{ + struct mpsc_queue_node *prev; + + atomic_store_relaxed(&node->next, NULL); + prev = atomic_exchange_explicit(&queue->head, node, memory_order_acq_rel); + atomic_store_explicit(&prev->next, node, memory_order_release); +} diff --git a/lib/mpsc-queue.h b/lib/mpsc-queue.h new file mode 100644 index 000000000..3bb9e3bee --- /dev/null +++ b/lib/mpsc-queue.h @@ -0,0 +1,190 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef MPSC_QUEUE_H +#define MPSC_QUEUE_H 1 + +#include +#include +#include + +#include +#include + +#include "ovs-atomic.h" + +/* Multi-producer, single-consumer queue + * ===================================== + * + * This data structure is a lockless queue implementation with + * the following properties: + * + * * Multi-producer: multiple threads can write concurrently. + * Insertion in the queue is thread-safe, no inter-thread + * synchronization is necessary. + * + * * Single-consumer: only a single thread can safely remove + * nodes from the queue. The queue must be 'acquired' using + * 'mpsc_queue_acquire()' before removing nodes. + * + * * Unbounded: the queue is backed by a linked-list and is not + * limited in number of elements. + * + * * Intrusive: queue elements are allocated as part of larger + * objects. Objects are retrieved by offset manipulation. + * + * * per-producer FIFO: Elements in the queue are kept in the + * order their producer inserted them. The consumer retrieves + * them in in the same insertion order. When multiple + * producers insert at the same time, either will proceed. + * + * This queue is well-suited for message passing between threads, + * where any number of thread can insert a message and a single + * thread is meant to receive and process it. + * + * Thread-safety + * ============= + * + * The consumer thread must acquire the queue using 'mpsc_queue_acquire()'. + * Once the queue is protected against concurrent reads, the thread can call + * the consumer API: + * + * * mpsc_queue_poll() to peek and return the tail of the queue + * * mpsc_queue_pop() to remove the tail of the queue + * * mpsc_queue_tail() to read the current tail + * * mpsc_queue_push_front() to enqueue an element safely at the tail + * * MPSC_QUEUE_FOR_EACH() to iterate over the current elements, + * without removing them. + * * MPSC_QUEUE_FOR_EACH_POP() to iterate over the elements while + * removing them. + * + * When a thread is finished with reading the queue, it can release the + * reader lock using 'mpsc_queue_release()'. + * + * Producers can always insert elements in the queue, even if no consumer + * acquired the reader lock. No inter-producer synchronization is needed. + * + * The consumer thread is also allowed to insert elements while it holds the + * reader lock. + * + * Producer threads must never be cancelled while writing to the queue. + * This will block the consumer, that will then lose any subsequent elements + * in the queue. Producers should ideally be cooperatively managed or + * the queue insertion should be within non-cancellable sections. + * + * Queue state + * =========== + * + * When polling the queue, three states can be observed: 'empty', 'non-empty', + * and 'inconsistent'. Three polling results are defined, respectively: + * + * * MPSC_QUEUE_EMPTY: the queue is empty. + * * MPSC_QUEUE_ITEM: an item was available and has been removed. + * * MPSC_QUEUE_RETRY: the queue is inconsistent. + * + * If 'MPSC_QUEUE_RETRY' is returned, then a producer has not yet finished + * writing to the queue and the list of nodes is not coherent. The consumer + * can retry shortly to check if the producer has finished. + * + * This behavior is the reason the removal function is called + * 'mpsc_queue_poll()'. + * + */ + +struct mpsc_queue_node { + ATOMIC(struct mpsc_queue_node *) next; +}; + +struct mpsc_queue { + ATOMIC(struct mpsc_queue_node *) head; + ATOMIC(struct mpsc_queue_node *) tail; + struct mpsc_queue_node stub; + struct ovs_mutex read_lock; +}; + +#define MPSC_QUEUE_INITIALIZER(Q) { \ + .head = ATOMIC_VAR_INIT(&(Q)->stub), \ + .tail = ATOMIC_VAR_INIT(&(Q)->stub), \ + .stub = { .next = ATOMIC_VAR_INIT(NULL) }, \ + .read_lock = OVS_MUTEX_INITIALIZER, \ +} + +/* Consumer API. */ + +/* Initialize the queue. Not necessary is 'MPSC_QUEUE_INITIALIZER' was used. */ +void mpsc_queue_init(struct mpsc_queue *queue); +/* The reader lock must be released prior to destroying the queue. */ +void mpsc_queue_destroy(struct mpsc_queue *queue); + +/* Acquire and release the consumer lock. */ +#define mpsc_queue_acquire(q) do { \ + ovs_mutex_lock(&(q)->read_lock); \ + } while (0) +#define mpsc_queue_release(q) do { \ + ovs_mutex_unlock(&(q)->read_lock); \ + } while (0) + +enum mpsc_queue_poll_result { + /* Queue is empty. */ + MPSC_QUEUE_EMPTY, + /* Polling the queue returned an item. */ + MPSC_QUEUE_ITEM, + /* Data has been enqueued but one or more producer thread have not + * finished writing it. The queue is in an inconsistent state. + * Retrying shortly, if the producer threads are still active, will + * return the data. + */ + MPSC_QUEUE_RETRY, +}; + +/* Set 'node' to a removed item from the queue if 'MPSC_QUEUE_ITEM' is + * returned, otherwise 'node' is not set. + */ +enum mpsc_queue_poll_result mpsc_queue_poll(struct mpsc_queue *queue, + struct mpsc_queue_node **node) + OVS_REQUIRES(queue->read_lock); + +/* Pop an element if there is any in the queue. */ +struct mpsc_queue_node *mpsc_queue_pop(struct mpsc_queue *queue) + OVS_REQUIRES(queue->read_lock); + +/* Insert at the front of the queue. Only the consumer can do it. */ +void mpsc_queue_push_front(struct mpsc_queue *queue, + struct mpsc_queue_node *node) + OVS_REQUIRES(queue->read_lock); + +/* Get the current queue tail. */ +struct mpsc_queue_node *mpsc_queue_tail(struct mpsc_queue *queue) + OVS_REQUIRES(queue->read_lock); + +/* Get the next element of a node. */ +struct mpsc_queue_node *mpsc_queue_next(struct mpsc_queue *queue, + struct mpsc_queue_node *prev) + OVS_REQUIRES(queue->read_lock); + +#define MPSC_QUEUE_FOR_EACH(node, queue) \ + for (node = mpsc_queue_tail(queue); node != NULL; \ + node = mpsc_queue_next((queue), node)) + +#define MPSC_QUEUE_FOR_EACH_POP(node, queue) \ + for (node = mpsc_queue_pop(queue); node != NULL; \ + node = mpsc_queue_pop(queue)) + +/* Producer API. */ + +void mpsc_queue_insert(struct mpsc_queue *queue, struct mpsc_queue_node *node); + +#endif /* MPSC_QUEUE_H */ diff --git a/tests/automake.mk b/tests/automake.mk index a32abd41c..4588d5b49 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -466,6 +466,7 @@ tests_ovstest_SOURCES = \ tests/test-list.c \ tests/test-lockfile.c \ tests/test-multipath.c \ + tests/test-mpsc-queue.c \ tests/test-netflow.c \ tests/test-odp.c \ tests/test-ofpbuf.c \ diff --git a/tests/library.at b/tests/library.at index e572c22e3..0e47bc445 100644 --- a/tests/library.at +++ b/tests/library.at @@ -259,3 +259,8 @@ AT_SETUP([stopwatch module]) AT_CHECK([ovstest test-stopwatch], [0], [...... ], [ignore]) AT_CLEANUP + +AT_SETUP([mpsc-queue module]) +AT_CHECK([ovstest test-mpsc-queue check], [0], [.... +]) +AT_CLEANUP diff --git a/tests/test-mpsc-queue.c b/tests/test-mpsc-queue.c new file mode 100644 index 000000000..7bcecb8ff --- /dev/null +++ b/tests/test-mpsc-queue.c @@ -0,0 +1,772 @@ +/* + * Copyright (c) 2020 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#undef NDEBUG +#include +#include +#include + +#include + +#include "command-line.h" +#include "guarded-list.h" +#include "mpsc-queue.h" +#include "openvswitch/list.h" +#include "openvswitch/util.h" +#include "openvswitch/vlog.h" +#include "ovs-rcu.h" +#include "ovs-thread.h" +#include "ovstest.h" +#include "timeval.h" +#include "util.h" + +struct element { + union { + struct mpsc_queue_node mpscq; + struct ovs_list list; + } node; + uint64_t mark; +}; + +static void +test_mpsc_queue_mark_element(struct mpsc_queue_node *node, + uint64_t mark, + unsigned int *counter) +{ + struct element *elem; + + elem = CONTAINER_OF(node, struct element, node.mpscq); + elem->mark = mark; + *counter += 1; +} + +static void +test_mpsc_queue_insert(void) +{ + struct element elements[100]; + struct mpsc_queue_node *node; + struct mpsc_queue queue; + unsigned int counter; + size_t i; + + memset(elements, 0, sizeof(elements)); + mpsc_queue_init(&queue); + mpsc_queue_acquire(&queue); + + for (i = 0; i < ARRAY_SIZE(elements); i++) { + mpsc_queue_insert(&queue, &elements[i].node.mpscq); + } + + counter = 0; + while (mpsc_queue_poll(&queue, &node) == MPSC_QUEUE_ITEM) { + test_mpsc_queue_mark_element(node, 1, &counter); + } + + mpsc_queue_release(&queue); + mpsc_queue_destroy(&queue); + + ovs_assert(counter == ARRAY_SIZE(elements)); + for (i = 0; i < ARRAY_SIZE(elements); i++) { + ovs_assert(elements[i].mark == 1); + } + + printf("."); +} + +static void +test_mpsc_queue_removal_fifo(void) +{ + struct element elements[100]; + struct mpsc_queue_node *node; + struct mpsc_queue queue; + unsigned int counter; + size_t i; + + memset(elements, 0, sizeof(elements)); + + mpsc_queue_init(&queue); + mpsc_queue_acquire(&queue); + + for (i = 0; i < ARRAY_SIZE(elements); i++) { + mpsc_queue_insert(&queue, &elements[i].node.mpscq); + } + + /* Elements are in the same order in the list as they + * were declared / initialized. + */ + counter = 0; + while (mpsc_queue_poll(&queue, &node) == MPSC_QUEUE_ITEM) { + test_mpsc_queue_mark_element(node, counter, &counter); + } + + /* The list is valid once extracted from the queue, + * the queue can be destroyed here. + */ + mpsc_queue_release(&queue); + mpsc_queue_destroy(&queue); + + for (i = 0; i < ARRAY_SIZE(elements) - 1; i++) { + struct element *e1, *e2; + + e1 = &elements[i]; + e2 = &elements[i + 1]; + + ovs_assert(e1->mark < e2->mark); + } + + printf("."); +} + +/* Partial insert: + * + * Those functions are 'mpsc_queue_insert()' divided in two parts. + * They serve to test the behavior of the queue when forcing the potential + * condition of a thread starting an insertion then yielding. + */ +static struct mpsc_queue_node * +mpsc_queue_insert_begin(struct mpsc_queue *queue, struct mpsc_queue_node *node) +{ + struct mpsc_queue_node *prev; + + atomic_store_explicit(&node->next, NULL, memory_order_relaxed); + prev = atomic_exchange_explicit(&queue->head, node, memory_order_acq_rel); + return prev; +} + +static void +mpsc_queue_insert_end(struct mpsc_queue_node *prev, + struct mpsc_queue_node *node) +{ + atomic_store_explicit(&prev->next, node, memory_order_release); +} + +static void +test_mpsc_queue_insert_partial(void) +{ + struct element elements[10]; + struct mpsc_queue_node *prevs[ARRAY_SIZE(elements)]; + struct mpsc_queue_node *node; + struct mpsc_queue queue, *q = &queue; + size_t i; + + mpsc_queue_init(q); + + /* Insert the first half of elements entirely, + * insert the second hald of elements partially. + */ + for (i = 0; i < ARRAY_SIZE(elements); i++) { + elements[i].mark = i; + if (i > ARRAY_SIZE(elements) / 2) { + prevs[i] = mpsc_queue_insert_begin(q, &elements[i].node.mpscq); + } else { + prevs[i] = NULL; + mpsc_queue_insert(q, &elements[i].node.mpscq); + } + } + + mpsc_queue_acquire(q); + + /* Verify that when the chain is broken, iterators will stop. */ + i = 0; + MPSC_QUEUE_FOR_EACH (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e == &elements[i]); + i++; + } + ovs_assert(i < ARRAY_SIZE(elements)); + + for (i = 0; i < ARRAY_SIZE(elements); i++) { + if (prevs[i] != NULL) { + mpsc_queue_insert_end(prevs[i], &elements[i].node.mpscq); + } + } + + i = 0; + MPSC_QUEUE_FOR_EACH (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e == &elements[i]); + i++; + } + ovs_assert(i == ARRAY_SIZE(elements)); + + MPSC_QUEUE_FOR_EACH_POP (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e->mark == (unsigned int)(e - elements)); + } + + mpsc_queue_release(q); + mpsc_queue_destroy(q); + + printf("."); +} + +static void +test_mpsc_queue_push_front(void) +{ + struct mpsc_queue queue, *q = &queue; + struct mpsc_queue_node *node; + struct element elements[10]; + size_t i; + + mpsc_queue_init(q); + mpsc_queue_acquire(q); + + ovs_assert(mpsc_queue_pop(q) == NULL); + mpsc_queue_push_front(q, &elements[0].node.mpscq); + node = mpsc_queue_pop(q); + ovs_assert(node == &elements[0].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == NULL); + + mpsc_queue_push_front(q, &elements[0].node.mpscq); + mpsc_queue_push_front(q, &elements[1].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[1].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[0].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == NULL); + + mpsc_queue_push_front(q, &elements[1].node.mpscq); + mpsc_queue_push_front(q, &elements[0].node.mpscq); + mpsc_queue_insert(q, &elements[2].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[0].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[1].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == &elements[2].node.mpscq); + ovs_assert(mpsc_queue_pop(q) == NULL); + + for (i = 0; i < ARRAY_SIZE(elements); i++) { + elements[i].mark = i; + mpsc_queue_insert(q, &elements[i].node.mpscq); + } + + node = mpsc_queue_pop(q); + mpsc_queue_push_front(q, node); + ovs_assert(mpsc_queue_pop(q) == node); + mpsc_queue_push_front(q, node); + + i = 0; + MPSC_QUEUE_FOR_EACH (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e == &elements[i]); + i++; + } + ovs_assert(i == ARRAY_SIZE(elements)); + + MPSC_QUEUE_FOR_EACH_POP (node, q) { + struct element *e = CONTAINER_OF(node, struct element, node.mpscq); + ovs_assert(e->mark == (unsigned int)(e - elements)); + } + + mpsc_queue_release(q); + mpsc_queue_destroy(q); + + printf("."); +} + +static void +run_tests(struct ovs_cmdl_context *ctx OVS_UNUSED) +{ + /* Verify basic insertion. */ + test_mpsc_queue_insert(); + /* Test partial insertion. */ + test_mpsc_queue_insert_partial(); + /* Verify removal order is respected. */ + test_mpsc_queue_removal_fifo(); + /* Verify tail-end insertion works. */ + test_mpsc_queue_push_front(); + printf("\n"); +} + +static struct element *elements; +static uint64_t *thread_working_ms; /* Measured work time. */ + +static unsigned int n_threads; +static unsigned int n_elems; + +static struct ovs_barrier barrier; +static volatile bool working; + +static int +elapsed(const struct timeval *start) +{ + struct timeval end; + + xgettimeofday(&end); + return timeval_to_msec(&end) - timeval_to_msec(start); +} + +static void +print_result(const char *prefix, int reader_elapsed) +{ + uint64_t avg; + size_t i; + + avg = 0; + for (i = 0; i < n_threads; i++) { + avg += thread_working_ms[i]; + } + avg /= n_threads; + printf("%s: %6d", prefix, reader_elapsed); + for (i = 0; i < n_threads; i++) { + printf(" %6" PRIu64, thread_working_ms[i]); + } + printf(" %6" PRIu64 " ms\n", avg); +} + +struct mpscq_aux { + struct mpsc_queue *queue; + atomic_uint thread_id; +}; + +static void * +mpsc_queue_insert_thread(void *aux_) +{ + unsigned int n_elems_per_thread; + struct element *th_elements; + struct mpscq_aux *aux = aux_; + struct timeval start; + unsigned int id; + size_t i; + + atomic_add(&aux->thread_id, 1u, &id); + n_elems_per_thread = n_elems / n_threads; + th_elements = &elements[id * n_elems_per_thread]; + + ovs_barrier_block(&barrier); + xgettimeofday(&start); + + for (i = 0; i < n_elems_per_thread; i++) { + mpsc_queue_insert(aux->queue, &th_elements[i].node.mpscq); + } + + thread_working_ms[id] = elapsed(&start); + ovs_barrier_block(&barrier); + + working = false; + + return NULL; +} + +static void +benchmark_mpsc_queue(void) +{ + struct mpsc_queue_node *node; + struct mpsc_queue queue; + struct timeval start; + unsigned int counter; + bool work_complete; + pthread_t *threads; + struct mpscq_aux aux; + uint64_t epoch; + size_t i; + + memset(elements, 0, n_elems & sizeof *elements); + memset(thread_working_ms, 0, n_threads & sizeof *thread_working_ms); + + mpsc_queue_init(&queue); + + aux.queue = &queue; + atomic_store(&aux.thread_id, 0); + + for (i = n_elems - (n_elems % n_threads); i < n_elems; i++) { + mpsc_queue_insert(&queue, &elements[i].node.mpscq); + } + + working = true; + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("sc_queue_insert", + mpsc_queue_insert_thread, &aux); + } + + mpsc_queue_acquire(&queue); + xgettimeofday(&start); + + counter = 0; + epoch = 1; + do { + while (mpsc_queue_poll(&queue, &node) == MPSC_QUEUE_ITEM) { + test_mpsc_queue_mark_element(node, epoch, &counter); + } + if (epoch == UINT64_MAX) { + epoch = 0; + } + epoch++; + } while (working); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + /* Elements might have been inserted before threads were joined. */ + while (mpsc_queue_poll(&queue, &node) == MPSC_QUEUE_ITEM) { + test_mpsc_queue_mark_element(node, epoch, &counter); + } + + print_result(" mpsc-queue", elapsed(&start)); + + mpsc_queue_release(&queue); + mpsc_queue_destroy(&queue); + ovs_barrier_destroy(&barrier); + free(threads); + + work_complete = true; + for (i = 0; i < n_elems; i++) { + if (elements[i].mark == 0) { + printf("Element %" PRIuSIZE " was never consumed.\n", i); + work_complete = false; + } + } + ovs_assert(work_complete); + ovs_assert(counter == n_elems); +} + +#ifdef HAVE_PTHREAD_SPIN_LOCK +#define spin_lock_type ovs_spin +#define spin_lock_init(l) ovs_spin_init(l) +#define spin_lock_destroy(l) ovs_spin_destroy(l) +#define spin_lock(l) ovs_spin_lock(l) +#define spin_unlock(l) ovs_spin_unlock(l) +#else +#define spin_lock_type ovs_mutex +#define spin_lock_init(l) ovs_mutex_init(l) +#define spin_lock_destroy(l) ovs_mutex_destroy(l) +#define spin_lock(l) ovs_mutex_lock(l) +#define spin_unlock(l) ovs_mutex_unlock(l) +#endif + +struct list_aux { + struct ovs_list *list; + struct ovs_mutex *mutex; + struct spin_lock_type *spin; + atomic_uint thread_id; +}; + +static void * +locked_list_insert_main(void *aux_) + OVS_NO_THREAD_SAFETY_ANALYSIS +{ + unsigned int n_elems_per_thread; + struct element *th_elements; + struct list_aux *aux = aux_; + struct timeval start; + unsigned int id; + size_t i; + + atomic_add(&aux->thread_id, 1u, &id); + n_elems_per_thread = n_elems / n_threads; + th_elements = &elements[id * n_elems_per_thread]; + + ovs_barrier_block(&barrier); + xgettimeofday(&start); + + for (i = 0; i < n_elems_per_thread; i++) { + aux->mutex ? ovs_mutex_lock(aux->mutex) + : spin_lock(aux->spin); + ovs_list_push_front(aux->list, &th_elements[i].node.list); + aux->mutex ? ovs_mutex_unlock(aux->mutex) + : spin_unlock(aux->spin); + } + + thread_working_ms[id] = elapsed(&start); + ovs_barrier_block(&barrier); + + working = false; + + return NULL; +} + +static void +benchmark_list(bool use_mutex) +{ + struct ovs_mutex mutex; + struct spin_lock_type spin; + struct ovs_list list; + struct element *elem; + struct timeval start; + unsigned int counter; + bool work_complete; + pthread_t *threads; + struct list_aux aux; + uint64_t epoch; + size_t i; + + memset(elements, 0, n_elems * sizeof *elements); + memset(thread_working_ms, 0, n_threads * sizeof *thread_working_ms); + + use_mutex ? ovs_mutex_init(&mutex) : spin_lock_init(&spin); + + ovs_list_init(&list); + + aux.list = &list; + aux.mutex = use_mutex ? &mutex : NULL; + aux.spin = use_mutex ? NULL : &spin; + atomic_store(&aux.thread_id, 0); + + for (i = n_elems - (n_elems % n_threads); i < n_elems; i++) { + ovs_list_push_front(&list, &elements[i].node.list); + } + + working = true; + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("locked_list_insert", + locked_list_insert_main, &aux); + } + + xgettimeofday(&start); + + counter = 0; + epoch = 1; + do { + if (use_mutex) { + ovs_mutex_lock(&mutex); + LIST_FOR_EACH_POP (elem, node.list, &list) { + elem->mark = epoch; + counter++; + } + ovs_mutex_unlock(&mutex); + } else { + struct ovs_list *node = NULL; + + spin_lock(&spin); + if (!ovs_list_is_empty(&list)) { + node = ovs_list_pop_front(&list); + } + spin_unlock(&spin); + + if (!node) { + continue; + } + + elem = CONTAINER_OF(node, struct element, node.list); + elem->mark = epoch; + counter++; + } + if (epoch == UINT64_MAX) { + epoch = 0; + } + epoch++; + } while (working); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + /* Elements might have been inserted before threads were joined. */ + LIST_FOR_EACH_POP (elem, node.list, &list) { + elem->mark = epoch; + counter++; + } + + if (use_mutex) { + print_result(" list(mutex)", elapsed(&start)); + } else { + print_result(" list(spin)", elapsed(&start)); + } + + use_mutex ? ovs_mutex_destroy(&mutex) : spin_lock_destroy(&spin); + ovs_barrier_destroy(&barrier); + free(threads); + + work_complete = true; + for (i = 0; i < n_elems; i++) { + if (elements[i].mark == 0) { + printf("Element %" PRIuSIZE " was never consumed.\n", i); + work_complete = false; + } + } + ovs_assert(work_complete); + ovs_assert(counter == n_elems); +} + +struct guarded_list_aux { + struct guarded_list *glist; + atomic_uint thread_id; +}; + +static void * +guarded_list_insert_thread(void *aux_) +{ + unsigned int n_elems_per_thread; + struct element *th_elements; + struct guarded_list_aux *aux = aux_; + struct timeval start; + unsigned int id; + size_t i; + + atomic_add(&aux->thread_id, 1u, &id); + n_elems_per_thread = n_elems / n_threads; + th_elements = &elements[id * n_elems_per_thread]; + + ovs_barrier_block(&barrier); + xgettimeofday(&start); + + for (i = 0; i < n_elems_per_thread; i++) { + guarded_list_push_back(aux->glist, &th_elements[i].node.list, n_elems); + } + + thread_working_ms[id] = elapsed(&start); + ovs_barrier_block(&barrier); + + working = false; + + return NULL; +} + +static void +benchmark_guarded_list(void) +{ + struct guarded_list_aux aux; + struct ovs_list extracted; + struct guarded_list glist; + struct element *elem; + struct timeval start; + unsigned int counter; + bool work_complete; + pthread_t *threads; + uint64_t epoch; + size_t i; + + memset(elements, 0, n_elems * sizeof *elements); + memset(thread_working_ms, 0, n_threads * sizeof *thread_working_ms); + + guarded_list_init(&glist); + ovs_list_init(&extracted); + + aux.glist = &glist; + atomic_store(&aux.thread_id, 0); + + for (i = n_elems - (n_elems % n_threads); i < n_elems; i++) { + guarded_list_push_back(&glist, &elements[i].node.list, n_elems); + } + + working = true; + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("guarded_list_insert", + guarded_list_insert_thread, &aux); + } + + xgettimeofday(&start); + + counter = 0; + epoch = 1; + do { + guarded_list_pop_all(&glist, &extracted); + LIST_FOR_EACH_POP (elem, node.list, &extracted) { + elem->mark = epoch; + counter++; + } + if (epoch == UINT64_MAX) { + epoch = 0; + } + epoch++; + } while (working); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + /* Elements might have been inserted before threads were joined. */ + guarded_list_pop_all(&glist, &extracted); + LIST_FOR_EACH_POP (elem, node.list, &extracted) { + elem->mark = epoch; + counter++; + } + + print_result("guarded list", elapsed(&start)); + + ovs_barrier_destroy(&barrier); + free(threads); + guarded_list_destroy(&glist); + + work_complete = true; + for (i = 0; i < n_elems; i++) { + if (elements[i].mark == 0) { + printf("Element %" PRIuSIZE " was never consumed.\n", i); + work_complete = false; + } + } + ovs_assert(work_complete); + ovs_assert(counter == n_elems); +} + +static void +run_benchmarks(struct ovs_cmdl_context *ctx) +{ + long int l_threads; + long int l_elems; + size_t i; + + ovsrcu_quiesce_start(); + + l_elems = strtol(ctx->argv[1], NULL, 10); + l_threads = strtol(ctx->argv[2], NULL, 10); + ovs_assert(l_elems > 0 && l_threads > 0); + + n_elems = l_elems; + n_threads = l_threads; + + elements = xcalloc(n_elems, sizeof *elements); + thread_working_ms = xcalloc(n_threads, sizeof *thread_working_ms); + + printf("Benchmarking n=%u on 1 + %u threads.\n", n_elems, n_threads); + + printf(" type\\thread: Reader "); + for (i = 0; i < n_threads; i++) { + printf(" %3" PRIuSIZE " ", i + 1); + } + printf(" Avg\n"); + + benchmark_mpsc_queue(); +#ifdef HAVE_PTHREAD_SPIN_LOCK + benchmark_list(false); +#endif + benchmark_list(true); + benchmark_guarded_list(); + + free(thread_working_ms); + free(elements); +} + +static const struct ovs_cmdl_command commands[] = { + {"check", NULL, 0, 0, run_tests, OVS_RO}, + {"benchmark", " ", 2, 2, run_benchmarks, OVS_RO}, + {NULL, NULL, 0, 0, NULL, OVS_RO}, +}; + +static void +test_mpsc_queue_main(int argc, char *argv[]) +{ + struct ovs_cmdl_context ctx = { + .argc = argc - optind, + .argv = argv + optind, + }; + + vlog_set_levels(NULL, VLF_ANY_DESTINATION, VLL_OFF); + + set_program_name(argv[0]); + ovs_cmdl_run_command(&ctx, commands); +} + +OVSTEST_REGISTER("test-mpsc-queue", test_mpsc_queue_main); From patchwork Wed Jun 9 13:09:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489870 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=hxQvHDFH; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=UwR3y2j5; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SCM3q6qz9sW8 for ; Wed, 9 Jun 2021 23:10:39 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 87E09405EF; Wed, 9 Jun 2021 13:10:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7Bl71Rf34ihW; Wed, 9 Jun 2021 13:10:32 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 7E7CB40557; Wed, 9 Jun 2021 13:10:27 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5516AC000D; Wed, 9 Jun 2021 13:10:27 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2BCE9C0024 for ; Wed, 9 Jun 2021 13:10:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 9E3C160B15 for ; Wed, 9 Jun 2021 13:10:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="hxQvHDFH"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="UwR3y2j5" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qMsynDz7n8Kn for ; Wed, 9 Jun 2021 13:10:09 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp3.osuosl.org (Postfix) with ESMTPS id 05C8160A37 for ; Wed, 9 Jun 2021 13:10:05 +0000 (UTC) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 58189363 for ; Wed, 9 Jun 2021 09:10:05 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Wed, 09 Jun 2021 09:10:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; s=fm2; bh=1XcMic46qaOtxODTWfH30bFOLb Ll3+lB4r0ifFhuj60=; b=hxQvHDFHrRhQEsZHToyzko7lqoK/Qeoa/o41ST2Hfn A3Mb1ip1AAOuZDdy8JZNV1wEoq6t2D8pW5Qgsv23K7R6EQXaGNfg4JLnsYJDUUeo kruv85nZQ5MGtpgQ956JMzHzedyPSd3vsky9PrY4411w5FzeU7YmdXbPUyTiNt+x qXDrFH3up5oyeBZYXhnCrELY/q4IRBEdDuZOxxjhkUfq/kArn/25FB+qNTqN1mVE TxV/19h+RzYQw/qGaUQwC1OmXNGJUCCoBOUy5fbPUBWRWlCaPZa83JS3DJx3oUHd 2A/qPyAjyrUVmexEe/hpDS2olGrvYIVsQUXJEEPctmSA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=1XcMic46qaOtxODTWfH30bFOLbLl3+lB4r0ifFhuj60=; b=UwR3y2j5 gNNRpRo+gE2nPPLsNknyyYbOQjnGyuRVd2589l8YREzeA2Te635oo+0JqutrFksT Q0L/CsJl4CEsKidByvqD02skAn7LLXrBWgB0Bt8uuPkZQGL4sBQE3mJ6s2O33dW5 Pk1ZWh7t78t930f2HdCr6TDmbnzjKeyte48Wy+l0l7vVlQjBNRxwOnZ4u6ae5uDk ot2wHnyhY2tcqOQMuOktSpMooONR/yKxqrnCr6wIyRly3Bphg9K8OiejDmhqMDq8 MfUIuQDrzIKmV+9IlngV1Vm/05ChvDefcbWZbPP0l5t1Syi8udGtMp2wrw5HvD15 XcK1i7NsiUkQrw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgiedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhgggfestdekre dtredttdenucfhrhhomhepifgrvghtrghnucftihhvvghtuceoghhrihhvvgesuhdvheei rdhnvghtqeenucggtffrrghtthgvrhhnpeduhfefieekieeiudetgfehkeeludektdekje ehudehffetieduheeigfekvdelfeenucffohhmrghinheprghprggthhgvrdhorhhgnecu vehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 9 Jun 2021 09:10:04 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:21 +0200 Message-Id: <0a1797e66ed5146b20080069b8ccaeb1339b8b4b.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v4 13/27] id-fpool: Module for fast ID generation X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The current id-pool module is slow to allocate the next valid ID, and can be optimized when restricting some properties of the pool. Those restrictions are: * No ability to add a random ID to the pool. * A new ID is no more the smallest possible ID. It is however guaranteed to be in the range of [floor, last_alloc + nb_user * cache_size + 1]. where 'cache_size' is the number of ID in each per-user cache. It is defined as 'ID_FPOOL_CACHE_SIZE' to 64. * A user should never free an ID that is not allocated. No checks are done and doing so will duplicate the spurious ID. Refcounting or other memory management scheme should be used to ensure an object and its ID are only freed once. This allocator is designed to scale reasonably well in multithread setup. As it is aimed at being a faster replacement to the current id-pool, a benchmark has been implemented alongside unit tests. The benchmark is composed of 4 rounds: 'new', 'del', 'mix', and 'rnd'. Respectively + 'new': only allocate IDs + 'del': only free IDs + 'mix': allocate, sequential free, then allocate ID. + 'rnd': allocate, random free, allocate ID. Randomized freeing is done by swapping the latest allocated ID with any from the range of currently allocated ID, which is reminiscent of the Fisher-Yates shuffle. This evaluates freeing non-sequential IDs, which is the more natural use-case. For this specific round, the id-pool performance is such that a timeout of 10 seconds is added to the benchmark: $ ./tests/ovstest test-id-fpool benchmark 10000 1 Benchmarking n=10000 on 1 thread. type\thread: 1 Avg id-fpool new: 1 1 ms id-fpool del: 1 1 ms id-fpool mix: 2 2 ms id-fpool rnd: 2 2 ms id-pool new: 4 4 ms id-pool del: 2 2 ms id-pool mix: 6 6 ms id-pool rnd: 431 431 ms $ ./tests/ovstest test-id-fpool benchmark 100000 1 Benchmarking n=100000 on 1 thread. type\thread: 1 Avg id-fpool new: 2 2 ms id-fpool del: 2 2 ms id-fpool mix: 3 3 ms id-fpool rnd: 4 4 ms id-pool new: 12 12 ms id-pool del: 5 5 ms id-pool mix: 16 16 ms id-pool rnd: 10000+ -1 ms $ ./tests/ovstest test-id-fpool benchmark 1000000 1 Benchmarking n=1000000 on 1 thread. type\thread: 1 Avg id-fpool new: 15 15 ms id-fpool del: 12 12 ms id-fpool mix: 34 34 ms id-fpool rnd: 48 48 ms id-pool new: 276 276 ms id-pool del: 286 286 ms id-pool mix: 448 448 ms id-pool rnd: 10000+ -1 ms Running only a performance test on the fast pool: $ ./tests/ovstest test-id-fpool perf 1000000 1 Benchmarking n=1000000 on 1 thread. type\thread: 1 Avg id-fpool new: 15 15 ms id-fpool del: 12 12 ms id-fpool mix: 34 34 ms id-fpool rnd: 47 47 ms $ ./tests/ovstest test-id-fpool perf 1000000 2 Benchmarking n=1000000 on 2 threads. type\thread: 1 2 Avg id-fpool new: 11 11 11 ms id-fpool del: 10 10 10 ms id-fpool mix: 24 24 24 ms id-fpool rnd: 30 30 30 ms $ ./tests/ovstest test-id-fpool perf 1000000 4 Benchmarking n=1000000 on 4 threads. type\thread: 1 2 3 4 Avg id-fpool new: 9 11 11 10 10 ms id-fpool del: 5 6 6 5 5 ms id-fpool mix: 16 16 16 16 16 ms id-fpool rnd: 20 20 20 20 20 ms Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/automake.mk | 2 + lib/id-fpool.c | 279 +++++++++++++++++++ lib/id-fpool.h | 66 +++++ tests/automake.mk | 1 + tests/library.at | 4 + tests/test-id-fpool.c | 615 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 967 insertions(+) create mode 100644 lib/id-fpool.c create mode 100644 lib/id-fpool.h create mode 100644 tests/test-id-fpool.c diff --git a/lib/automake.mk b/lib/automake.mk index b45801852..afff2e09c 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -138,6 +138,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/hmap.c \ lib/hmapx.c \ lib/hmapx.h \ + lib/id-fpool.c \ + lib/id-fpool.h \ lib/id-pool.c \ lib/id-pool.h \ lib/if-notifier-manual.c \ diff --git a/lib/id-fpool.c b/lib/id-fpool.c new file mode 100644 index 000000000..15cef5d00 --- /dev/null +++ b/lib/id-fpool.c @@ -0,0 +1,279 @@ +/* + * Copyright (c) 2021 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "openvswitch/list.h" +#include "openvswitch/thread.h" +#include "openvswitch/util.h" +#include "ovs-atomic.h" +#include "id-fpool.h" + +#ifdef HAVE_PTHREAD_SPIN_LOCK +#define id_fpool_lock_type ovs_spin +#define id_fpool_lock_init(l) do { ovs_spin_init(l); } while (0) +#define id_fpool_lock_destroy(l) do { ovs_spin_destroy(l); } while (0) +#define id_fpool_lock(l) do { ovs_spin_lock(l); } while (0) +#define id_fpool_unlock(l) do { ovs_spin_unlock(l); } while (0) +#else +#define id_fpool_lock_type ovs_mutex +#define id_fpool_lock_init(l) do { ovs_mutex_init(l); } while (0) +#define id_fpool_lock_destroy(l) do { ovs_mutex_destroy(l); } while (0) +#define id_fpool_lock(l) do { ovs_mutex_lock(l); } while (0) +#define id_fpool_unlock(l) do { ovs_mutex_unlock(l); } while (0) +#endif + +struct id_slab { + struct ovs_list node; + uint32_t pos; + uint32_t ids[ID_FPOOL_CACHE_SIZE]; +}; + +struct per_user { +PADDED_MEMBERS(CACHE_LINE_SIZE, + struct id_fpool_lock_type user_lock; + struct id_slab *slab; +);}; + +struct id_fpool { + /* Constants */ + uint32_t floor; /* IDs are in the range of [floor, ceiling). */ + uint32_t ceiling; + size_t nb_user; /* Number of concurrent users. */ + + /* Shared mutable data protected by global lock. */ + struct id_fpool_lock_type pool_lock; + struct ovs_list free_slabs; + uint32_t next_id; + + /* Per-user mutable data protected by user locks. */ + struct per_user per_users[0]; +}; + +/* Lock precedence is + * 1: per_users.user_lock + * 2: pool_lock + */ + +static struct id_slab * +id_slab_create(uint32_t *next_id, uint32_t max) +{ + struct id_slab *slab; + size_t n_ids; + size_t pos; + + if (next_id[0] == max) { + return NULL; + } + + n_ids = max - next_id[0]; + slab = xmalloc(sizeof *slab); + ovs_list_init(&slab->node); + slab->pos = 0; + + for (pos = MIN(n_ids, ARRAY_SIZE(slab->ids)); pos > 0; pos--) { + slab->ids[pos - 1] = next_id[0]; + next_id[0]++; + slab->pos++; + } + + return slab; +} + +static bool +id_slab_insert(struct id_slab *slab, uint32_t id) +{ + if (slab == NULL) { + return false; + } + if (slab->pos >= ARRAY_SIZE(slab->ids)) { + return false; + } + slab->ids[slab->pos++] = id; + return true; +} + +static bool +id_slab_remove(struct id_slab *slab, uint32_t *id) +{ + if (slab == NULL) { + return false; + } + if (slab->pos == 0) { + return false; + } + *id = slab->ids[--slab->pos]; + return true; +} + +static void +per_user_init(struct per_user *pu, uint32_t *next_id, uint32_t max) +{ + id_fpool_lock_init(&pu->user_lock); + pu->slab = id_slab_create(next_id, max); +} + +static void +per_user_destroy(struct per_user *pu) +{ + id_fpool_lock(&pu->user_lock); + free(pu->slab); + pu->slab = NULL; + id_fpool_unlock(&pu->user_lock); + id_fpool_lock_destroy(&pu->user_lock); +} + +struct id_fpool * +id_fpool_create(unsigned int nb_user, uint32_t floor, uint32_t n_ids) +{ + struct id_fpool *pool; + size_t i; + + ovs_assert(nb_user != 0); + ovs_assert(floor <= UINT32_MAX - n_ids); + + pool = xmalloc(sizeof *pool + nb_user * sizeof(struct per_user)); + pool->next_id = floor; + pool->floor = floor; + pool->ceiling = floor + n_ids; + + for (i = 0; i < nb_user; i++) { + per_user_init(&pool->per_users[i], + &pool->next_id, pool->ceiling); + } + pool->nb_user = nb_user; + + id_fpool_lock_init(&pool->pool_lock); + ovs_list_init(&pool->free_slabs); + + return pool; +} + +void +id_fpool_destroy(struct id_fpool *pool) +{ + struct id_slab *slab; + struct id_slab *next; + size_t i; + + id_fpool_lock(&pool->pool_lock); + LIST_FOR_EACH_SAFE (slab, next, node, &pool->free_slabs) { + free(slab); + } + ovs_list_poison(&pool->free_slabs); + id_fpool_unlock(&pool->pool_lock); + id_fpool_lock_destroy(&pool->pool_lock); + + for (i = 0; i < pool->nb_user; i++) { + per_user_destroy(&pool->per_users[i]); + } + free(pool); +} + +bool +id_fpool_new_id(struct id_fpool *pool, unsigned int uid, uint32_t *id) +{ + struct per_user *pu; + unsigned int uid2; + bool res = false; + + ovs_assert(uid < pool->nb_user); + pu = &pool->per_users[uid]; + + id_fpool_lock(&pu->user_lock); + + if (id_slab_remove(pu->slab, id)) { + res = true; + goto unlock_and_ret; + } + free(pu->slab); + + id_fpool_lock(&pool->pool_lock); + if (!ovs_list_is_empty(&pool->free_slabs)) { + pu->slab = CONTAINER_OF(ovs_list_pop_front(&pool->free_slabs), + struct id_slab, node); + } else { + pu->slab = id_slab_create(&pool->next_id, pool->ceiling); + } + id_fpool_unlock(&pool->pool_lock); + + if (pu->slab != NULL) { + res = id_slab_remove(pu->slab, id); + goto unlock_and_ret; + } + + id_fpool_unlock(&pu->user_lock); + + /* No ID available in local slab, no slab available in shared list. + * The shared counter is maxed out. Attempt to steal an ID from another + * user slab. */ + + for (uid2 = 0; uid2 < pool->nb_user; uid2++) { + struct per_user *pu2 = &pool->per_users[uid2]; + + if (uid == uid2) { + continue; + } + id_fpool_lock(&pu2->user_lock);; + res = id_slab_remove(pu2->slab, id); + id_fpool_unlock(&pu2->user_lock);; + if (res) { + break; + } + } + + goto out; + +unlock_and_ret: + id_fpool_unlock(&pu->user_lock); +out: + return res; +} + +void +id_fpool_free_id(struct id_fpool *pool, unsigned int uid, uint32_t id) +{ + struct per_user *pu; + + if (id < pool->floor || id >= pool->ceiling) { + return; + } + + ovs_assert(uid < pool->nb_user); + pu = &pool->per_users[uid]; + + id_fpool_lock(&pu->user_lock); + + if (pu->slab == NULL) { + /* Create local slab with a single ID. */ + pu->slab = id_slab_create(&id, id + 1); + goto unlock; + } + + if (id_slab_insert(pu->slab, id)) { + goto unlock; + } + + id_fpool_lock(&pool->pool_lock); + ovs_list_push_back(&pool->free_slabs, &pu->slab->node); + id_fpool_unlock(&pool->pool_lock); + + /* Create local slab with a single ID. */ + pu->slab = id_slab_create(&id, id + 1); + +unlock: + id_fpool_unlock(&pu->user_lock); +} diff --git a/lib/id-fpool.h b/lib/id-fpool.h new file mode 100644 index 000000000..f8d855938 --- /dev/null +++ b/lib/id-fpool.h @@ -0,0 +1,66 @@ +/* + * Copyright (c) 2021 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef ID_FPOOL_H +#define ID_FPOOL_H + +#include +#include +#include + +/* + * Fast ID pool. + * ============= + * + * Unordered pool of unique 32 bits IDs. + * + * Multiple users are registered at initialization. Each one gets a cache + * of ID. When each thread allocates from the pool using its own user ID, + * the pool scales for concurrent allocation. + * + * New IDs are always in the range of '[floor, next_id]', where 'next_id' is + * in the range of '[last_allocated_ID + nb_user * cache_size + 1]'. + * This means that a new ID is not always the smallest available ID, but it is + * still from a limited range. + * + * Users should ensure that an ID is *never* freed twice. Not doing so will + * have the effect of double-allocating such ID afterward. + * + * Thread-safety + * ============= + * + * APIs are thread safe. + * Multiple threads can share the same user ID if necessary. + */ + +#define ID_FPOOL_CACHE_SIZE 64 + +struct id_fpool; + +/* nb_user is the number of expected users of the pool, + * in terms of execution threads. */ +struct id_fpool *id_fpool_create(unsigned int nb_user, + uint32_t base, uint32_t n_ids); +void id_fpool_destroy(struct id_fpool *pool); + +/* uid is the thread user-id. It should be within '[0, nb_user)'. */ +bool id_fpool_new_id(struct id_fpool *pool, unsigned int uid, uint32_t *id); + +/* uid is the thread user-id. It should be within '[0, nb_user)'. + * An allocated ID must never be freed twice. */ +void id_fpool_free_id(struct id_fpool *pool, unsigned int uid, uint32_t id); + +#endif /* ID_FPOOL_H */ diff --git a/tests/automake.mk b/tests/automake.mk index 4588d5b49..59e078074 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -461,6 +461,7 @@ tests_ovstest_SOURCES = \ tests/test-heap.c \ tests/test-hindex.c \ tests/test-hmap.c \ + tests/test-id-fpool.c \ tests/test-json.c \ tests/test-jsonrpc.c \ tests/test-list.c \ diff --git a/tests/library.at b/tests/library.at index 0e47bc445..40e08c40c 100644 --- a/tests/library.at +++ b/tests/library.at @@ -264,3 +264,7 @@ AT_SETUP([mpsc-queue module]) AT_CHECK([ovstest test-mpsc-queue check], [0], [.... ]) AT_CLEANUP + +AT_SETUP([id-fpool module]) +AT_CHECK([ovstest test-id-fpool check], [0], []) +AT_CLEANUP diff --git a/tests/test-id-fpool.c b/tests/test-id-fpool.c new file mode 100644 index 000000000..25275d9ae --- /dev/null +++ b/tests/test-id-fpool.c @@ -0,0 +1,615 @@ +/* + * Copyright (c) 2021 NVIDIA Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#undef NDEBUG +#include +#include +#include + +#include + +#include "command-line.h" +#include "id-fpool.h" +#include "id-pool.h" +#include "openvswitch/vlog.h" +#include "openvswitch/util.h" +#include "ovs-thread.h" +#include "ovs-rcu.h" +#include "ovs-numa.h" +#include "ovstest.h" +#include "random.h" +#include "timeval.h" +#include "util.h" + +static void +test_id_fpool_alloc(void) +{ + const uint32_t base = 0; + const uint32_t n_id = 10; + struct id_fpool *pool = id_fpool_create(1, base, n_id); + uint32_t ids[10]; + size_t i; + + /* Can do n_id allocs. */ + for (i = 0; i < n_id; i++) { + ovs_assert(id_fpool_new_id(pool, 0, &ids[i])); + ovs_assert(ids[i] >= base); + ovs_assert(ids[i] < base + n_id); + } + /* Only n_id successful allocations. */ + ovs_assert(id_fpool_new_id(pool, 0, NULL) == false); + + /* Monotonic alloc. */ + for (i = 0; i < n_id - 1; i++) { + ovs_assert(ids[i] < ids[i + 1]); + } + + for (i = 0; i < n_id; i++) { + id_fpool_free_id(pool, 0, ids[i]); + } + + /* Can do n_id new allocs. */ + for (i = 0; i < n_id; i++) { + ovs_assert(id_fpool_new_id(pool, 0, &ids[i])); + ovs_assert(ids[i] >= base); + ovs_assert(ids[i] < base + n_id); + } + /* Only n_id successful allocations. */ + ovs_assert(id_fpool_new_id(pool, 0, NULL) == false); + + for (i = 0; i < n_id; i++) { + id_fpool_free_id(pool, 0, ids[i]); + } + + id_fpool_destroy(pool); +} + +static void +test_id_fpool_alloc_range(void) +{ + const uint32_t base = 200; + const uint32_t n_id = 100; + const uint32_t ceil = base + n_id; + struct id_fpool *pool = id_fpool_create(1, base, n_id); + bool id_allocated[100]; + size_t i; + + memset(id_allocated, 0, sizeof id_allocated); + + /* Allocate all IDs only once. */ + for (i = 0; i < n_id; i++) { + uint32_t id; + + ovs_assert(id_fpool_new_id(pool, 0, &id)); + ovs_assert(id >= base); + ovs_assert(id < ceil); + + ovs_assert(id_allocated[id - base] == false); + id_allocated[id - base] = true; + } + /* Only n_id successful allocations. */ + ovs_assert(id_fpool_new_id(pool, 0, NULL) == false); + + for (i = 0; i < n_id; i++) { + ovs_assert(id_allocated[i]); + id_fpool_free_id(pool, 0, base + i); + id_allocated[i] = false; + } + + /* The full range is again fully available. */ + for (i = 0; i < n_id; i++) { + uint32_t id; + + ovs_assert(id_fpool_new_id(pool, 0, &id)); + ovs_assert(id >= base); + ovs_assert(id < ceil); + + ovs_assert(id_allocated[id - base] == false); + id_allocated[id - base] = true; + } + + id_fpool_destroy(pool); +} + +static void +test_id_fpool_alloc_steal(void) +{ + /* N must be less than a slab size to force the second user + * to steal from the first. + */ +#define N (ID_FPOOL_CACHE_SIZE / 2) + bool ids[N]; + struct id_fpool *pool; + uint32_t id; + size_t i; + + memset(ids, 0, sizeof ids); + pool = id_fpool_create(2, 0, N); + + /* Fill up user 0 cache. */ + ovs_assert(id_fpool_new_id(pool, 0, &id)); + for (i = 0; i < N - 1; i++) { + /* Check that user 1 can still alloc from user 0 cache. */ + ovs_assert(id_fpool_new_id(pool, 1, &id)); + } + + id_fpool_destroy(pool); +} + +static void +test_id_fpool_alloc_under_limit(void) +{ + const size_t n_id = 100; + uint32_t ids[100]; + unsigned int limit; + struct id_fpool *pool; + size_t i; + + memset(ids, 0, sizeof ids); + pool = id_fpool_create(1, 0, n_id); + + for (limit = 1; limit < n_id; limit++) { + /* Allocate until arbitrary limit then free allocated ids. */ + for (i = 0; i < limit; i++) { + ovs_assert(id_fpool_new_id(pool, 0, &ids[i])); + } + for (i = 0; i < limit; i++) { + id_fpool_free_id(pool, 0, ids[i]); + } + /* Verify that the N='limit' next allocations are under limit. */ + for (i = 0; i < limit; i++) { + ovs_assert(id_fpool_new_id(pool, 0, &ids[i])); + ovs_assert(ids[i] < limit + ID_FPOOL_CACHE_SIZE); + } + for (i = 0; i < limit; i++) { + id_fpool_free_id(pool, 0, ids[i]); + } + } + + id_fpool_destroy(pool); +} + +static void +run_tests(struct ovs_cmdl_context *ctx OVS_UNUSED) +{ + test_id_fpool_alloc(); + test_id_fpool_alloc_range(); + test_id_fpool_alloc_steal(); + test_id_fpool_alloc_under_limit(); +} + +static uint32_t *ids; +static uint64_t *thread_working_ms; /* Measured work time. */ + +static unsigned int n_threads; +static unsigned int n_ids; + +static struct ovs_barrier barrier; + +#define TIMEOUT_MS (10 * 1000) /* 10 sec timeout */ +static int running_time_ms; +static volatile bool stop = false; + +static int +elapsed(int *start) +{ + return running_time_ms - *start; +} + +static void +swap_u32(uint32_t *a, uint32_t *b) +{ + uint32_t t; + t = *a; + *a = *b; + *b = t; +} + +static void +shuffle(uint32_t *p, size_t n) +{ + for (; n > 1; n--, p++) { + uint32_t *q = &p[random_range(n)]; + swap_u32(p, q); + } +} + +static void +print_result(const char *prefix) +{ + uint64_t avg; + size_t i; + + avg = 0; + for (i = 0; i < n_threads; i++) { + avg += thread_working_ms[i]; + } + avg /= n_threads; + printf("%s: ", prefix); + for (i = 0; i < n_threads; i++) { + if (thread_working_ms[i] >= TIMEOUT_MS) { + printf(" %5" PRIu64 "+", thread_working_ms[i]); + } else { + printf(" %6" PRIu64, thread_working_ms[i]); + } + } + if (avg >= TIMEOUT_MS) { + printf(" -1 ms\n"); + } else { + printf(" %6" PRIu64 " ms\n", avg); + } +} + +struct id_fpool_aux { + struct id_fpool *pool; + atomic_uint thread_id; +}; + +static void * +id_fpool_thread(void *aux_) +{ + unsigned int n_ids_per_thread; + struct id_fpool_aux *aux = aux_; + uint32_t *th_ids; + unsigned int tid; + int start; + size_t i; + + atomic_add(&aux->thread_id, 1u, &tid); + n_ids_per_thread = n_ids / n_threads; + th_ids = &ids[tid * n_ids_per_thread]; + + /* NEW / ALLOC */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ignore(id_fpool_new_id(aux->pool, tid, &th_ids[i])); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* DEL */ + + shuffle(th_ids, n_ids_per_thread); + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + id_fpool_free_id(aux->pool, tid, th_ids[i]); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* MIX */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ignore(id_fpool_new_id(aux->pool, tid, &th_ids[i])); + id_fpool_free_id(aux->pool, tid, th_ids[i]); + ignore(id_fpool_new_id(aux->pool, tid, &th_ids[i])); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* Do not interfere with other threads still in 'MIX' phase. */ + for (i = 0; i < n_ids_per_thread; i++) { + id_fpool_free_id(aux->pool, tid, th_ids[i]); + } + + ovs_barrier_block(&barrier); + + /* MIX SHUFFLED */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + if (elapsed(&start) >= TIMEOUT_MS) { + break; + } + ignore(id_fpool_new_id(aux->pool, tid, &th_ids[i])); + swap_u32(&th_ids[i], &th_ids[random_range(i + 1)]); + id_fpool_free_id(aux->pool, tid, th_ids[i]); + ignore(id_fpool_new_id(aux->pool, tid, &th_ids[i])); + } + thread_working_ms[tid] = elapsed(&start); + + return NULL; +} + +static void +benchmark_id_fpool(void) +{ + pthread_t *threads; + struct id_fpool_aux aux; + size_t i; + + memset(ids, 0, n_ids & sizeof *ids); + memset(thread_working_ms, 0, n_threads & sizeof *thread_working_ms); + + aux.pool = id_fpool_create(n_threads, 0, n_ids); + atomic_store(&aux.thread_id, 0); + + for (i = n_ids - (n_ids % n_threads); i < n_ids; i++) { + id_fpool_new_id(aux.pool, 0, &ids[i]); + } + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads + 1); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("id_fpool_alloc", + id_fpool_thread, &aux); + } + + ovs_barrier_block(&barrier); + + print_result("id-fpool new"); + + ovs_barrier_block(&barrier); + + print_result("id-fpool del"); + + ovs_barrier_block(&barrier); + /* Cleanup. */ + ovs_barrier_block(&barrier); + + print_result("id-fpool mix"); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + print_result("id-fpool rnd"); + + id_fpool_destroy(aux.pool); + ovs_barrier_destroy(&barrier); + free(threads); +} + +struct id_pool_aux { + struct id_pool *pool; + struct ovs_mutex *lock; + atomic_uint thread_id; +}; + +static void * +id_pool_thread(void *aux_) +{ + unsigned int n_ids_per_thread; + struct id_pool_aux *aux = aux_; + uint32_t *th_ids; + unsigned int tid; + int start; + size_t i; + + atomic_add(&aux->thread_id, 1u, &tid); + n_ids_per_thread = n_ids / n_threads; + th_ids = &ids[tid * n_ids_per_thread]; + + /* NEW */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ovs_mutex_lock(aux->lock); + ovs_assert(id_pool_alloc_id(aux->pool, &th_ids[i])); + ovs_mutex_unlock(aux->lock); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* DEL */ + + shuffle(th_ids, n_ids_per_thread); + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ovs_mutex_lock(aux->lock); + id_pool_free_id(aux->pool, th_ids[i]); + ovs_mutex_unlock(aux->lock); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* MIX */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + ovs_mutex_lock(aux->lock); + ignore(id_pool_alloc_id(aux->pool, &th_ids[i])); + id_pool_free_id(aux->pool, th_ids[i]); + ignore(id_pool_alloc_id(aux->pool, &th_ids[i])); + ovs_mutex_unlock(aux->lock); + } + thread_working_ms[tid] = elapsed(&start); + + ovs_barrier_block(&barrier); + + /* Do not interfere with other threads still in 'MIX' phase. */ + ovs_mutex_lock(aux->lock); + for (i = 0; i < n_ids_per_thread; i++) { + id_pool_free_id(aux->pool, th_ids[i]); + } + ovs_mutex_unlock(aux->lock); + + ovs_barrier_block(&barrier); + + /* MIX SHUFFLED */ + + start = running_time_ms; + for (i = 0; i < n_ids_per_thread; i++) { + if (elapsed(&start) >= TIMEOUT_MS) { + break; + } + ovs_mutex_lock(aux->lock); + ignore(id_pool_alloc_id(aux->pool, &th_ids[i])); + swap_u32(&th_ids[i], &th_ids[random_range(i + 1)]); + id_pool_free_id(aux->pool, th_ids[i]); + ignore(id_pool_alloc_id(aux->pool, &th_ids[i])); + ovs_mutex_unlock(aux->lock); + } + thread_working_ms[tid] = elapsed(&start); + + return NULL; +} + +OVS_UNUSED +static void +benchmark_id_pool(void) +{ + pthread_t *threads; + struct id_pool_aux aux; + struct ovs_mutex lock; + size_t i; + + memset(ids, 0, n_ids & sizeof *ids); + memset(thread_working_ms, 0, n_threads & sizeof *thread_working_ms); + + aux.pool = id_pool_create(0, n_ids); + aux.lock = &lock; + ovs_mutex_init(&lock); + atomic_store(&aux.thread_id, 0); + + for (i = n_ids - (n_ids % n_threads); i < n_ids; i++) { + id_pool_alloc_id(aux.pool, &ids[i]); + } + + threads = xmalloc(n_threads * sizeof *threads); + ovs_barrier_init(&barrier, n_threads + 1); + + for (i = 0; i < n_threads; i++) { + threads[i] = ovs_thread_create("id_pool_alloc", id_pool_thread, &aux); + } + + ovs_barrier_block(&barrier); + + print_result(" id-pool new"); + + ovs_barrier_block(&barrier); + + print_result(" id-pool del"); + + ovs_barrier_block(&barrier); + /* Cleanup. */ + ovs_barrier_block(&barrier); + + print_result(" id-pool mix"); + + for (i = 0; i < n_threads; i++) { + xpthread_join(threads[i], NULL); + } + + print_result(" id-pool rnd"); + + id_pool_destroy(aux.pool); + ovs_barrier_destroy(&barrier); + free(threads); +} + +static void * +clock_main(void *arg OVS_UNUSED) +{ + struct timeval start; + struct timeval end; + + xgettimeofday(&start); + while (!stop) { + xgettimeofday(&end); + running_time_ms = timeval_to_msec(&end) - timeval_to_msec(&start); + xnanosleep(1000); + } + + return NULL; +} + +static void +do_perf_test(struct ovs_cmdl_context *ctx, bool test_id_pool) +{ + pthread_t clock; + long int l_threads; + long int l_ids; + size_t i; + + l_ids = strtol(ctx->argv[1], NULL, 10); + l_threads = strtol(ctx->argv[2], NULL, 10); + ovs_assert(l_ids > 0 && l_threads > 0); + + n_ids = l_ids; + n_threads = l_threads; + + ids = xcalloc(n_ids, sizeof *ids); + thread_working_ms = xcalloc(n_threads, sizeof *thread_working_ms); + + clock = ovs_thread_create("clock", clock_main, NULL); + + printf("Benchmarking n=%u on %u thread%s.\n", n_ids, n_threads, + n_threads > 1 ? "s" : ""); + + printf(" type\\thread: "); + for (i = 0; i < n_threads; i++) { + printf(" %3" PRIuSIZE " ", i + 1); + } + printf(" Avg\n"); + + ovsrcu_quiesce_start(); + + benchmark_id_fpool(); + if (test_id_pool) { + benchmark_id_pool(); + } + + stop = true; + + free(thread_working_ms); + xpthread_join(clock, NULL); +} + +static void +run_benchmark(struct ovs_cmdl_context *ctx) +{ + do_perf_test(ctx, true); +} + +static void +run_perf(struct ovs_cmdl_context *ctx) +{ + do_perf_test(ctx, false); +} + +static const struct ovs_cmdl_command commands[] = { + {"check", NULL, 0, 0, run_tests, OVS_RO}, + {"benchmark", " ", 2, 2, run_benchmark, OVS_RO}, + {"perf", " ", 2, 2, run_perf, OVS_RO}, + {NULL, NULL, 0, 0, NULL, OVS_RO}, +}; + +static void +id_fpool_test_main(int argc, char *argv[]) +{ + struct ovs_cmdl_context ctx = { + .argc = argc - optind, + .argv = argv + optind, + }; + + vlog_set_levels(NULL, VLF_ANY_DESTINATION, VLL_OFF); + + set_program_name(argv[0]); + ovs_cmdl_run_command(&ctx, commands); +} + +OVSTEST_REGISTER("test-id-fpool", id_fpool_test_main); From patchwork Wed Jun 9 13:09:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489871 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=3okxlccK; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=N1KFiiCS; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SCf6DrZz9sSs for ; Wed, 9 Jun 2021 23:10:54 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E94BC40611; Wed, 9 Jun 2021 13:10:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S3dz2zJVQICG; Wed, 9 Jun 2021 13:10:51 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id CBA2B41CC3; Wed, 9 Jun 2021 13:10:35 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 93028C000D; Wed, 9 Jun 2021 13:10:35 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2542DC000B for ; Wed, 9 Jun 2021 13:10:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 3E49260A88 for ; Wed, 9 Jun 2021 13:10:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="3okxlccK"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="N1KFiiCS" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v4c8iVMycU3V for ; Wed, 9 Jun 2021 13:10:11 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp3.osuosl.org (Postfix) with ESMTPS id C83DF608EC for ; Wed, 9 Jun 2021 13:10:07 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id E51BD9CD; Wed, 9 Jun 2021 09:10:06 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 09 Jun 2021 09:10:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=bU36bJU1AnRGZ iZNTqJYc2lxZ509njyAUEVbWoORfsQ=; b=3okxlccKmwA0cMklno4q/oB7/fSDq RpasoHNKdFoFQcqkCOO45WSRF25CGNijN6Hb8Jc50OEpXraAxOa6dEeWTG7BHY13 I54OnkrexeyqaySe8i5CTmPsSazVxk07il36vUnK5Tt+4LEyOpjlUS7BSDJEv6IR V8RJWstfvJVtjWcHruwwlSBCXjvBf+J/L6vJhcrICZcYYrRbURGJptQxfwaN/FxE 6VgkbCQ5ikkd4ayA7PfjjeuexAOBDmwn+fN2QYZrxSINszXc/rZuWS7CrhGWJ4uy s1xoLqZPZNNcAoa+sTRPjkV5Oi67CypKpWYoikjbuksfk7ew3NWOyB3rw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=bU36bJU1AnRGZiZNTqJYc2lxZ509njyAUEVbWoORfsQ=; b=N1KFiiCS OzmCqqdIN36/RCrQGwvGes8tFzQA/DTHe77FblGfJyLjn6h//Pc3q/CD/ENe7HK2 r8Ox9dZUtoqDSMDDfrQiW0gdlokr+Y7zkuJh6lygJ0CLLWrTde/nbY/mHMiqyqae 5d05ZbHOyvIuOEUO7X5pf3bgBdn3JifkZ406RwVdYSPHZA02mKsZ/N40u3+x9/fM V4UgtrPyCk5TwOxfh7kdCqmjijBQGRo+RoV5Ik+SFRGAbxm56P6fDnFRqkXlj9Zr jmYiy7q9nF6cLPEA61ClPHns+JyPJhyyV2CKsDmJ+LlOaGJ4SXJGuQBhHTv+SGg7 KhFmab4pY1vakQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:05 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:22 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 14/27] netdev-offload: Add multi-thread API X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Expose functions reporting user configuration of offloading threads, as well as utility functions for multithreading. This will only expose the configuration knob to the user, while no datapath will implement the multiple thread request. This will allow implementations to use this API for offload thread management in relevant layers before enabling the actual dataplane implementation. The offload thread ID is lazily allocated and can as such be in a different order than the offload thread start sequence. The RCU thread will sometime access hardware-offload objects from a provider for reclamation purposes. In such case, it will get a default offload thread ID of 0. Care must be taken that using this thread ID is safe concurrently with the offload threads. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-provider.h | 1 + lib/netdev-offload.c | 88 ++++++++++++++++++++++++++++++++++- lib/netdev-offload.h | 19 ++++++++ vswitchd/vswitch.xml | 16 +++++++ 4 files changed, 122 insertions(+), 2 deletions(-) diff --git a/lib/netdev-offload-provider.h b/lib/netdev-offload-provider.h index 2127599d3..e02330a43 100644 --- a/lib/netdev-offload-provider.h +++ b/lib/netdev-offload-provider.h @@ -84,6 +84,7 @@ struct netdev_flow_api { struct dpif_flow_stats *); /* Get the number of flows offloaded to netdev. + * 'n_flows' is an array of counters, one per offload thread. * Return 0 if successful, otherwise returns a positive errno value. */ int (*flow_get_n_flows)(struct netdev *, uint64_t *n_flows); diff --git a/lib/netdev-offload.c b/lib/netdev-offload.c index deefefd63..087302fd3 100644 --- a/lib/netdev-offload.c +++ b/lib/netdev-offload.c @@ -60,6 +60,12 @@ VLOG_DEFINE_THIS_MODULE(netdev_offload); static bool netdev_flow_api_enabled = false; +#define DEFAULT_OFFLOAD_THREAD_NB 1 +#define MAX_OFFLOAD_THREAD_NB 10 + +static unsigned int offload_thread_nb = DEFAULT_OFFLOAD_THREAD_NB; +DEFINE_EXTERN_PER_THREAD_DATA(netdev_offload_thread_id, OVSTHREAD_ID_UNSET); + /* Protects 'netdev_flow_apis'. */ static struct ovs_mutex netdev_flow_api_provider_mutex = OVS_MUTEX_INITIALIZER; @@ -436,6 +442,64 @@ netdev_is_flow_api_enabled(void) return netdev_flow_api_enabled; } +unsigned int +netdev_offload_thread_nb(void) +{ + return offload_thread_nb; +} + +unsigned int +netdev_offload_ufid_to_thread_id(const ovs_u128 ufid) +{ + uint32_t ufid_hash; + + if (netdev_offload_thread_nb() == 1) { + return 0; + } + + ufid_hash = hash_words64_inline( + (const uint64_t [2]){ ufid.u64.lo, + ufid.u64.hi }, 2, 1); + return ufid_hash % netdev_offload_thread_nb(); +} + +unsigned int +netdev_offload_thread_init(void) +{ + static atomic_count next_id = ATOMIC_COUNT_INIT(0); + bool thread_is_hw_offload; + bool thread_is_rcu; + + thread_is_hw_offload = !strncmp(get_subprogram_name(), + "hw_offload", strlen("hw_offload")); + thread_is_rcu = !strncmp(get_subprogram_name(), "urcu", strlen("urcu")); + + /* Panic if any other thread besides offload and RCU tries + * to initialize their thread ID. */ + ovs_assert(thread_is_hw_offload || thread_is_rcu); + + if (*netdev_offload_thread_id_get() == OVSTHREAD_ID_UNSET) { + unsigned int id; + + if (thread_is_rcu) { + /* RCU will compete with other threads for shared object access. + * Reclamation functions using a thread ID must be thread-safe. + * For that end, and because RCU must consider all potential shared + * objects anyway, its thread-id can be whichever, so return 0. + */ + id = 0; + } else { + /* Only the actual offload threads have their own ID. */ + id = atomic_count_inc(&next_id); + } + /* Panic if any offload thread is getting a spurious ID. */ + ovs_assert(id < netdev_offload_thread_nb()); + return *netdev_offload_thread_id_get() = id; + } else { + return *netdev_offload_thread_id_get(); + } +} + void netdev_ports_flow_flush(const char *dpif_type) { @@ -627,7 +691,16 @@ netdev_ports_get_n_flows(const char *dpif_type, odp_port_t port_no, ovs_rwlock_rdlock(&netdev_hmap_rwlock); data = netdev_ports_lookup(port_no, dpif_type); if (data) { - ret = netdev_flow_get_n_flows(data->netdev, n_flows); + uint64_t thread_n_flows[MAX_OFFLOAD_THREAD_NB] = {0}; + unsigned int tid; + + ret = netdev_flow_get_n_flows(data->netdev, thread_n_flows); + *n_flows = 0; + if (!ret) { + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + *n_flows += thread_n_flows[tid]; + } + } } ovs_rwlock_unlock(&netdev_hmap_rwlock); return ret; @@ -680,7 +753,18 @@ netdev_set_flow_api_enabled(const struct smap *ovs_other_config) if (ovsthread_once_start(&once)) { netdev_flow_api_enabled = true; - VLOG_INFO("netdev: Flow API Enabled"); + offload_thread_nb = smap_get_ullong(ovs_other_config, + "n-offload-threads", + DEFAULT_OFFLOAD_THREAD_NB); + if (offload_thread_nb > MAX_OFFLOAD_THREAD_NB) { + VLOG_WARN("netdev: Invalid number of threads requested: %u", + offload_thread_nb); + offload_thread_nb = DEFAULT_OFFLOAD_THREAD_NB; + } + + VLOG_INFO("netdev: Flow API Enabled, using %u thread%s", + offload_thread_nb, + offload_thread_nb > 1 ? "s" : ""); #ifdef __linux__ tc_set_policy(smap_get_def(ovs_other_config, "tc-policy", diff --git a/lib/netdev-offload.h b/lib/netdev-offload.h index d820e23ed..6e72456e1 100644 --- a/lib/netdev-offload.h +++ b/lib/netdev-offload.h @@ -21,6 +21,7 @@ #include "openvswitch/netdev.h" #include "openvswitch/types.h" #include "ovs-rcu.h" +#include "ovs-thread.h" #include "packets.h" #include "flow.h" @@ -80,6 +81,24 @@ struct offload_info { * to delete the original flow. */ }; +DECLARE_EXTERN_PER_THREAD_DATA(unsigned int, netdev_offload_thread_id); + +unsigned int netdev_offload_thread_nb(void); +unsigned int netdev_offload_thread_init(void); +unsigned int netdev_offload_ufid_to_thread_id(const ovs_u128 ufid); + +static inline unsigned int +netdev_offload_thread_id(void) +{ + unsigned int id = *netdev_offload_thread_id_get(); + + if (OVS_UNLIKELY(id == OVSTHREAD_ID_UNSET)) { + id = netdev_offload_thread_init(); + } + + return id; +} + int netdev_flow_flush(struct netdev *); int netdev_flow_dump_create(struct netdev *, struct netdev_flow_dump **dump, bool terse); diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 4597a215d..022de0fe7 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -247,6 +247,22 @@

+ +

+ Set this value to the number of threads created to manage hardware + offloads. +

+

+ The default value is 1. Changing this value requires + restarting the daemon. +

+

+ This is only relevant if + is enabled. +

+ + From patchwork Wed Jun 9 13:09:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489872 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=IgDtx5zT; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=sQ4YPmOO; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SD12ncqz9sWD for ; Wed, 9 Jun 2021 23:11:13 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 7AFF560A58; Wed, 9 Jun 2021 13:11:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HdhodJ_WoIdI; Wed, 9 Jun 2021 13:11:06 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id A6A7D60A3F; Wed, 9 Jun 2021 13:10:46 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7347DC000B; Wed, 9 Jun 2021 13:10:46 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 25232C000D for ; Wed, 9 Jun 2021 13:10:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 28EAB83CED for ; Wed, 9 Jun 2021 13:10:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="IgDtx5zT"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="sQ4YPmOO" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WPi6Fzgad86l for ; Wed, 9 Jun 2021 13:10:19 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp1.osuosl.org (Postfix) with ESMTPS id B830183D28 for ; Wed, 9 Jun 2021 13:10:09 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 89A4A31E; Wed, 9 Jun 2021 09:10:08 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Wed, 09 Jun 2021 09:10:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=FW999+fbN+b07 wCBEw8Y1fEtb4/PK284WWVtaM6yq0U=; b=IgDtx5zT82M4bav9PyTsmt34Gltnr tftNVxzlh6tXESDYTXmELJBiLbU2OlF5alPwDtKlm3Z+D3p0DazUVmFlYAsebVud lstYWISdoFZovX4hxlcWPU8VAhjy/xifgNcafJUVBXJV7J8cHgdD9kzLeAMhky9j Fg2LKuNI+oddRHAUqvxW8pcw4A13Z3BeEp5iqH6U6U+s209sA5z7gnW38l6y66ih moiHwthmVsVg/GWYA0XUN2+suufizgSF6qYBl0vbGnF3TfSNySguiDRw+SU1V40k LXqq6V42mCy8ZBug2Z4x+QEe1W2NV0BoFOnePsxKol1dPucuY7JjC+UOw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=FW999+fbN+b07wCBEw8Y1fEtb4/PK284WWVtaM6yq0U=; b=sQ4YPmOO 68h9c5foH313l3X3ueEjoaYXwm9yz9bkpE8+gl+3wGtVFr5GydyFo066Mt+YiHX3 wRJ/nZR4bFV+JsgsF9N9aXj0lHMQysicihVpZpJFnRTd458xzLCvTUmMEfVINsAC 3KauqSalsglDob1pttG+Wd6LN4tkAP4WMQMuTvshe6vdXOulxijkP+OiVpHz+U0L vx4xFr/4ZTsthpGxbxI0xrMGny/H5toWh9bC4eVRI4ABIZN5j4oJB9SFv6whSUIp TB+nzuRU3FXGmuEBnrqaQv+NiQHYiho5BKEdbbyZJVo1LIjUhdV2+J/o5Pk05BWq +pxsNhBykI77Cw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:07 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:23 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein Subject: [ovs-dev] [PATCH v4 15/27] dpif-netdev: Quiesce offload thread periodically X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Similar to what was done for the PMD threads [1], reduce the performance impact of quiescing too often in the offload thread. After each processed offload, the offload thread currently quiesce and will sync with RCU. This synchronization can be lengthy and make the thread unnecessary slow. Instead attempt to quiesce every 10 ms at most. While the queue is empty, the offload thread remains quiescent. [1]: 81ac8b3b194c ("dpif-netdev: Do RCU synchronization at fixed interval in PMD main loop.") Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index a20eeda4d..e403e461a 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2740,15 +2740,20 @@ err_free: return -1; } +#define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ + static void * dp_netdev_flow_offload_main(void *data OVS_UNUSED) { struct dp_offload_thread_item *offload; struct ovs_list *list; long long int latency_us; + long long int next_rcu; + long long int now; const char *op; int ret; + next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; for (;;) { ovs_mutex_lock(&dp_offload_thread.mutex); if (ovs_list_is_empty(&dp_offload_thread.list)) { @@ -2756,6 +2761,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) ovs_mutex_cond_wait(&dp_offload_thread.cond, &dp_offload_thread.mutex); ovsrcu_quiesce_end(); + next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; } list = ovs_list_pop_front(&dp_offload_thread.list); dp_offload_thread.enqueued_item--; @@ -2779,7 +2785,9 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) OVS_NOT_REACHED(); } - latency_us = time_usec() - offload->timestamp; + now = time_usec(); + + latency_us = now - offload->timestamp; mov_avg_cma_update(&dp_offload_thread.cma, latency_us); mov_avg_ema_update(&dp_offload_thread.ema, latency_us); @@ -2787,7 +2795,12 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) ret == 0 ? "succeed" : "failed", op, UUID_ARGS((struct uuid *) &offload->flow->mega_ufid)); dp_netdev_free_flow_offload(offload); - ovsrcu_quiesce(); + + /* Do RCU synchronization at fixed interval. */ + if (now > next_rcu) { + ovsrcu_quiesce(); + next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; + } } return NULL; From patchwork Wed Jun 9 13:09:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489873 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=X85oq9ZD; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=JJahtP9Q; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SD42NlSz9sWF for ; Wed, 9 Jun 2021 23:11:16 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 387DF41CB3; Wed, 9 Jun 2021 13:11:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VwkBIJhVuaWv; Wed, 9 Jun 2021 13:11:10 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 9F2AB419BA; Wed, 9 Jun 2021 13:10:55 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5DA66C000D; Wed, 9 Jun 2021 13:10:55 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 19A56C000B for ; Wed, 9 Jun 2021 13:10:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 06D7060A8E for ; Wed, 9 Jun 2021 13:10:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="X85oq9ZD"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="JJahtP9Q" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id px5p-wl1wR8N for ; Wed, 9 Jun 2021 13:10:18 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp3.osuosl.org (Postfix) with ESMTPS id 6DFDA60633 for ; Wed, 9 Jun 2021 13:10:11 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id A554D1577; Wed, 9 Jun 2021 09:10:10 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 09 Jun 2021 09:10:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=tlPCM1TlkgBkV +uca4LOQEuzw0MZeVnQqvEFetCE+G8=; b=X85oq9ZDFUq0auS06PAyCYw2U0mLa /KxZtyjGPFt+9K72E1HMWmTlBnwfmEOd6832MSSml9tEOnfW2wms28rRM+uOsRnp FU7SJRaqlJC4OkV5FziBlR5KT88Tl6BzqhTI6hK6DZG7i62hKEi0s0z6183bHagF wfK+zpCXMZhSFspqhAGIWASPPYrDVdUCGgKL624A/+goxk1rpFmRVzW+YteCd925 Z0cRalE5bKSRo+P3+LqzdPJBzQI0kVglUXCeA5vHCQ9BFRBKF/c33Rw6/a0RNMMS pjiZ1uVmPM6FNVRRVktXJQKWi4XV0GvCKLqyz+G2QlAWt/Q41CsRq+DaQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=tlPCM1TlkgBkV+uca4LOQEuzw0MZeVnQqvEFetCE+G8=; b=JJahtP9Q qB+sli1WU/PD4+xJT14z3sICbTt4SpmqhZ0JJbVCRDIFgoNTFzOqtbH8VNI3AT0+ JBTvDI6ZylE/P60NCh4lMMHvT/ndV8KCDiigCs/Dn6KPT5WAXhNbW1e5tGu7+qDB 6+WM+igPF2oLbQ1bh6M7oDRrOFtd4rOK/I0BylWgzGWj5TsPC6KlxoRQNNb6yg8U L/2c635KXI+fnH0860fzPiGcTXzIl0143DQF9qjXj6DfZ36z9IWILQMon+zeUGZG UmI0/h6opLPf0oQB8BrPEw/wxyKrHlTRY/CsUsk6IOTG/wxzk+2UGF+NXNapqCa0 0Vvv2eLFTzXwjQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:09 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:24 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 16/27] dpif-netdev: Postpone flow offload item freeing X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Profiling the HW offload thread, the flow offload freeing takes approximatively 25% of the time. Most of this time is spent waiting on the futex used by the libc free(), as it triggers a syscall and reschedule the thread. Avoid the syscall and its expensive context switch. Batch the offload messages freeing using the RCU. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index e403e461a..75b289904 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2625,14 +2625,19 @@ dp_netdev_alloc_flow_offload(struct dp_netdev_pmd_thread *pmd, return offload; } +static void +dp_netdev_free_flow_offload__(struct dp_offload_thread_item *offload) +{ + free(offload->actions); + free(offload); +} + static void dp_netdev_free_flow_offload(struct dp_offload_thread_item *offload) { dp_netdev_pmd_unref(offload->pmd); dp_netdev_flow_unref(offload->flow); - - free(offload->actions); - free(offload); + ovsrcu_postpone(dp_netdev_free_flow_offload__, offload); } static void From patchwork Wed Jun 9 13:09:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489876 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=HheDehSV; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=hJ9PZiQ0; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SDd3MkJz9sSs for ; Wed, 9 Jun 2021 23:11:45 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id AC7DF6F798; Wed, 9 Jun 2021 13:11:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TcJtrT8npNNK; Wed, 9 Jun 2021 13:11:40 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4B9CA60D64; Wed, 9 Jun 2021 13:11:27 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1940BC000F; Wed, 9 Jun 2021 13:11:27 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 28275C0024 for ; Wed, 9 Jun 2021 13:11:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id BA20A83D56 for ; Wed, 9 Jun 2021 13:10:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="HheDehSV"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="hJ9PZiQ0" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y4WTmq03y2Hb for ; Wed, 9 Jun 2021 13:10:25 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1C60683CD2 for ; Wed, 9 Jun 2021 13:10:13 +0000 (UTC) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 56C041609; Wed, 9 Jun 2021 09:10:12 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Wed, 09 Jun 2021 09:10:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=LF5H3eV0Yw1KX 179cgYoU0s/IPznI1QMMMikh7uVUVs=; b=HheDehSVRNfLlYYxSVvqInL8jMscY lAGd86b4KAcnSHpw9KWnjFgOcwiGajAKWD/jezN/zGZpkvFNF0lydw4kihC8gdaK 2cKiAw3yADuLd5ilGOEg40XZhHQKV80kRh3knJM9+gccCJtICWkjGw12+gyYPzqP L8jshnCNj6x8x9TWpQ/dgouj6w4+Wq+vlSBTRGr8pEZjFF+N8w3rhFbtv8ENrDDz psAssgg2gJ1NG1sZBZGFqjxQZmQU0lrXMKlZKaqH5SzcMP9MjApUPyTrzgqz0FuO SyRLzB4dNyttxDHlBGTA67D4A+gkaMSk5orHvMF41p/DvrcSKs4SLdyzg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=LF5H3eV0Yw1KX179cgYoU0s/IPznI1QMMMikh7uVUVs=; b=hJ9PZiQ0 7/5PivOwQL+TBP3VUzxJBTQKBPaod5AhMMZ8FPHulY4bi79YpifIN4TysY/1tjw5 2PGc5m/6q/wrFRI1CT2+oTl4qxwuQ7tc5RF+XZ6D5wti0+LFlBQbVn0+YaEjV9r2 hz2TPeU1ToobjWnsWreUQ81JHprEC7zisQDaaZMGaKXiqY1RAtyXmjhh7dS6GsdK h5FszQq6P933KirNOiqhaxe7Qtc8IKE5OqWR0CjUvRsieNJUUuRZQmonVb8MKRu/ aiSf9nrzn+OCkCcfqf2fFOQY5JzCpYSJGLsIjaQphBK2TCQ+6/QPBBzDlDFQDwHL sHeHrUP3uUrDVQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:11 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:25 +0200 Message-Id: <80bc7bbb86a0002b254f169c77fe04c1adf233d8.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 17/27] dpif-netdev: Use id-fpool for mark allocation X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Use the netdev-offload multithread API to allow multiple thread allocating marks concurrently. Initialize only once the pool in a multithread context by using the ovsthread_once type. Use the id-fpool module for faster concurrent ID allocation. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 75b289904..b8fd49f5d 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -49,6 +49,7 @@ #include "fat-rwlock.h" #include "flow.h" #include "hmapx.h" +#include "id-fpool.h" #include "id-pool.h" #include "ipf.h" #include "mov-avg.h" @@ -2418,7 +2419,7 @@ struct megaflow_to_mark_data { struct flow_mark { struct cmap megaflow_to_mark; struct cmap mark_to_flow; - struct id_pool *pool; + struct id_fpool *pool; }; static struct flow_mark flow_mark = { @@ -2429,14 +2430,18 @@ static struct flow_mark flow_mark = { static uint32_t flow_mark_alloc(void) { + static struct ovsthread_once pool_init = OVSTHREAD_ONCE_INITIALIZER; + unsigned int tid = netdev_offload_thread_id(); uint32_t mark; - if (!flow_mark.pool) { + if (ovsthread_once_start(&pool_init)) { /* Haven't initiated yet, do it here */ - flow_mark.pool = id_pool_create(1, MAX_FLOW_MARK); + flow_mark.pool = id_fpool_create(netdev_offload_thread_nb(), + 1, MAX_FLOW_MARK); + ovsthread_once_done(&pool_init); } - if (id_pool_alloc_id(flow_mark.pool, &mark)) { + if (id_fpool_new_id(flow_mark.pool, tid, &mark)) { return mark; } @@ -2446,7 +2451,9 @@ flow_mark_alloc(void) static void flow_mark_free(uint32_t mark) { - id_pool_free_id(flow_mark.pool, mark); + unsigned int tid = netdev_offload_thread_id(); + + id_fpool_free_id(flow_mark.pool, tid, mark); } /* associate megaflow with a mark, which is a 1:1 mapping */ From patchwork Wed Jun 9 13:09:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489874 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=g/QxWWrY; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=mcSkMLPH; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SDC0HRnz9sSn for ; Wed, 9 Jun 2021 23:11:22 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 48C9E405BC; Wed, 9 Jun 2021 13:11:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZUDFh5rIWVWh; Wed, 9 Jun 2021 13:11:18 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 38D48405CA; Wed, 9 Jun 2021 13:11:17 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0FA08C000F; Wed, 9 Jun 2021 13:11:17 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2BF64C0024 for ; Wed, 9 Jun 2021 13:11:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 9487D405B5 for ; Wed, 9 Jun 2021 13:10:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fwWcLhJTgwlS for ; Wed, 9 Jun 2021 13:10:17 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp4.osuosl.org (Postfix) with ESMTPS id 989C1404A5 for ; Wed, 9 Jun 2021 13:10:14 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id CA1E11971; Wed, 9 Jun 2021 09:10:13 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=KgyI37vvgyeAf fFsZUudo/Jq1CFJ3YyAfCHQSdBa2vk=; b=g/QxWWrYijUD6stoaf+Wfb16Q2/hr StxTLef2GRS63/NHl96406sLHV+w3qcOoeKzozz6qUDAt2cIM8VTH+KytQIYJQeu nQ2c9GW0Erfu7JanIGhnm1u+2V+j9uBFYe2GkpxTrA0pGMvQSyG1gIPaXXmWqqvN 6EeGiprZnYtdIJ6VHewMlx4WZidFwBCiSWQNVDtirQn62d3St7rkQM1J0VYKyagN SAvYOogC9Knj/YhN1pt6zdUnq/HxaV3PO3lYSiPa2SHnWekW9DMz8rsUZFG3Trd8 kcAiN03sA8IPZXkZJZfjMg8C6sm4DOcO1zNiIN00o4uJgxVxJ/1k18BLQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=KgyI37vvgyeAffFsZUudo/Jq1CFJ3YyAfCHQSdBa2vk=; b=mcSkMLPH jHF6UiN94dXCLRkrF6umGUIUIHh1Fo6Y2hFPkiMWcc/i1nYjqDglSY51sCARi1Yg r5tlpvNYl59HfZHL2OCsMdXF5wPPstkH6L0l/e1mPhjVmo0rqI/NidQXQQFkwQsa cfSoL0co4l2BY+iVl31UKDptWkVPOzgc5cRR++YrxjwHLwzMl02pSEklAjTdXn2D tORApciPjtX1u7Lde7F4lAlC1P9BTQN2LXgoaLWKUDT3GToVZOOTZ4gkQnX2dXqt SOV9BvHYAaiOavchntkhAu8Q23WQV6coxOgLsW5VYIqo7xyjpIGgBgq+YxJOfPDb QgTqXhxFEWyksA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:12 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:26 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Maxime Coquelin Subject: [ovs-dev] [PATCH v4 18/27] dpif-netdev: Introduce tagged union of offload requests X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Offload requests are currently only supporting flow offloads. As a pre-step before supporting an offload flush request, modify the layout of an offload request item, to become a tagged union. Future offload types won't be forced to re-use the full flow offload structure, which consumes a lot of memory. Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 128 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 89 insertions(+), 39 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index b8fd49f5d..1d7e55d47 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -420,22 +420,34 @@ enum rxq_cycles_counter_type { RXQ_N_CYCLES }; +enum dp_offload_type { + DP_OFFLOAD_FLOW, +}; + enum { DP_NETDEV_FLOW_OFFLOAD_OP_ADD, DP_NETDEV_FLOW_OFFLOAD_OP_MOD, DP_NETDEV_FLOW_OFFLOAD_OP_DEL, }; -struct dp_offload_thread_item { +struct dp_offload_flow_item { struct dp_netdev_pmd_thread *pmd; struct dp_netdev_flow *flow; int op; struct match match; struct nlattr *actions; size_t actions_len; - long long int timestamp; +}; +union dp_offload_thread_data { + struct dp_offload_flow_item flow; +}; + +struct dp_offload_thread_item { struct ovs_list node; + enum dp_offload_type type; + long long int timestamp; + union dp_offload_thread_data data[0]; }; struct dp_offload_thread { @@ -2619,34 +2631,55 @@ dp_netdev_alloc_flow_offload(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow, int op) { - struct dp_offload_thread_item *offload; + struct dp_offload_thread_item *item; + struct dp_offload_flow_item *flow_offload; + + item = xzalloc(sizeof *item + sizeof *flow_offload); + flow_offload = &item->data->flow; - offload = xzalloc(sizeof(*offload)); - offload->pmd = pmd; - offload->flow = flow; - offload->op = op; + item->type = DP_OFFLOAD_FLOW; + + flow_offload->pmd = pmd; + flow_offload->flow = flow; + flow_offload->op = op; dp_netdev_flow_ref(flow); dp_netdev_pmd_try_ref(pmd); - return offload; + return item; } static void dp_netdev_free_flow_offload__(struct dp_offload_thread_item *offload) { - free(offload->actions); + struct dp_offload_flow_item *flow_offload = &offload->data->flow; + + free(flow_offload->actions); free(offload); } static void dp_netdev_free_flow_offload(struct dp_offload_thread_item *offload) { - dp_netdev_pmd_unref(offload->pmd); - dp_netdev_flow_unref(offload->flow); + struct dp_offload_flow_item *flow_offload = &offload->data->flow; + + dp_netdev_pmd_unref(flow_offload->pmd); + dp_netdev_flow_unref(flow_offload->flow); ovsrcu_postpone(dp_netdev_free_flow_offload__, offload); } +static void +dp_netdev_free_offload(struct dp_offload_thread_item *offload) +{ + switch (offload->type) { + case DP_OFFLOAD_FLOW: + dp_netdev_free_flow_offload(offload); + break; + default: + OVS_NOT_REACHED(); + }; +} + static void dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) { @@ -2658,7 +2691,7 @@ dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) } static int -dp_netdev_flow_offload_del(struct dp_offload_thread_item *offload) +dp_netdev_flow_offload_del(struct dp_offload_flow_item *offload) { return mark_to_flow_disassociate(offload->pmd, offload->flow); } @@ -2675,7 +2708,7 @@ dp_netdev_flow_offload_del(struct dp_offload_thread_item *offload) * valid, thus only item 2 needed. */ static int -dp_netdev_flow_offload_put(struct dp_offload_thread_item *offload) +dp_netdev_flow_offload_put(struct dp_offload_flow_item *offload) { struct dp_netdev_pmd_thread *pmd = offload->pmd; struct dp_netdev_flow *flow = offload->flow; @@ -2752,6 +2785,35 @@ err_free: return -1; } +static void +dp_offload_flow(struct dp_offload_thread_item *item) +{ + struct dp_offload_flow_item *flow_offload = &item->data->flow; + const char *op; + int ret; + + switch (flow_offload->op) { + case DP_NETDEV_FLOW_OFFLOAD_OP_ADD: + op = "add"; + ret = dp_netdev_flow_offload_put(flow_offload); + break; + case DP_NETDEV_FLOW_OFFLOAD_OP_MOD: + op = "modify"; + ret = dp_netdev_flow_offload_put(flow_offload); + break; + case DP_NETDEV_FLOW_OFFLOAD_OP_DEL: + op = "delete"; + ret = dp_netdev_flow_offload_del(flow_offload); + break; + default: + OVS_NOT_REACHED(); + } + + VLOG_DBG("%s to %s netdev flow "UUID_FMT, + ret == 0 ? "succeed" : "failed", op, + UUID_ARGS((struct uuid *) &flow_offload->flow->mega_ufid)); +} + #define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ static void * @@ -2762,8 +2824,6 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) long long int latency_us; long long int next_rcu; long long int now; - const char *op; - int ret; next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; for (;;) { @@ -2780,18 +2840,9 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); ovs_mutex_unlock(&dp_offload_thread.mutex); - switch (offload->op) { - case DP_NETDEV_FLOW_OFFLOAD_OP_ADD: - op = "add"; - ret = dp_netdev_flow_offload_put(offload); - break; - case DP_NETDEV_FLOW_OFFLOAD_OP_MOD: - op = "modify"; - ret = dp_netdev_flow_offload_put(offload); - break; - case DP_NETDEV_FLOW_OFFLOAD_OP_DEL: - op = "delete"; - ret = dp_netdev_flow_offload_del(offload); + switch (offload->type) { + case DP_OFFLOAD_FLOW: + dp_offload_flow(offload); break; default: OVS_NOT_REACHED(); @@ -2803,10 +2854,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) mov_avg_cma_update(&dp_offload_thread.cma, latency_us); mov_avg_ema_update(&dp_offload_thread.ema, latency_us); - VLOG_DBG("%s to %s netdev flow "UUID_FMT, - ret == 0 ? "succeed" : "failed", op, - UUID_ARGS((struct uuid *) &offload->flow->mega_ufid)); - dp_netdev_free_flow_offload(offload); + dp_netdev_free_offload(offload); /* Do RCU synchronization at fixed interval. */ if (now > next_rcu) { @@ -2841,7 +2889,8 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow, struct match *match, const struct nlattr *actions, size_t actions_len) { - struct dp_offload_thread_item *offload; + struct dp_offload_thread_item *item; + struct dp_offload_flow_item *flow_offload; int op; if (!netdev_is_flow_api_enabled()) { @@ -2859,14 +2908,15 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, } else { op = DP_NETDEV_FLOW_OFFLOAD_OP_ADD; } - offload = dp_netdev_alloc_flow_offload(pmd, flow, op); - offload->match = *match; - offload->actions = xmalloc(actions_len); - memcpy(offload->actions, actions, actions_len); - offload->actions_len = actions_len; + item = dp_netdev_alloc_flow_offload(pmd, flow, op); + flow_offload = &item->data->flow; + flow_offload->match = *match; + flow_offload->actions = xmalloc(actions_len); + memcpy(flow_offload->actions, actions, actions_len); + flow_offload->actions_len = actions_len; - offload->timestamp = pmd->ctx.now; - dp_netdev_append_flow_offload(offload); + item->timestamp = pmd->ctx.now; + dp_netdev_append_flow_offload(item); } static void From patchwork Wed Jun 9 13:09:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489875 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=SzXyIYHB; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=dR4kS1iu; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SDW2hsvz9sSn for ; Wed, 9 Jun 2021 23:11:39 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7DF9840575; Wed, 9 Jun 2021 13:11:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 95XnQXR3k4In; Wed, 9 Jun 2021 13:11:33 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 22C5E40641; Wed, 9 Jun 2021 13:11:32 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D693EC000D; Wed, 9 Jun 2021 13:11:31 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1AB58C000D for ; Wed, 9 Jun 2021 13:11:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 85D9483D3C for ; Wed, 9 Jun 2021 13:10:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="SzXyIYHB"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="dR4kS1iu" Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y0CUImA2jouO for ; Wed, 9 Jun 2021 13:10:29 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1BCE983CBA for ; Wed, 9 Jun 2021 13:10:15 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 54A01210E; Wed, 9 Jun 2021 09:10:15 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 09 Jun 2021 09:10:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=+bAJMRnSatumX 5Gn7Iqip09REuuixH0SWPMMESizR74=; b=SzXyIYHBsE6Jo0KCbtf19XB+QFbwM klYP4sT4vkWZlm7GmjCHT91Omrfegse3HEj+Y7sTSb8o/08bbzbwyyXo5tijFOTr XY1RAWht8gzcIoYYA/awpmg5eJzZpJbckMDwAI+7Qko5YitQwbSYsZi2JTrFHpMn 9GaNXHLUsIPCQBKymCnnUHRMWXXqmo7f9T7Tt3V3sSu98wePbpJYoKanrk/ZI5Vv v6K0vtz2KGJkZnxlgO5riEodG6g3IkKFDsCotQVR/e0cliARv1jLiepHzzIfQ+HR JCf10p6/uztfaFg52+utr5oQ3Uq8o0liEt0ptfV52v2dBpklIMSVTECJw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=+bAJMRnSatumX5Gn7Iqip09REuuixH0SWPMMESizR74=; b=dR4kS1iu U2FWnuqSTUf2WqaF9dFbwXHyVPW02mNb1b5i3ZIc8518uQDpnJ8VbIMc1MZb1rRQ 3l6aZi59QhepFBG7y4B3ujHVpV+eAZNn90S/QxmD5zE71DudjkULcny84rpU262a jyVgHKXiGY1ZbXfrF0jOVLbcbYNAU4HtTdo8LiuZEjJdHNa/Oz5K1OtjVia9BlnI q00Sk2c/WJhvEbtIdVcPmrqRWpt0JTsvGPpcp0tNP5As8Afk6X/OIZjDGBkFV9BN ZLn72j8xyab80dZZL9qpCo9DO5d27FVA0jUgilvpkFW+Q/GlfibZ3T2DH9vsA2aT AJxZO4vQNqMBQw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:14 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:27 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Maxime Coquelin Subject: [ovs-dev] [PATCH v4 19/27] dpif-netdev: Execute flush from offload thread X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When a port is deleted, its offloads must be flushed. The operation runs in the thread that initiated it. Offload data is thus accessed jointly by the port deletion thread(s) and the offload thread, which complicates the data access model. To simplify this model, as a pre-step toward introducing parallel offloads, execute the flush operation in the offload thread. Signed-off-by: Gaetan Rivet Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 126 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 122 insertions(+), 4 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 1d7e55d47..82e55e60b 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -422,6 +422,7 @@ enum rxq_cycles_counter_type { enum dp_offload_type { DP_OFFLOAD_FLOW, + DP_OFFLOAD_FLUSH, }; enum { @@ -439,8 +440,15 @@ struct dp_offload_flow_item { size_t actions_len; }; +struct dp_offload_flush_item { + struct dp_netdev *dp; + struct netdev *netdev; + struct ovs_barrier *barrier; +}; + union dp_offload_thread_data { struct dp_offload_flow_item flow; + struct dp_offload_flush_item flush; }; struct dp_offload_thread_item { @@ -905,6 +913,9 @@ static void dp_netdev_del_bond_tx_from_pmd(struct dp_netdev_pmd_thread *pmd, uint32_t bond_id) OVS_EXCLUDED(pmd->bond_mutex); +static void dp_netdev_offload_flush(struct dp_netdev *dp, + struct dp_netdev_port *port); + static void reconfigure_datapath(struct dp_netdev *dp) OVS_REQUIRES(dp->port_mutex); static bool dp_netdev_pmd_try_ref(struct dp_netdev_pmd_thread *pmd); @@ -2305,7 +2316,7 @@ static void do_del_port(struct dp_netdev *dp, struct dp_netdev_port *port) OVS_REQUIRES(dp->port_mutex) { - netdev_flow_flush(port->netdev); + dp_netdev_offload_flush(dp, port); netdev_uninit_flow_api(port->netdev); hmap_remove(&dp->ports, &port->node); seq_change(dp->port_seq); @@ -2675,13 +2686,16 @@ dp_netdev_free_offload(struct dp_offload_thread_item *offload) case DP_OFFLOAD_FLOW: dp_netdev_free_flow_offload(offload); break; + case DP_OFFLOAD_FLUSH: + free(offload); + break; default: OVS_NOT_REACHED(); }; } static void -dp_netdev_append_flow_offload(struct dp_offload_thread_item *offload) +dp_netdev_append_offload(struct dp_offload_thread_item *offload) { ovs_mutex_lock(&dp_offload_thread.mutex); ovs_list_push_back(&dp_offload_thread.list, &offload->node); @@ -2814,6 +2828,23 @@ dp_offload_flow(struct dp_offload_thread_item *item) UUID_ARGS((struct uuid *) &flow_offload->flow->mega_ufid)); } +static void +dp_offload_flush(struct dp_offload_thread_item *item) +{ + struct dp_offload_flush_item *flush = &item->data->flush; + + ovs_mutex_lock(&flush->dp->port_mutex); + netdev_flow_flush(flush->netdev); + ovs_mutex_unlock(&flush->dp->port_mutex); + + ovs_barrier_block(flush->barrier); + + /* Allow the other thread to take again the port lock, before + * continuing offload operations in this thread. + */ + ovs_barrier_block(flush->barrier); +} + #define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ static void * @@ -2844,6 +2875,9 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) case DP_OFFLOAD_FLOW: dp_offload_flow(offload); break; + case DP_OFFLOAD_FLUSH: + dp_offload_flush(offload); + break; default: OVS_NOT_REACHED(); } @@ -2881,7 +2915,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, offload = dp_netdev_alloc_flow_offload(pmd, flow, DP_NETDEV_FLOW_OFFLOAD_OP_DEL); offload->timestamp = pmd->ctx.now; - dp_netdev_append_flow_offload(offload); + dp_netdev_append_offload(offload); } static void @@ -2916,7 +2950,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, flow_offload->actions_len = actions_len; item->timestamp = pmd->ctx.now; - dp_netdev_append_flow_offload(item); + dp_netdev_append_offload(item); } static void @@ -2940,6 +2974,90 @@ dp_netdev_pmd_remove_flow(struct dp_netdev_pmd_thread *pmd, dp_netdev_flow_unref(flow); } +static void +dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, + struct netdev *netdev, + struct ovs_barrier *barrier) +{ + struct dp_offload_thread_item *item; + struct dp_offload_flush_item *flush; + + if (ovsthread_once_start(&offload_thread_once)) { + xpthread_cond_init(&dp_offload_thread.cond, NULL); + ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); + ovsthread_once_done(&offload_thread_once); + } + + item = xmalloc(sizeof *item + sizeof *flush); + item->type = DP_OFFLOAD_FLUSH; + item->timestamp = time_usec(); + + flush = &item->data->flush; + flush->dp = dp; + flush->netdev = netdev; + flush->barrier = barrier; + + dp_netdev_append_offload(item); +} + +/* Blocking call that will wait on the offload thread to + * complete its work. As the flush order will only be + * enqueued after existing offload requests, those previous + * offload requests must be processed, which requires being + * able to lock the 'port_mutex' from the offload thread. + * + * Flow offload flush is done when a port is being deleted. + * Right after this call executes, the offload API is disabled + * for the port. This call must be made blocking until the + * offload provider completed its job. + */ +static void +dp_netdev_offload_flush(struct dp_netdev *dp, + struct dp_netdev_port *port) + OVS_REQUIRES(dp->port_mutex) +{ + /* The flush mutex only serves to protect the static memory barrier. + * The memory barrier needs to go beyond the function scope as + * the other thread can resume from blocking after this function + * already finished. + * As the barrier is made static, then it will be shared by + * calls to this function, and it needs to be protected from + * concurrent use. + */ + static struct ovs_mutex flush_mutex = OVS_MUTEX_INITIALIZER; + static struct ovs_barrier barrier OVS_GUARDED_BY(flush_mutex); + struct netdev *netdev; + + if (!netdev_is_flow_api_enabled()) { + return; + } + + ovs_mutex_unlock(&dp->port_mutex); + ovs_mutex_lock(&flush_mutex); + + /* This thread and the offload thread. */ + ovs_barrier_init(&barrier, 2); + + netdev = netdev_ref(port->netdev); + dp_netdev_offload_flush_enqueue(dp, netdev, &barrier); + ovs_barrier_block(&barrier); + netdev_close(netdev); + + /* Take back the datapath port lock before allowing the offload + * thread to proceed further. The port deletion must complete first, + * to ensure no further offloads are inserted after the flush. + * + * Some offload provider (e.g. DPDK) keeps a netdev reference with + * the offload data. If this reference is not closed, the netdev is + * kept indefinitely. */ + ovs_mutex_lock(&dp->port_mutex); + + ovs_barrier_block(&barrier); + ovs_barrier_destroy(&barrier); + + ovs_mutex_unlock(&flush_mutex); +} + static void dp_netdev_pmd_flow_flush(struct dp_netdev_pmd_thread *pmd) { From patchwork Wed Jun 9 13:09:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489877 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=rgMfsazo; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=siVpp9LE; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SDm1t8xz9sWH for ; Wed, 9 Jun 2021 23:11:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id BEAA341D90; Wed, 9 Jun 2021 13:11:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TnItF538Cnha; Wed, 9 Jun 2021 13:11:46 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1B1E541D75; Wed, 9 Jun 2021 13:11:45 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id D2BD4C000D; Wed, 9 Jun 2021 13:11:44 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 19848C000B for ; Wed, 9 Jun 2021 13:11:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 53E5941CA1 for ; Wed, 9 Jun 2021 13:10:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6K_tA9srXH8p for ; Wed, 9 Jun 2021 13:10:25 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id C889440599 for ; Wed, 9 Jun 2021 13:10:17 +0000 (UTC) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id F1E1024D5; Wed, 9 Jun 2021 09:10:16 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 09 Jun 2021 09:10:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=89XG27xZ7NeFZ OOo89I3IwXxQ4Kzoj8p7COgtGlnXyc=; b=rgMfsazocfkW959eml27c46Y+mxjx Kp/QU/ejFfdZeX+zNm4uti64rM+uc+qItoEotysuvIWx3MQQf73lbyRpzIwateSH m0MF4uZOGXc5P0R9gNuey4PGZvE2olx5bJtTu8GxCbVfVJB2gaZXrS7eAbkBrcsJ 96xQGs2EVuauNUT+/bC4FhFajcPJF1tlkn3HKFAdg5eBSLQkRW2LC6CzVbOLqXUB 0Vnf7dDBrAFAHYIPeZLmltfGyZiie1uOqsd7Qj09IVG2H5NTZ8OLn+cnfu4dtKiy dOTT7Zcp+3QKA9kUNYo/qYD+fPJ6bzujv2OyZvLx38xZTVd5UitquXECA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=89XG27xZ7NeFZOOo89I3IwXxQ4Kzoj8p7COgtGlnXyc=; b=siVpp9LE ftK95Km/2m25BOwedbWcpPoNDXs77edSixmeMwmHMLFeh+9xl3uMDRM4AZ3rQvcw gb9pzFMiRJIWkcKLWjrYQxkIYPOsRPKeHgM4SeIUnRBf89qGU0vNPiuy3I30SMPn zsaWgY+LMt49aTt9GH/k1QvYOCMDm3Mmv3smBK17JVDGco1z0vUEGfEXKiNBrPJx PSt2LMU6EBhxK3FkPnRDWsMlQkuPkEMF+Dr0JWjF2tljgqimHUU/CqjoqQw1tJXD f6iFY8XoRgUj+XnTA+O1m4BGEgQaC9nY3Mv6NBugtdaKqs5qr/7Xa39kffslwuDY VuckurDwpIKgJw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:15 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:28 +0200 Message-Id: <748b89db825812497f8939393f8765a7c99866cd.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 20/27] netdev-offload-dpdk: Use per-thread HW offload stats X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The implementation of hardware offload counters in currently meant to be managed by a single thread. Use the offload thread pool API to manage one counter per thread. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-dpdk.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index c43e8b968..ecdc846e1 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -65,7 +65,7 @@ struct ufid_to_rte_flow_data { struct netdev_offload_dpdk_data { struct cmap ufid_to_rte_flow; - uint64_t rte_flow_counter; + uint64_t *rte_flow_counters; }; static int @@ -75,6 +75,8 @@ offload_data_init(struct netdev *netdev) data = xzalloc(sizeof *data); cmap_init(&data->ufid_to_rte_flow); + data->rte_flow_counters = xcalloc(netdev_offload_thread_nb(), + sizeof *data->rte_flow_counters); ovsrcu_set(&netdev->hw_info.offload_data, (void *) data); @@ -84,6 +86,7 @@ offload_data_init(struct netdev *netdev) static void offload_data_destroy__(struct netdev_offload_dpdk_data *data) { + free(data->rte_flow_counters); free(data); } @@ -646,10 +649,11 @@ netdev_offload_dpdk_flow_create(struct netdev *netdev, flow = netdev_dpdk_rte_flow_create(netdev, attr, items, actions, error); if (flow) { struct netdev_offload_dpdk_data *data; + unsigned int tid = netdev_offload_thread_id(); data = (struct netdev_offload_dpdk_data *) ovsrcu_get(void *, &netdev->hw_info.offload_data); - data->rte_flow_counter++; + data->rte_flow_counters[tid]++; if (!VLOG_DROP_DBG(&rl)) { dump_flow(&s, &s_extra, attr, items, actions); @@ -1532,10 +1536,11 @@ netdev_offload_dpdk_flow_destroy(struct ufid_to_rte_flow_data *rte_flow_data) if (ret == 0) { struct netdev_offload_dpdk_data *data; + unsigned int tid = netdev_offload_thread_id(); data = (struct netdev_offload_dpdk_data *) ovsrcu_get(void *, &netdev->hw_info.offload_data); - data->rte_flow_counter--; + data->rte_flow_counters[tid]--; ufid_to_rte_flow_disassociate(rte_flow_data); VLOG_DBG_RL(&rl, "%s: rte_flow 0x%"PRIxPTR @@ -1698,6 +1703,7 @@ netdev_offload_dpdk_get_n_flows(struct netdev *netdev, uint64_t *n_flows) { struct netdev_offload_dpdk_data *data; + unsigned int tid; data = (struct netdev_offload_dpdk_data *) ovsrcu_get(void *, &netdev->hw_info.offload_data); @@ -1705,7 +1711,9 @@ netdev_offload_dpdk_get_n_flows(struct netdev *netdev, return -1; } - *n_flows = data->rte_flow_counter; + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + n_flows[tid] = data->rte_flow_counters[tid]; + } return 0; } From patchwork Wed Jun 9 13:09:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489878 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=SAphuM9+; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=KQ8n8rEc; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SDw6368z9sW8 for ; Wed, 9 Jun 2021 23:12:00 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 237C660BBF; Wed, 9 Jun 2021 13:11:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HZCUDOIGm0Lk; Wed, 9 Jun 2021 13:11:55 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 198206F686; Wed, 9 Jun 2021 13:11:51 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C5D88C000D; Wed, 9 Jun 2021 13:11:50 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 11790C000D for ; Wed, 9 Jun 2021 13:11:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id B9F3A40687 for ; Wed, 9 Jun 2021 13:10:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="SAphuM9+"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="KQ8n8rEc" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vM6PVJH9ueIj for ; Wed, 9 Jun 2021 13:10:27 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2C335403D8 for ; Wed, 9 Jun 2021 13:10:19 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 97F9E2601; Wed, 9 Jun 2021 09:10:18 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=ZyH8IMveGFsx9 iKebYiiQOEGHKpDoeqjjuKGkCncT+c=; b=SAphuM9+N4p0PwkwxZWgTAsRSTWJG TnPQjXTF6g4hglnXcCIcF/yr68VXfJjDDGq7Cd4+lTugVY2CgJBol56b4qXxGvBU AQeaFLbTMktf4Z6kHanZmpgN0co4/qXB4UOH1ZQ+aBky03xp92YA7TIe6grysx0N rpfU2FafX/gvDpcP2MZIud1Bxct23hOfXoaMnTOMtzfwoqrmNVBwNQ2p3jE0qO3A iXLUan34qwFFOsjO7iEK9RKGIALoakp7RLQrPM3/TSZr2XIlhBT+AuWlGU2pAUr2 QD0QVSlDKOkHMC7AxNm7rFZRfxR7ZSq9Yz3Oy+Vua7d40ubHamSLcMZGQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=ZyH8IMveGFsx9iKebYiiQOEGHKpDoeqjjuKGkCncT+c=; b=KQ8n8rEc EB72LMnig3m8KpEHmmmfRGPDK5+AQoYDqUg5gEQdsUeEWwfsRQyAmj9E/Ql6QGB1 LrS6if3a+dKWaR4Qua0VZIESMR4kjlgXOjcbLdg6i7As8qJoHkHtnmiDwhbcCOeH DSkPeL5icvqfbWBLb9q6b8kiAXOlaROJmPPHPzDM7k7FHqp89NYmrsgNxDbUJk8l gN1qtr0tihrCmAVZCJv78ZUmovy2BVFjPJN3f1F3AFH0M70Lli2ytknpaxEvMhLF vODNuXczkSUH1/GN50Gjk+cgxFvmwh0EqTY18jhrOhjHEM3WxWorSj8SE4v1w2lW HcTL0iWBGlyzEg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepgeenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:17 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:29 +0200 Message-Id: <2c1e273f07119d244993fe1d3d971d8f2c01b312.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 21/27] netdev-offload-dpdk: Lock rte_flow map access X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a lock to access the ufid to rte_flow map. This will protect it from concurrent write accesses when multiple threads attempt it. At this point, the reason for taking the lock is not to fullfill the needs of the DPDK offload implementation anymore. Rewrite the comments to reflect this change. The lock is still needed to protect against changes to netdev port mapping. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 8 ++--- lib/netdev-offload-dpdk.c | 61 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 61 insertions(+), 8 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 82e55e60b..1daaecb1c 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -2590,7 +2590,7 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, port = netdev_ports_get(in_port, dpif_type_str); if (port) { /* Taking a global 'port_mutex' to fulfill thread safety - * restrictions for the netdev-offload-dpdk module. */ + * restrictions regarding netdev port mapping. */ ovs_mutex_lock(&pmd->dp->port_mutex); ret = netdev_flow_del(port, &flow->mega_ufid, NULL); ovs_mutex_unlock(&pmd->dp->port_mutex); @@ -2770,8 +2770,8 @@ dp_netdev_flow_offload_put(struct dp_offload_flow_item *offload) netdev_close(port); goto err_free; } - /* Taking a global 'port_mutex' to fulfill thread safety restrictions for - * the netdev-offload-dpdk module. */ + /* Taking a global 'port_mutex' to fulfill thread safety + * restrictions regarding the netdev port mapping. */ ovs_mutex_lock(&pmd->dp->port_mutex); ret = netdev_flow_put(port, &offload->match, CONST_CAST(struct nlattr *, offload->actions), @@ -3573,7 +3573,7 @@ dpif_netdev_get_flow_offload_status(const struct dp_netdev *dp, } ofpbuf_use_stack(&buf, &act_buf, sizeof act_buf); /* Taking a global 'port_mutex' to fulfill thread safety - * restrictions for the netdev-offload-dpdk module. + * restrictions regarding netdev port mapping. * * XXX: Main thread will try to pause/stop all revalidators during datapath * reconfiguration via datapath purge callback (dp_purge_cb) while diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index ecdc846e1..4459a0aa1 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -38,9 +38,6 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(100, 5); * * Below API is NOT thread safe in following terms: * - * - The caller must be sure that none of these functions will be called - * simultaneously. Even for different 'netdev's. - * * - The caller must be sure that 'netdev' will not be destructed/deallocated. * * - The caller must be sure that 'netdev' configuration will not be changed. @@ -66,6 +63,7 @@ struct ufid_to_rte_flow_data { struct netdev_offload_dpdk_data { struct cmap ufid_to_rte_flow; uint64_t *rte_flow_counters; + struct ovs_mutex map_lock; }; static int @@ -74,6 +72,7 @@ offload_data_init(struct netdev *netdev) struct netdev_offload_dpdk_data *data; data = xzalloc(sizeof *data); + ovs_mutex_init(&data->map_lock); cmap_init(&data->ufid_to_rte_flow); data->rte_flow_counters = xcalloc(netdev_offload_thread_nb(), sizeof *data->rte_flow_counters); @@ -86,6 +85,7 @@ offload_data_init(struct netdev *netdev) static void offload_data_destroy__(struct netdev_offload_dpdk_data *data) { + ovs_mutex_destroy(&data->map_lock); free(data->rte_flow_counters); free(data); } @@ -117,6 +117,34 @@ offload_data_destroy(struct netdev *netdev) ovsrcu_set(&netdev->hw_info.offload_data, NULL); } +static void +offload_data_lock(struct netdev *netdev) + OVS_NO_THREAD_SAFETY_ANALYSIS +{ + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + if (!data) { + return; + } + ovs_mutex_lock(&data->map_lock); +} + +static void +offload_data_unlock(struct netdev *netdev) + OVS_NO_THREAD_SAFETY_ANALYSIS +{ + struct netdev_offload_dpdk_data *data; + + data = (struct netdev_offload_dpdk_data *) + ovsrcu_get(void *, &netdev->hw_info.offload_data); + if (!data) { + return; + } + ovs_mutex_unlock(&data->map_lock); +} + static struct cmap * offload_data_map(struct netdev *netdev) { @@ -155,6 +183,24 @@ ufid_to_rte_flow_data_find(struct netdev *netdev, return NULL; } +/* Find rte_flow with @ufid, lock-protected. */ +static struct ufid_to_rte_flow_data * +ufid_to_rte_flow_data_find_protected(struct netdev *netdev, + const ovs_u128 *ufid) +{ + size_t hash = hash_bytes(ufid, sizeof *ufid, 0); + struct ufid_to_rte_flow_data *data; + struct cmap *map = offload_data_map(netdev); + + CMAP_FOR_EACH_WITH_HASH_PROTECTED (data, node, hash, map) { + if (ovs_u128_equals(*ufid, data->ufid)) { + return data; + } + } + + return NULL; +} + static inline struct ufid_to_rte_flow_data * ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, struct rte_flow *rte_flow, bool actions_offloaded) @@ -170,13 +216,15 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, data = xzalloc(sizeof *data); + offload_data_lock(netdev); + /* * We should not simply overwrite an existing rte flow. * We should have deleted it first before re-adding it. * Thus, if following assert triggers, something is wrong: * the rte_flow is not destroyed. */ - data_prev = ufid_to_rte_flow_data_find(netdev, ufid, false); + data_prev = ufid_to_rte_flow_data_find_protected(netdev, ufid); if (data_prev) { ovs_assert(data_prev->rte_flow == NULL); } @@ -187,6 +235,8 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, data->actions_offloaded = actions_offloaded; cmap_insert(map, CONST_CAST(struct cmap_node *, &data->node), hash); + + offload_data_unlock(netdev); return data; } @@ -200,7 +250,10 @@ ufid_to_rte_flow_disassociate(struct ufid_to_rte_flow_data *data) return; } + offload_data_lock(data->netdev); cmap_remove(map, CONST_CAST(struct cmap_node *, &data->node), hash); + offload_data_unlock(data->netdev); + netdev_close(data->netdev); ovsrcu_postpone(free, data); } From patchwork Wed Jun 9 13:09:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489879 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=BXvseIqG; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=A5Bd9a9l; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SDy5xccz9sXL for ; Wed, 9 Jun 2021 23:12:02 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id F15C641DBB; Wed, 9 Jun 2021 13:12:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FkpQXbLYv4xS; Wed, 9 Jun 2021 13:11:57 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 5300F41D6E; Wed, 9 Jun 2021 13:11:56 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2DBFCC0024; Wed, 9 Jun 2021 13:11:56 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 235FFC0024 for ; Wed, 9 Jun 2021 13:11:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 5761D405FE for ; Wed, 9 Jun 2021 13:10:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H5qp0s0J1AJN for ; Wed, 9 Jun 2021 13:10:30 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id BFFEF419BA for ; Wed, 9 Jun 2021 13:10:20 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 33A3E260C; Wed, 9 Jun 2021 09:10:20 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=/HaOcMxuQaon1 FZdqqHULzAU5+YeR5UeqM+D3ZN1DnM=; b=BXvseIqGctypyl/bOZUDvWy9zpuz3 QPyI2l01rf9iSOp+fYfFzNYpkvIJZfD5/SnEaRuOKYcUPhGlg7ElDkTgGIVffucY e2KA1VaKgfkHr6sHUtZLR/xWFU2d7s1hhurKA1buNIXk2Cw4S3xf8BF9prAym6Vp ZGPju2PSzJdQYBp5sovekAMAzOM8ce3U8zW6+KywFG/9X8u4XoogbaO/ll4a/qvp WAS9HsS3TsXxjH0QT+lwAErzDjdCe9c8DqkNcrCAOQ0iN7G0yUhqjnYp7585BK1/ oNQ8+BnQGnT1mDyVNllWbmtuQfIjuVO4nuJKIQMDwVBCV02Bgvd4Pdopg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=/HaOcMxuQaon1FZdqqHULzAU5+YeR5UeqM+D3ZN1DnM=; b=A5Bd9a9l MiMeIgeyCqf0WlMa6bVeVe/kfszF+O7yW+xaMyhXUI0fIyIfICjg9bm3S88R3HOx yxjzwy6RqYI0hXM42VBTFzDJa7Yqjc2Q46R2lD1N1R4qbCNz1PZ68LaSPP+D7dWE hBcF9n/2ZIIHxmbn/Hs2unaysF/ibGnd1WdgP4ViPNqj/JVBMogsVJncrLZIV1Ko levpmaXDU6MEio1MpaHwXAcpsTZonojMxwC18foQ5Wxb4oX65ZdglvS8tls95OP/ hRA9NCOpb3ire81b//77eBrfk5R91PNie2Arm9o1CtXf9MpSsKsZuVJqL+Y4PzYv CmtdhhpvQhRH2Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepgeenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:19 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:30 +0200 Message-Id: <7c5893d998ab29f8eeec3f0f9e481894a4925034.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 22/27] netdev-offload-dpdk: Protect concurrent offload destroy/query X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The rte_flow API in DPDK is now thread safe for insertion and deletion. It is not however safe for concurrent query while the offload is being inserted or deleted. Insertion is not an issue as the rte_flow handle will be published to other threads only once it has been inserted in the hardware, so the query will only be able to proceed once it is already available. For the deletion path however, offload status queries can be made while an offload is being destroyed. This would create race conditions and use-after-free if not properly protected. As a pre-step before removing the OVS-level locks on the rte_flow API, mutually exclude offload query and deletion from concurrent execution. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-offload-dpdk.c | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index 4459a0aa1..13e017ef8 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -58,6 +58,8 @@ struct ufid_to_rte_flow_data { struct rte_flow *rte_flow; bool actions_offloaded; struct dpif_flow_stats stats; + struct ovs_mutex lock; + bool dead; }; struct netdev_offload_dpdk_data { @@ -233,6 +235,7 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, data->netdev = netdev_ref(netdev); data->rte_flow = rte_flow; data->actions_offloaded = actions_offloaded; + ovs_mutex_init(&data->lock); cmap_insert(map, CONST_CAST(struct cmap_node *, &data->node), hash); @@ -240,8 +243,16 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, return data; } +static void +rte_flow_data_unref(struct ufid_to_rte_flow_data *data) +{ + ovs_mutex_destroy(&data->lock); + free(data); +} + static inline void ufid_to_rte_flow_disassociate(struct ufid_to_rte_flow_data *data) + OVS_REQUIRES(data->lock) { size_t hash = hash_bytes(&data->ufid, sizeof data->ufid, 0); struct cmap *map = offload_data_map(data->netdev); @@ -255,7 +266,7 @@ ufid_to_rte_flow_disassociate(struct ufid_to_rte_flow_data *data) offload_data_unlock(data->netdev); netdev_close(data->netdev); - ovsrcu_postpone(free, data); + ovsrcu_postpone(rte_flow_data_unref, data); } /* @@ -1581,6 +1592,15 @@ netdev_offload_dpdk_flow_destroy(struct ufid_to_rte_flow_data *rte_flow_data) ovs_u128 *ufid; int ret; + ovs_mutex_lock(&rte_flow_data->lock); + + if (rte_flow_data->dead) { + ovs_mutex_unlock(&rte_flow_data->lock); + return 0; + } + + rte_flow_data->dead = true; + rte_flow = rte_flow_data->rte_flow; netdev = rte_flow_data->netdev; ufid = &rte_flow_data->ufid; @@ -1607,6 +1627,8 @@ netdev_offload_dpdk_flow_destroy(struct ufid_to_rte_flow_data *rte_flow_data) UUID_ARGS((struct uuid *) ufid)); } + ovs_mutex_unlock(&rte_flow_data->lock); + return ret; } @@ -1702,8 +1724,19 @@ netdev_offload_dpdk_flow_get(struct netdev *netdev, struct rte_flow_error error; int ret = 0; + attrs->dp_extra_info = NULL; + rte_flow_data = ufid_to_rte_flow_data_find(netdev, ufid, false); - if (!rte_flow_data || !rte_flow_data->rte_flow) { + if (!rte_flow_data || !rte_flow_data->rte_flow || + rte_flow_data->dead || ovs_mutex_trylock(&rte_flow_data->lock)) { + return -1; + } + + /* Check again whether the data is dead, as it could have been + * updated while the lock was not yet taken. The first check above + * was only to avoid unnecessary locking if possible. + */ + if (rte_flow_data->dead) { ret = -1; goto out; } @@ -1730,7 +1763,7 @@ netdev_offload_dpdk_flow_get(struct netdev *netdev, } memcpy(stats, &rte_flow_data->stats, sizeof *stats); out: - attrs->dp_extra_info = NULL; + ovs_mutex_unlock(&rte_flow_data->lock); return ret; } From patchwork Wed Jun 9 13:09:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489883 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=I4tPaUkt; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=qC8lrFlI; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SFJ0h5pz9sRK for ; Wed, 9 Jun 2021 23:12:19 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 4934B6F81F; Wed, 9 Jun 2021 13:12:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0rx9V_ObsERo; Wed, 9 Jun 2021 13:12:11 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4733B6F8FF; Wed, 9 Jun 2021 13:12:06 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 32994C002B; Wed, 9 Jun 2021 13:12:05 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1026BC000D for ; Wed, 9 Jun 2021 13:12:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E2254419B6 for ; Wed, 9 Jun 2021 13:10:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="I4tPaUkt"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="qC8lrFlI" Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id d7EB7IqiDml6 for ; Wed, 9 Jun 2021 13:10:34 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp2.osuosl.org (Postfix) with ESMTPS id 5D886419A6 for ; Wed, 9 Jun 2021 13:10:22 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id BB60C261A; Wed, 9 Jun 2021 09:10:21 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=ZumV+8X9Mn6u+ /qmPgn+yEp9EzKIxnUdKsSMAdWZHrM=; b=I4tPaUktRYg5Jh/O1SgiNdYKmOtK5 DMc8cepczFC0cQWHaJuTbp5PBo60e3AVq5rGhkFQUhNeuQszse3GGjdC7GJ8UDqK wiiQDy0ryi4WCYMerOVc/LOSTp3/KZzAVWX6Ck92j26LS5eVWTOrYJe9Xi/wc0Qp /56DGsEuKnowBO5lIDklJlcRteS/cogAMKd8M9qjTcZ5xlOd+aHEwcQXLNFWBYPO tVqI9VCGSU7FL1yyNu9kdqV3/JiYJQwgZlQmRI8HoW5goV1VR4adXvXeEmA1EqFW d8qOOOppFXH1va068rn6M6BS8vXLK1HNC/JFzBS1UMF+uX7bQDGcp+s3w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=ZumV+8X9Mn6u+/qmPgn+yEp9EzKIxnUdKsSMAdWZHrM=; b=qC8lrFlI eYZHF0LpAUH1i8LmaNBDBfkrJ02+51OVNSVyT3BE/WGy9WfLLDXpm6b7wDMACN+d wl7+ho5xA+fdnx504mlb++zusuEOUZb4MwecHZxagyQ+sIzGQzHEsCRruZdgmiRo eWcvsYm6pvogpKLtJuiL2bkTxAm9cQZp023tZSeSPvcoxbWtFzn1II0etS0IkAvL ialY962CqS132ZrC8M0MSdBvJaVF+S7jLvttjKPXhMJC+eeEYPrizFoqASp5s9L7 +/vgyDsWZhJdwML/O4DrdNaJx4wa1+qZ5juscsIkj/huwsuSWOm0qNiuRJKymP67 /V/vGIo1HH7BQw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepieenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:20 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:31 +0200 Message-Id: <533f7f8b9c2ddd343b884b2d6921b9285598ed37.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 23/27] dpif-netdev: Use lockless queue to manage offloads X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The dataplane threads (PMDs) send offloading commands to a dedicated offload management thread. The current implementation uses a lock and benchmarks show a high contention on the queue in some cases. With high-contention, the mutex will more often lead to the locking thread yielding in wait, using a syscall. This should be avoided in a userland dataplane. The mpsc-queue can be used instead. It uses less cycles and has lower latency. Benchmarks show better behavior as multiple revalidators and one or multiple PMDs writes to a single queue while another thread polls it. One trade-off with the new scheme however is to be forced to poll the queue from the offload thread. Without mutex, a cond_wait cannot be used for signaling. The offload thread is implementing an exponential backoff and will sleep in short increments when no data is available. This makes the thread yield, at the price of some latency to manage offloads after an inactivity period. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 109 ++++++++++++++++++++++++---------------------- 1 file changed, 57 insertions(+), 52 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 1daaecb1c..68dcdf39a 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -53,6 +53,7 @@ #include "id-pool.h" #include "ipf.h" #include "mov-avg.h" +#include "mpsc-queue.h" #include "netdev.h" #include "netdev-offload.h" #include "netdev-provider.h" @@ -452,25 +453,22 @@ union dp_offload_thread_data { }; struct dp_offload_thread_item { - struct ovs_list node; + struct mpsc_queue_node node; enum dp_offload_type type; long long int timestamp; union dp_offload_thread_data data[0]; }; struct dp_offload_thread { - struct ovs_mutex mutex; - struct ovs_list list; - uint64_t enqueued_item; + struct mpsc_queue queue; + atomic_uint64_t enqueued_item; struct mov_avg_cma cma; struct mov_avg_ema ema; - pthread_cond_t cond; }; static struct dp_offload_thread dp_offload_thread = { - .mutex = OVS_MUTEX_INITIALIZER, - .list = OVS_LIST_INITIALIZER(&dp_offload_thread.list), - .enqueued_item = 0, + .queue = MPSC_QUEUE_INITIALIZER(&dp_offload_thread.queue), + .enqueued_item = ATOMIC_VAR_INIT(0), .cma = MOV_AVG_CMA_INITIALIZER, .ema = MOV_AVG_EMA_INITIALIZER(100), }; @@ -2697,11 +2695,8 @@ dp_netdev_free_offload(struct dp_offload_thread_item *offload) static void dp_netdev_append_offload(struct dp_offload_thread_item *offload) { - ovs_mutex_lock(&dp_offload_thread.mutex); - ovs_list_push_back(&dp_offload_thread.list, &offload->node); - dp_offload_thread.enqueued_item++; - xpthread_cond_signal(&dp_offload_thread.cond); - ovs_mutex_unlock(&dp_offload_thread.mutex); + mpsc_queue_insert(&dp_offload_thread.queue, &offload->node); + atomic_count_inc64(&dp_offload_thread.enqueued_item); } static int @@ -2845,58 +2840,68 @@ dp_offload_flush(struct dp_offload_thread_item *item) ovs_barrier_block(flush->barrier); } +#define DP_NETDEV_OFFLOAD_BACKOFF_MIN 1 +#define DP_NETDEV_OFFLOAD_BACKOFF_MAX 64 #define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ static void * dp_netdev_flow_offload_main(void *data OVS_UNUSED) { struct dp_offload_thread_item *offload; - struct ovs_list *list; + struct mpsc_queue_node *node; + struct mpsc_queue *queue; long long int latency_us; long long int next_rcu; long long int now; + uint64_t backoff; - next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; - for (;;) { - ovs_mutex_lock(&dp_offload_thread.mutex); - if (ovs_list_is_empty(&dp_offload_thread.list)) { - ovsrcu_quiesce_start(); - ovs_mutex_cond_wait(&dp_offload_thread.cond, - &dp_offload_thread.mutex); - ovsrcu_quiesce_end(); - next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; - } - list = ovs_list_pop_front(&dp_offload_thread.list); - dp_offload_thread.enqueued_item--; - offload = CONTAINER_OF(list, struct dp_offload_thread_item, node); - ovs_mutex_unlock(&dp_offload_thread.mutex); - - switch (offload->type) { - case DP_OFFLOAD_FLOW: - dp_offload_flow(offload); - break; - case DP_OFFLOAD_FLUSH: - dp_offload_flush(offload); - break; - default: - OVS_NOT_REACHED(); + queue = &dp_offload_thread.queue; + mpsc_queue_acquire(queue); + + while (true) { + backoff = DP_NETDEV_OFFLOAD_BACKOFF_MIN; + while (mpsc_queue_tail(queue) == NULL) { + xnanosleep(backoff * 1E6); + if (backoff < DP_NETDEV_OFFLOAD_BACKOFF_MAX) { + backoff <<= 1; + } } - now = time_usec(); + next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; + MPSC_QUEUE_FOR_EACH_POP (node, queue) { + offload = CONTAINER_OF(node, struct dp_offload_thread_item, node); + atomic_count_dec64(&dp_offload_thread.enqueued_item); - latency_us = now - offload->timestamp; - mov_avg_cma_update(&dp_offload_thread.cma, latency_us); - mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + switch (offload->type) { + case DP_OFFLOAD_FLOW: + dp_offload_flow(offload); + break; + case DP_OFFLOAD_FLUSH: + dp_offload_flush(offload); + break; + default: + OVS_NOT_REACHED(); + } - dp_netdev_free_offload(offload); + now = time_usec(); - /* Do RCU synchronization at fixed interval. */ - if (now > next_rcu) { - ovsrcu_quiesce(); - next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; + latency_us = now - offload->timestamp; + mov_avg_cma_update(&dp_offload_thread.cma, latency_us); + mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + + dp_netdev_free_offload(offload); + + /* Do RCU synchronization at fixed interval. */ + if (now > next_rcu) { + ovsrcu_quiesce(); + next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; + } } } + OVS_NOT_REACHED(); + mpsc_queue_release(queue); + return NULL; } @@ -2907,7 +2912,7 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, struct dp_offload_thread_item *offload; if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_offload_thread.cond, NULL); + mpsc_queue_init(&dp_offload_thread.queue); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -2932,7 +2937,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, } if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_offload_thread.cond, NULL); + mpsc_queue_init(&dp_offload_thread.queue); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -2983,7 +2988,7 @@ dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, struct dp_offload_flush_item *flush; if (ovsthread_once_start(&offload_thread_once)) { - xpthread_cond_init(&dp_offload_thread.cond, NULL); + mpsc_queue_init(&dp_offload_thread.queue); ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); ovsthread_once_done(&offload_thread_once); } @@ -4470,8 +4475,8 @@ dpif_netdev_offload_stats_get(struct dpif *dpif, } ovs_mutex_unlock(&dp->port_mutex); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value = - dp_offload_thread.enqueued_item; + atomic_read_relaxed(&dp_offload_thread.enqueued_item, + &stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value); stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED].value = nb_offloads; stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].value = mov_avg_cma(&dp_offload_thread.cma); From patchwork Wed Jun 9 13:09:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489884 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=gPAhhCD3; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=MUDyXir+; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SFL1DQ0z9sWD for ; Wed, 9 Jun 2021 23:12:21 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 428BA84225; Wed, 9 Jun 2021 13:12:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 438tt3Ni3fop; Wed, 9 Jun 2021 13:12:18 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 9047D84183; Wed, 9 Jun 2021 13:12:16 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 65244C000E; Wed, 9 Jun 2021 13:12:16 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 03096C000B for ; Wed, 9 Jun 2021 13:12:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id EF9AA60C10 for ; Wed, 9 Jun 2021 13:10:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="gPAhhCD3"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="MUDyXir+" Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9un7I_3v_Auj for ; Wed, 9 Jun 2021 13:10:37 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp3.osuosl.org (Postfix) with ESMTPS id 0DC0160AF9 for ; Wed, 9 Jun 2021 13:10:24 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 61288261C; Wed, 9 Jun 2021 09:10:23 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=dwRg3a0mEAU5g ExNnQsOtUpdFkEVIH+jwOcReTYFtEk=; b=gPAhhCD3gUu3k3M6c3rQvMeW3tL3f d11vH6RIcNfxhCyMRB+jwZe+99SmirY4B5F64L3p/eIVM/nmtVOnVuLrqvWUOE99 EzwH7PauD/V5snHplUwTFzaNAKKbhWfhS49VPNyH6mdsL60xfrYAtyevf7iJfgQG 2sT/BRLSNx0XRs3Tos4pZUMSCfyjgnzPpY/W/fGAQqjEv9r7z0OKbkle+HBmWhy6 CzPWev8V2QbLXhzPHX+j/G0ZHkTtuuvm8wLHLlrcmK+Op5G6nOKKqOFlBv26AkP/ E10TbVxAX7OVnWVXEzN/lx0XOna7NwrXtC1HllKYkgmrkzyoasLFmIV8w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=dwRg3a0mEAU5gExNnQsOtUpdFkEVIH+jwOcReTYFtEk=; b=MUDyXir+ EBKHuU82M7nHtqq9iRwHupLzgkt78egxHhID5BiRXBlS2hh+c0vf4N0KN+uwmiAp wJ0ZLwQ2wVR7x0z9W8BzRJggcc6f58LW06/Xe8fKd0UhhAdLCOFGQZ52lSoiWW/S 8Z5I85VLG0Y6y+BdzimoBGi1LitXPM2Gp0bljkVyXCXufU7dNrqi5WaYdfK9Boc4 lp6Zwzmp4T7zszsr6U2KWxsuSEdGkGuCuXqJy7UYmoiJv4QqiBGC6gEO6lvXxXqg xkLmy0tIVPm7sRPXQtx37LvstTJVvVZRLQ1Qw0/ok46BNGpzI4czRGH2X3LV8Frn MPhRNFKiu59nGA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepieenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:22 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:32 +0200 Message-Id: <6f4eb9a71ca4b4ca17ce3d1c11181b3bf12b5970.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 24/27] dpif-netdev: Make megaflow and mark mappings thread objects X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" In later commits hardware offloads are managed in several threads. Each offload is managed by a thread determined by its flow's 'mega_ufid'. As megaflow to mark and mark to flow mappings are 1:1 and 1:N respectively, then a single mark exists for a single 'mega_ufid', and multiple flows uses the same 'mega_ufid'. Because the managing thread will be chosen using the 'mega_ufid', then each mapping does not need to be shared with other offload threads. The mappings are kept as cmap as upcalls will sometimes query them before enqueuing orders to the offload threads. To prepare this change, move the mappings within the offload thread structure. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 47 ++++++++++++++++++++++------------------------- 1 file changed, 22 insertions(+), 25 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 68dcdf39a..8fe794557 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -462,12 +462,16 @@ struct dp_offload_thread_item { struct dp_offload_thread { struct mpsc_queue queue; atomic_uint64_t enqueued_item; + struct cmap megaflow_to_mark; + struct cmap mark_to_flow; struct mov_avg_cma cma; struct mov_avg_ema ema; }; static struct dp_offload_thread dp_offload_thread = { .queue = MPSC_QUEUE_INITIALIZER(&dp_offload_thread.queue), + .megaflow_to_mark = CMAP_INITIALIZER, + .mark_to_flow = CMAP_INITIALIZER, .enqueued_item = ATOMIC_VAR_INIT(0), .cma = MOV_AVG_CMA_INITIALIZER, .ema = MOV_AVG_EMA_INITIALIZER(100), @@ -2437,32 +2441,23 @@ struct megaflow_to_mark_data { uint32_t mark; }; -struct flow_mark { - struct cmap megaflow_to_mark; - struct cmap mark_to_flow; - struct id_fpool *pool; -}; - -static struct flow_mark flow_mark = { - .megaflow_to_mark = CMAP_INITIALIZER, - .mark_to_flow = CMAP_INITIALIZER, -}; +static struct id_fpool *flow_mark_pool; static uint32_t flow_mark_alloc(void) { - static struct ovsthread_once pool_init = OVSTHREAD_ONCE_INITIALIZER; + static struct ovsthread_once init_once = OVSTHREAD_ONCE_INITIALIZER; unsigned int tid = netdev_offload_thread_id(); uint32_t mark; - if (ovsthread_once_start(&pool_init)) { + if (ovsthread_once_start(&init_once)) { /* Haven't initiated yet, do it here */ - flow_mark.pool = id_fpool_create(netdev_offload_thread_nb(), + flow_mark_pool = id_fpool_create(netdev_offload_thread_nb(), 1, MAX_FLOW_MARK); - ovsthread_once_done(&pool_init); + ovsthread_once_done(&init_once); } - if (id_fpool_new_id(flow_mark.pool, tid, &mark)) { + if (id_fpool_new_id(flow_mark_pool, tid, &mark)) { return mark; } @@ -2474,7 +2469,7 @@ flow_mark_free(uint32_t mark) { unsigned int tid = netdev_offload_thread_id(); - id_fpool_free_id(flow_mark.pool, tid, mark); + id_fpool_free_id(flow_mark_pool, tid, mark); } /* associate megaflow with a mark, which is a 1:1 mapping */ @@ -2487,7 +2482,7 @@ megaflow_to_mark_associate(const ovs_u128 *mega_ufid, uint32_t mark) data->mega_ufid = *mega_ufid; data->mark = mark; - cmap_insert(&flow_mark.megaflow_to_mark, + cmap_insert(&dp_offload_thread.megaflow_to_mark, CONST_CAST(struct cmap_node *, &data->node), hash); } @@ -2498,9 +2493,10 @@ megaflow_to_mark_disassociate(const ovs_u128 *mega_ufid) size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data; - CMAP_FOR_EACH_WITH_HASH (data, node, hash, &flow_mark.megaflow_to_mark) { + CMAP_FOR_EACH_WITH_HASH (data, node, hash, + &dp_offload_thread.megaflow_to_mark) { if (ovs_u128_equals(*mega_ufid, data->mega_ufid)) { - cmap_remove(&flow_mark.megaflow_to_mark, + cmap_remove(&dp_offload_thread.megaflow_to_mark, CONST_CAST(struct cmap_node *, &data->node), hash); ovsrcu_postpone(free, data); return; @@ -2517,7 +2513,8 @@ megaflow_to_mark_find(const ovs_u128 *mega_ufid) size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data; - CMAP_FOR_EACH_WITH_HASH (data, node, hash, &flow_mark.megaflow_to_mark) { + CMAP_FOR_EACH_WITH_HASH (data, node, hash, + &dp_offload_thread.megaflow_to_mark) { if (ovs_u128_equals(*mega_ufid, data->mega_ufid)) { return data->mark; } @@ -2534,7 +2531,7 @@ mark_to_flow_associate(const uint32_t mark, struct dp_netdev_flow *flow) { dp_netdev_flow_ref(flow); - cmap_insert(&flow_mark.mark_to_flow, + cmap_insert(&dp_offload_thread.mark_to_flow, CONST_CAST(struct cmap_node *, &flow->mark_node), hash_int(mark, 0)); flow->mark = mark; @@ -2549,7 +2546,7 @@ flow_mark_has_no_ref(uint32_t mark) struct dp_netdev_flow *flow; CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash_int(mark, 0), - &flow_mark.mark_to_flow) { + &dp_offload_thread.mark_to_flow) { if (flow->mark == mark) { return false; } @@ -2574,7 +2571,7 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, return EINVAL; } - cmap_remove(&flow_mark.mark_to_flow, mark_node, hash_int(mark, 0)); + cmap_remove(&dp_offload_thread.mark_to_flow, mark_node, hash_int(mark, 0)); flow->mark = INVALID_FLOW_MARK; /* @@ -2611,7 +2608,7 @@ flow_mark_flush(struct dp_netdev_pmd_thread *pmd) { struct dp_netdev_flow *flow; - CMAP_FOR_EACH (flow, mark_node, &flow_mark.mark_to_flow) { + CMAP_FOR_EACH (flow, mark_node, &dp_offload_thread.mark_to_flow) { if (flow->pmd_id == pmd->core_id) { queue_netdev_flow_del(pmd, flow); } @@ -2625,7 +2622,7 @@ mark_to_flow_find(const struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow; CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash_int(mark, 0), - &flow_mark.mark_to_flow) { + &dp_offload_thread.mark_to_flow) { if (flow->mark == mark && flow->pmd_id == pmd->core_id && flow->dead == false) { return flow; From patchwork Wed Jun 9 13:09:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489880 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=BQ6e/AYG; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=hrde6bi5; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SF4192vz9sWX for ; Wed, 9 Jun 2021 23:12:07 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C4C3A41DCA; Wed, 9 Jun 2021 13:12:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uFhONsTaRXsj; Wed, 9 Jun 2021 13:12:01 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 90C7A41D9E; Wed, 9 Jun 2021 13:11:59 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 118A2C0028; Wed, 9 Jun 2021 13:11:57 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3500AC0025 for ; Wed, 9 Jun 2021 13:11:56 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 4C9FC405D9 for ; Wed, 9 Jun 2021 13:10:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="BQ6e/AYG"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="hrde6bi5" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tLo081UqIvt4 for ; Wed, 9 Jun 2021 13:10:29 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp4.osuosl.org (Postfix) with ESMTPS id A4B54405BA for ; Wed, 9 Jun 2021 13:10:25 +0000 (UTC) Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 06760259F; Wed, 9 Jun 2021 09:10:24 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 09 Jun 2021 09:10:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=lzRUGwxkMuk7v KebKFJ0b9uQ42tr6hECUq7/F5V+yvo=; b=BQ6e/AYGWgKCloOKFdSbNp3t3A5fJ VrjL04T0tbNdn/sHXuVxLsejcaV7U5IZWGd6iTf7/6Cbj+YFBgKtwiRDiA+beUs4 6pPzT4uilwvBrQhxy3NBPyHW2a+J0dHBQBqvQkUsAUnU9J6UlICxG7vgan2Bp5Fd sBjWqUvUBGgaBmA21hMXSMkvMv8CA9arzfogZwkY/8aWz+1zCDH+TxyjuSeamBCr qA74tgYejdWcqjFqhpu6YCtFcVDrzdF9FGwuB43I6+NTwiIMX/40+o8SPw0B0EoO qxDqmOsQYQ9Ks5S4nwaDE7WxkrvOC8L1bWYU4BAWuzaIeCe0XuC6ZqxLw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=lzRUGwxkMuk7vKebKFJ0b9uQ42tr6hECUq7/F5V+yvo=; b=hrde6bi5 vcdnTvdOfU+LYCL06BKSfldk8VQETzoYtQTyWR29tnyylciA/YrmJs0lM3oA/wx2 zg25ANl7lWe0FFiVpvn0Ec58ZWZJnzOsL7wyV88NOpiAp9uy4ai2+ZzC6dp8nMO6 RBxN1fNa1WJsOoycsysylsYVe8+L2NHR7yOSfpSDF46qUbc3gU5lDZvRfKT/6bkf MyFLH64f9zuUBpxlq3OxrXIL5GMB9ayhPBL90icZHRM992bswLprlmdPtM5GkgP+ 3ah+jIzTDqXAmd+pN92wU9V/3OW4cFw4JL5I7TKadgkN9ZPIWDOEmSQt3qwF/9la 03M3qunh2X9wKg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh ephefgveffkeetheetfeeifedvheelfeejfeehveduteejhfekuedtkeeiuedvteehnecu vehluhhsthgvrhfuihiivgepgeenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvg esuhdvheeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:23 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:33 +0200 Message-Id: X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 25/27] dpif-netdev: Replace port mutex by rwlock X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The port mutex protects the netdev mapping, that can be changed by port addition or port deletion. HW offloads operations can be considered read operations on the port mapping itself. Use a rwlock to differentiate between read and write operations, allowing concurrent queries and offload insertions. Because offload queries, deletion, and reconfigure_datapath() calls are all rdlock, the deadlock fixed by [1] is still avoided, as the rdlock side is recursive as prescribed by the POSIX standard. Executing 'reconfigure_datapath()' only requires a rdlock taken, but it is sometimes executed in contexts where wrlock is taken ('do_add_port()' and 'do_del_port()'). This means that the deadlock described in [2] is still valid and should be mitigated. The rdlock is taken using 'tryrdlock()' during offload query, keeping the current behavior. [1]: 81e89d5c2645 ("dpif-netdev: Make datapath port mutex recursive.") [2]: 12d0edd75eba ("dpif-netdev: Avoid deadlock with offloading during PMD thread deletion."). Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 139 +++++++++++++++++++------------------- lib/netdev-offload-dpdk.c | 4 +- 2 files changed, 72 insertions(+), 71 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 8fe794557..cc7a979d7 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -333,8 +333,8 @@ struct dp_netdev { /* Ports. * * Any lookup into 'ports' or any access to the dp_netdev_ports found - * through 'ports' requires taking 'port_mutex'. */ - struct ovs_mutex port_mutex; + * through 'ports' requires taking 'port_rwlock'. */ + struct ovs_rwlock port_rwlock; struct hmap ports; struct seq *port_seq; /* Incremented whenever a port changes. */ @@ -410,7 +410,7 @@ static void meter_unlock(const struct dp_netdev *dp, uint32_t meter_id) static struct dp_netdev_port *dp_netdev_lookup_port(const struct dp_netdev *dp, odp_port_t) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_RDLOCK(dp->port_rwlock); enum rxq_cycles_counter_type { RXQ_CYCLES_PROC_CURR, /* Cycles spent successfully polling and @@ -851,17 +851,17 @@ struct dpif_netdev { static int get_port_by_number(struct dp_netdev *dp, odp_port_t port_no, struct dp_netdev_port **portp) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_RDLOCK(dp->port_rwlock); static int get_port_by_name(struct dp_netdev *dp, const char *devname, struct dp_netdev_port **portp) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_RDLOCK(dp->port_rwlock); static void dp_netdev_free(struct dp_netdev *) OVS_REQUIRES(dp_netdev_mutex); static int do_add_port(struct dp_netdev *dp, const char *devname, const char *type, odp_port_t port_no) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_WRLOCK(dp->port_rwlock); static void do_del_port(struct dp_netdev *dp, struct dp_netdev_port *) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_WRLOCK(dp->port_rwlock); static int dpif_netdev_open(const struct dpif_class *, const char *name, bool create, struct dpif **); static void dp_netdev_execute_actions(struct dp_netdev_pmd_thread *pmd, @@ -882,7 +882,7 @@ static void dp_netdev_configure_pmd(struct dp_netdev_pmd_thread *pmd, int numa_id); static void dp_netdev_destroy_pmd(struct dp_netdev_pmd_thread *pmd); static void dp_netdev_set_nonpmd(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_WRLOCK(dp->port_rwlock); static void *pmd_thread_main(void *); static struct dp_netdev_pmd_thread *dp_netdev_get_pmd(struct dp_netdev *dp, @@ -919,7 +919,7 @@ static void dp_netdev_offload_flush(struct dp_netdev *dp, struct dp_netdev_port *port); static void reconfigure_datapath(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex); + OVS_REQ_RDLOCK(dp->port_rwlock); static bool dp_netdev_pmd_try_ref(struct dp_netdev_pmd_thread *pmd); static void dp_netdev_pmd_unref(struct dp_netdev_pmd_thread *pmd); static void dp_netdev_pmd_flow_flush(struct dp_netdev_pmd_thread *pmd); @@ -1425,8 +1425,8 @@ dpif_netdev_subtable_lookup_set(struct unixctl_conn *conn, int argc, struct dp_netdev_pmd_thread **pmd_list; sorted_poll_thread_list(dp, &pmd_list, &n); - /* take port mutex as HMAP iters over them. */ - ovs_mutex_lock(&dp->port_mutex); + /* take port rwlock as HMAP iters over them. */ + ovs_rwlock_rdlock(&dp->port_rwlock); for (size_t i = 0; i < n; i++) { struct dp_netdev_pmd_thread *pmd = pmd_list[i]; @@ -1449,8 +1449,8 @@ dpif_netdev_subtable_lookup_set(struct unixctl_conn *conn, int argc, } } - /* release port mutex before netdev mutex. */ - ovs_mutex_unlock(&dp->port_mutex); + /* release port rwlock before netdev mutex. */ + ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_unlock(&dp_netdev_mutex); struct ds reply = DS_EMPTY_INITIALIZER; @@ -1743,7 +1743,7 @@ create_dpif_netdev(struct dp_netdev *dp) * Return ODPP_NONE on failure. */ static odp_port_t choose_port(struct dp_netdev *dp, const char *name) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { uint32_t port_no; @@ -1806,7 +1806,7 @@ create_dp_netdev(const char *name, const struct dpif_class *class, ovs_refcount_init(&dp->ref_cnt); atomic_flag_clear(&dp->destroyed); - ovs_mutex_init_recursive(&dp->port_mutex); + ovs_rwlock_init(&dp->port_rwlock); hmap_init(&dp->ports); dp->port_seq = seq_create(); ovs_mutex_init(&dp->bond_mutex); @@ -1841,7 +1841,7 @@ create_dp_netdev(const char *name, const struct dpif_class *class, ovs_mutex_init_recursive(&dp->non_pmd_mutex); ovsthread_key_create(&dp->per_pmd_key, NULL); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); /* non-PMD will be created before all other threads and will * allocate static_tx_qid = 0. */ dp_netdev_set_nonpmd(dp); @@ -1849,7 +1849,7 @@ create_dp_netdev(const char *name, const struct dpif_class *class, error = do_add_port(dp, name, dpif_netdev_port_open_type(dp->class, "internal"), ODPP_LOCAL); - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); if (error) { dp_netdev_free(dp); return error; @@ -1935,11 +1935,11 @@ dp_netdev_free(struct dp_netdev *dp) shash_find_and_delete(&dp_netdevs, dp->name); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); HMAP_FOR_EACH_SAFE (port, next, node, &dp->ports) { do_del_port(dp, port); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_lock(&dp->bond_mutex); CMAP_FOR_EACH (bond, node, &dp->tx_bonds) { @@ -1964,7 +1964,7 @@ dp_netdev_free(struct dp_netdev *dp) seq_destroy(dp->port_seq); hmap_destroy(&dp->ports); - ovs_mutex_destroy(&dp->port_mutex); + ovs_rwlock_destroy(&dp->port_rwlock); cmap_destroy(&dp->tx_bonds); ovs_mutex_destroy(&dp->bond_mutex); @@ -2132,7 +2132,7 @@ out: static int do_add_port(struct dp_netdev *dp, const char *devname, const char *type, odp_port_t port_no) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_WRLOCK(dp->port_rwlock) { struct netdev_saved_flags *sf; struct dp_netdev_port *port; @@ -2184,7 +2184,7 @@ dpif_netdev_port_add(struct dpif *dpif, struct netdev *netdev, odp_port_t port_no; int error; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); dpif_port = netdev_vport_get_dpif_port(netdev, namebuf, sizeof namebuf); if (*port_nop != ODPP_NONE) { port_no = *port_nop; @@ -2197,7 +2197,7 @@ dpif_netdev_port_add(struct dpif *dpif, struct netdev *netdev, *port_nop = port_no; error = do_add_port(dp, dpif_port, netdev_get_type(netdev), port_no); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -2208,7 +2208,7 @@ dpif_netdev_port_del(struct dpif *dpif, odp_port_t port_no) struct dp_netdev *dp = get_dp_netdev(dpif); int error; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); if (port_no == ODPP_LOCAL) { error = EINVAL; } else { @@ -2219,7 +2219,7 @@ dpif_netdev_port_del(struct dpif *dpif, odp_port_t port_no) do_del_port(dp, port); } } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -2232,7 +2232,7 @@ is_valid_port_number(odp_port_t port_no) static struct dp_netdev_port * dp_netdev_lookup_port(const struct dp_netdev *dp, odp_port_t port_no) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; @@ -2247,7 +2247,7 @@ dp_netdev_lookup_port(const struct dp_netdev *dp, odp_port_t port_no) static int get_port_by_number(struct dp_netdev *dp, odp_port_t port_no, struct dp_netdev_port **portp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { if (!is_valid_port_number(port_no)) { *portp = NULL; @@ -2282,7 +2282,7 @@ port_destroy(struct dp_netdev_port *port) static int get_port_by_name(struct dp_netdev *dp, const char *devname, struct dp_netdev_port **portp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; @@ -2301,7 +2301,7 @@ get_port_by_name(struct dp_netdev *dp, /* Returns 'true' if there is a port with pmd netdev. */ static bool has_pmd_port(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; @@ -2316,7 +2316,7 @@ has_pmd_port(struct dp_netdev *dp) static void do_del_port(struct dp_netdev *dp, struct dp_netdev_port *port) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_WRLOCK(dp->port_rwlock) { dp_netdev_offload_flush(dp, port); netdev_uninit_flow_api(port->netdev); @@ -2345,12 +2345,12 @@ dpif_netdev_port_query_by_number(const struct dpif *dpif, odp_port_t port_no, struct dp_netdev_port *port; int error; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); error = get_port_by_number(dp, port_no, &port); if (!error && dpif_port) { answer_port_query(port, dpif_port); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -2363,12 +2363,12 @@ dpif_netdev_port_query_by_name(const struct dpif *dpif, const char *devname, struct dp_netdev_port *port; int error; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); error = get_port_by_name(dp, devname, &port); if (!error && dpif_port) { answer_port_query(port, dpif_port); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -2586,9 +2586,9 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, if (port) { /* Taking a global 'port_mutex' to fulfill thread safety * restrictions regarding netdev port mapping. */ - ovs_mutex_lock(&pmd->dp->port_mutex); + ovs_rwlock_rdlock(&pmd->dp->port_rwlock); ret = netdev_flow_del(port, &flow->mega_ufid, NULL); - ovs_mutex_unlock(&pmd->dp->port_mutex); + ovs_rwlock_unlock(&pmd->dp->port_rwlock); netdev_close(port); } @@ -2764,12 +2764,12 @@ dp_netdev_flow_offload_put(struct dp_offload_flow_item *offload) } /* Taking a global 'port_mutex' to fulfill thread safety * restrictions regarding the netdev port mapping. */ - ovs_mutex_lock(&pmd->dp->port_mutex); + ovs_rwlock_rdlock(&pmd->dp->port_rwlock); ret = netdev_flow_put(port, &offload->match, CONST_CAST(struct nlattr *, offload->actions), offload->actions_len, &flow->mega_ufid, &info, NULL); - ovs_mutex_unlock(&pmd->dp->port_mutex); + ovs_rwlock_unlock(&pmd->dp->port_rwlock); netdev_close(port); if (ret) { @@ -2825,9 +2825,9 @@ dp_offload_flush(struct dp_offload_thread_item *item) { struct dp_offload_flush_item *flush = &item->data->flush; - ovs_mutex_lock(&flush->dp->port_mutex); + ovs_rwlock_rdlock(&flush->dp->port_rwlock); netdev_flow_flush(flush->netdev); - ovs_mutex_unlock(&flush->dp->port_mutex); + ovs_rwlock_unlock(&flush->dp->port_rwlock); ovs_barrier_block(flush->barrier); @@ -3006,7 +3006,7 @@ dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, * complete its work. As the flush order will only be * enqueued after existing offload requests, those previous * offload requests must be processed, which requires being - * able to lock the 'port_mutex' from the offload thread. + * able to lock the 'port_rwlock' from the offload thread. * * Flow offload flush is done when a port is being deleted. * Right after this call executes, the offload API is disabled @@ -3016,7 +3016,7 @@ dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, static void dp_netdev_offload_flush(struct dp_netdev *dp, struct dp_netdev_port *port) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_WRLOCK(dp->port_rwlock) { /* The flush mutex only serves to protect the static memory barrier. * The memory barrier needs to go beyond the function scope as @@ -3034,7 +3034,7 @@ dp_netdev_offload_flush(struct dp_netdev *dp, return; } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_lock(&flush_mutex); /* This thread and the offload thread. */ @@ -3052,7 +3052,7 @@ dp_netdev_offload_flush(struct dp_netdev *dp, * Some offload provider (e.g. DPDK) keeps a netdev reference with * the offload data. If this reference is not closed, the netdev is * kept indefinitely. */ - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); ovs_barrier_block(&barrier); ovs_barrier_destroy(&barrier); @@ -3106,7 +3106,7 @@ dpif_netdev_port_dump_next(const struct dpif *dpif, void *state_, struct hmap_node *node; int retval; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); node = hmap_at_position(&dp->ports, &state->position); if (node) { struct dp_netdev_port *port; @@ -3123,7 +3123,7 @@ dpif_netdev_port_dump_next(const struct dpif *dpif, void *state_, } else { retval = EOF; } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return retval; } @@ -3574,24 +3574,24 @@ dpif_netdev_get_flow_offload_status(const struct dp_netdev *dp, return false; } ofpbuf_use_stack(&buf, &act_buf, sizeof act_buf); - /* Taking a global 'port_mutex' to fulfill thread safety + /* Taking a global 'port_rwlock' to fulfill thread safety * restrictions regarding netdev port mapping. * * XXX: Main thread will try to pause/stop all revalidators during datapath * reconfiguration via datapath purge callback (dp_purge_cb) while - * holding 'dp->port_mutex'. So we're not waiting for mutex here. - * Otherwise, deadlock is possible, bcause revalidators might sleep + * rw-holding 'dp->port_rwlock'. So we're not waiting for lock here. + * Otherwise, deadlock is possible, because revalidators might sleep * waiting for the main thread to release the lock and main thread * will wait for them to stop processing. * This workaround might make statistics less accurate. Especially * for flow deletion case, since there will be no other attempt. */ - if (!ovs_mutex_trylock(&dp->port_mutex)) { + if (!ovs_rwlock_tryrdlock(&dp->port_rwlock)) { ret = netdev_flow_get(netdev, &match, &actions, &netdev_flow->mega_ufid, stats, attrs, &buf); /* Storing statistics and attributes from the last request for * later use on mutex contention. */ dp_netdev_flow_set_last_stats_attrs(netdev_flow, stats, attrs, ret); - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); } else { dp_netdev_flow_get_last_stats_attrs(netdev_flow, stats, attrs, &ret); if (!ret && !attrs->dp_layer) { @@ -4461,7 +4461,7 @@ dpif_netdev_offload_stats_get(struct dpif *dpif, nb_offloads = 0; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); HMAP_FOR_EACH (port, node, &dp->ports) { uint64_t port_nb_offloads = 0; @@ -4470,7 +4470,7 @@ dpif_netdev_offload_stats_get(struct dpif *dpif, nb_offloads += port_nb_offloads; } } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); atomic_read_relaxed(&dp_offload_thread.enqueued_item, &stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value); @@ -4780,7 +4780,7 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, const char *affinity_list = smap_get(cfg, "pmd-rxq-affinity"); bool emc_enabled = smap_get_bool(cfg, "emc-enable", true); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); error = get_port_by_number(dp, port_no, &port); if (error) { goto unlock; @@ -4834,7 +4834,7 @@ dpif_netdev_port_set_config(struct dpif *dpif, odp_port_t port_no, dp_netdev_request_reconfigure(dp); unlock: - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); return error; } @@ -5324,7 +5324,8 @@ compare_rxq_cycles(const void *a, const void *b) * The function doesn't touch the pmd threads, it just stores the assignment * in the 'pmd' member of each rxq. */ static void -rxq_scheduling(struct dp_netdev *dp, bool pinned) OVS_REQUIRES(dp->port_mutex) +rxq_scheduling(struct dp_netdev *dp, bool pinned) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; struct rr_numa_list rr; @@ -5468,7 +5469,7 @@ reload_affected_pmds(struct dp_netdev *dp) static void reconfigure_pmd_threads(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_pmd_thread *pmd; struct ovs_numa_dump *pmd_cores; @@ -5566,7 +5567,7 @@ static void pmd_remove_stale_ports(struct dp_netdev *dp, struct dp_netdev_pmd_thread *pmd) OVS_EXCLUDED(pmd->port_mutex) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct rxq_poll *poll, *poll_next; struct tx_port *tx, *tx_next; @@ -5596,7 +5597,7 @@ pmd_remove_stale_ports(struct dp_netdev *dp, * rxqs and assigns all rxqs/txqs to pmd threads. */ static void reconfigure_datapath(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct hmapx busy_threads = HMAPX_INITIALIZER(&busy_threads); struct dp_netdev_pmd_thread *pmd; @@ -5780,7 +5781,7 @@ reconfigure_datapath(struct dp_netdev *dp) /* Returns true if one of the netdevs in 'dp' requires a reconfiguration */ static bool ports_require_restart(const struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; @@ -5831,7 +5832,7 @@ variance(uint64_t a[], int n) static bool get_dry_run_variance(struct dp_netdev *dp, uint32_t *core_list, uint32_t num_pmds, uint64_t *predicted_variance) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_port *port; struct dp_netdev_pmd_thread *pmd; @@ -5954,7 +5955,7 @@ cleanup: * better distribution of load on PMDs. */ static bool pmd_rebalance_dry_run(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_RDLOCK(dp->port_rwlock) { struct dp_netdev_pmd_thread *pmd; uint64_t *curr_pmd_usage; @@ -6049,7 +6050,7 @@ dpif_netdev_run(struct dpif *dpif) long long int now = time_msec(); struct dp_netdev_pmd_thread *pmd; - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); non_pmd = dp_netdev_get_pmd(dp, NON_PMD_CORE_ID); if (non_pmd) { ovs_mutex_lock(&dp->non_pmd_mutex); @@ -6121,7 +6122,7 @@ dpif_netdev_run(struct dpif *dpif) if (dp_netdev_is_reconf_required(dp) || ports_require_restart(dp)) { reconfigure_datapath(dp); } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); tnl_neigh_cache_run(); tnl_port_map_run(); @@ -6141,7 +6142,7 @@ dpif_netdev_wait(struct dpif *dpif) struct dp_netdev *dp = get_dp_netdev(dpif); ovs_mutex_lock(&dp_netdev_mutex); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_rdlock(&dp->port_rwlock); HMAP_FOR_EACH (port, node, &dp->ports) { netdev_wait_reconf_required(port->netdev); if (!netdev_is_pmd(port->netdev)) { @@ -6152,7 +6153,7 @@ dpif_netdev_wait(struct dpif *dpif) } } } - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_unlock(&dp_netdev_mutex); seq_wait(tnl_conf_seq, dp->last_tnl_conf_seq); } @@ -6769,7 +6770,7 @@ dp_netdev_get_pmd(struct dp_netdev *dp, unsigned core_id) /* Sets the 'struct dp_netdev_pmd_thread' for non-pmd threads. */ static void dp_netdev_set_nonpmd(struct dp_netdev *dp) - OVS_REQUIRES(dp->port_mutex) + OVS_REQ_WRLOCK(dp->port_rwlock) { struct dp_netdev_pmd_thread *non_pmd; @@ -8832,7 +8833,7 @@ dpif_dummy_change_port_number(struct unixctl_conn *conn, int argc OVS_UNUSED, ovs_refcount_ref(&dp->ref_cnt); ovs_mutex_unlock(&dp_netdev_mutex); - ovs_mutex_lock(&dp->port_mutex); + ovs_rwlock_wrlock(&dp->port_rwlock); if (get_port_by_name(dp, argv[2], &port)) { unixctl_command_reply_error(conn, "unknown port"); goto exit; @@ -8861,7 +8862,7 @@ dpif_dummy_change_port_number(struct unixctl_conn *conn, int argc OVS_UNUSED, unixctl_command_reply(conn, NULL); exit: - ovs_mutex_unlock(&dp->port_mutex); + ovs_rwlock_unlock(&dp->port_rwlock); dp_netdev_unref(dp); } diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index 13e017ef8..70e6843f5 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -44,8 +44,8 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(100, 5); * For example, simultaneous call of 'netdev_reconfigure()' for the same * 'netdev' is forbidden. * - * For current implementation all above restrictions could be fulfilled by - * taking the datapath 'port_mutex' in lib/dpif-netdev.c. */ + * For current implementation all above restrictions are fulfilled by + * read-locking the datapath 'port_rwlock' in lib/dpif-netdev.c. */ /* * A mapping from ufid to dpdk rte_flow. From patchwork Wed Jun 9 13:09:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489881 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=FxVCm3Lg; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=Ziui2JhW; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SF65tfgz9sX5 for ; Wed, 9 Jun 2021 23:12:10 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 927D96F785; Wed, 9 Jun 2021 13:12:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8Hb9y2rrbyDu; Wed, 9 Jun 2021 13:12:04 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 2C78560A49; Wed, 9 Jun 2021 13:12:01 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 64EA3C002E; Wed, 9 Jun 2021 13:11:59 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 35E07C0031 for ; Wed, 9 Jun 2021 13:11:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id B6AB14058F for ; Wed, 9 Jun 2021 13:10:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="FxVCm3Lg"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="Ziui2JhW" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WoKX0ursJclR for ; Wed, 9 Jun 2021 13:10:30 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp4.osuosl.org (Postfix) with ESMTPS id 42C14405F4 for ; Wed, 9 Jun 2021 13:10:27 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 977D92627; Wed, 9 Jun 2021 09:10:26 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=6/1sFNGZxM6vP D3IF7eTqNuIQdTuP9Ny1Q/JzsusnHg=; b=FxVCm3Lgmn9rW87fWqxVPibE7MJuP 4Etxu7/HuZOGNITDFE+BjDzZecVVR4ZrCJcGz1UzVknx1RjJpsfYxXetXEPMFGLS bb5Qh4MFJJCIFBZ7QGWyVilwY74j1giiBZ8EUYzM33eY06YzvXaTzSLTdOv9mchL OUBeLsLhm6Wya89xSewosnIePHdMt7LykWTsv2bM05ZLlP53bWzqB1BvsugSwQ6s jpVcvltFMjfEJnFtBIi6mHzoFK5cXPj1vN/ii3mEOcB13fwJZXZRY7eFjUPstMhS 0JTtPFWFYlPclmpyFlxHalyKC+Nyh8Ea4E/fdfnqyqQo7quJEVYL9BEGw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=6/1sFNGZxM6vPD3IF7eTqNuIQdTuP9Ny1Q/JzsusnHg=; b=Ziui2JhW qkSy+2XFAd+BWKnq0PC48khUKLKTeW/2fyfhm2W0+nKB7fDVAazDKZvDE5aDjJ6v k7INB1s35xDnv5Edi+Wtr3oKFirs0JJm6rPzr+ftPUuFEIMn94B/T/UZsfew+PTK mxSr24zSQoocs+2N4l9D6nfRx1psvoMPecAAAOCHr5r4LxaRcNaH20PoMfDhuNfm S26Y87SPQgFI2ycOkQl+WLF/PXcobgDPP5HMZ4DyJK+SNemvqKx5bttEUjHGAtwP SbDx+IyrWfslqOjn5ucEF9EDFP6zeiLIW9FprJaBuqEAAkGW0JWVtZcd5Qzr7jlj /Mc9VItE3TPo9g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epkeelhfffgfekudfggfduleeukefghefgtdfhvdekuefhffeutdetveevudeivdfgnecu ffhomhgrihhnpehmvggrnhdrthhothgrlhdpshhtugguvghvrdhtohhtrghlnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepghhrihhvvgesuhdv heeirdhnvght X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:25 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:34 +0200 Message-Id: <6b6fbd481cf85e4209ae47a6226a0e1841188801.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 26/27] dpif-netdev: Use one or more offload threads X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Read the user configuration in the netdev-offload module to modify the number of threads used to manage hardware offload requests. This allows processing insertion, deletion and modification concurrently. The offload thread structure was modified to contain all needed elements. This structure is multiplied by the number of requested threads and used separately. Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/dpif-netdev.c | 304 +++++++++++++++++++++++++------------- lib/netdev-offload-dpdk.c | 7 +- 2 files changed, 204 insertions(+), 107 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index cc7a979d7..73dec57c4 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -460,25 +460,47 @@ struct dp_offload_thread_item { }; struct dp_offload_thread { - struct mpsc_queue queue; - atomic_uint64_t enqueued_item; - struct cmap megaflow_to_mark; - struct cmap mark_to_flow; - struct mov_avg_cma cma; - struct mov_avg_ema ema; + PADDED_MEMBERS(CACHE_LINE_SIZE, + struct mpsc_queue queue; + atomic_uint64_t enqueued_item; + struct cmap megaflow_to_mark; + struct cmap mark_to_flow; + struct mov_avg_cma cma; + struct mov_avg_ema ema; + ); }; +static struct dp_offload_thread *dp_offload_threads; +static void *dp_netdev_flow_offload_main(void *arg); -static struct dp_offload_thread dp_offload_thread = { - .queue = MPSC_QUEUE_INITIALIZER(&dp_offload_thread.queue), - .megaflow_to_mark = CMAP_INITIALIZER, - .mark_to_flow = CMAP_INITIALIZER, - .enqueued_item = ATOMIC_VAR_INIT(0), - .cma = MOV_AVG_CMA_INITIALIZER, - .ema = MOV_AVG_EMA_INITIALIZER(100), -}; +static void +dp_netdev_offload_init(void) +{ + static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; + unsigned int nb_offload_thread = netdev_offload_thread_nb(); + unsigned int tid; + + if (!ovsthread_once_start(&once)) { + return; + } + + dp_offload_threads = xcalloc(nb_offload_thread, + sizeof *dp_offload_threads); -static struct ovsthread_once offload_thread_once - = OVSTHREAD_ONCE_INITIALIZER; + for (tid = 0; tid < nb_offload_thread; tid++) { + struct dp_offload_thread *thread; + + thread = &dp_offload_threads[tid]; + mpsc_queue_init(&thread->queue); + cmap_init(&thread->megaflow_to_mark); + cmap_init(&thread->mark_to_flow); + atomic_init(&thread->enqueued_item, 0); + mov_avg_cma_init(&thread->cma); + mov_avg_ema_init(&thread->ema, 100); + ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, thread); + } + + ovsthread_once_done(&once); +} #define XPS_TIMEOUT 500000LL /* In microseconds. */ @@ -2478,11 +2500,12 @@ megaflow_to_mark_associate(const ovs_u128 *mega_ufid, uint32_t mark) { size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data = xzalloc(sizeof(*data)); + unsigned int tid = netdev_offload_thread_id(); data->mega_ufid = *mega_ufid; data->mark = mark; - cmap_insert(&dp_offload_thread.megaflow_to_mark, + cmap_insert(&dp_offload_threads[tid].megaflow_to_mark, CONST_CAST(struct cmap_node *, &data->node), hash); } @@ -2492,11 +2515,12 @@ megaflow_to_mark_disassociate(const ovs_u128 *mega_ufid) { size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data; + unsigned int tid = netdev_offload_thread_id(); CMAP_FOR_EACH_WITH_HASH (data, node, hash, - &dp_offload_thread.megaflow_to_mark) { + &dp_offload_threads[tid].megaflow_to_mark) { if (ovs_u128_equals(*mega_ufid, data->mega_ufid)) { - cmap_remove(&dp_offload_thread.megaflow_to_mark, + cmap_remove(&dp_offload_threads[tid].megaflow_to_mark, CONST_CAST(struct cmap_node *, &data->node), hash); ovsrcu_postpone(free, data); return; @@ -2512,9 +2536,10 @@ megaflow_to_mark_find(const ovs_u128 *mega_ufid) { size_t hash = dp_netdev_flow_hash(mega_ufid); struct megaflow_to_mark_data *data; + unsigned int tid = netdev_offload_thread_id(); CMAP_FOR_EACH_WITH_HASH (data, node, hash, - &dp_offload_thread.megaflow_to_mark) { + &dp_offload_threads[tid].megaflow_to_mark) { if (ovs_u128_equals(*mega_ufid, data->mega_ufid)) { return data->mark; } @@ -2529,9 +2554,10 @@ megaflow_to_mark_find(const ovs_u128 *mega_ufid) static void mark_to_flow_associate(const uint32_t mark, struct dp_netdev_flow *flow) { + unsigned int tid = netdev_offload_thread_id(); dp_netdev_flow_ref(flow); - cmap_insert(&dp_offload_thread.mark_to_flow, + cmap_insert(&dp_offload_threads[tid].mark_to_flow, CONST_CAST(struct cmap_node *, &flow->mark_node), hash_int(mark, 0)); flow->mark = mark; @@ -2543,10 +2569,11 @@ mark_to_flow_associate(const uint32_t mark, struct dp_netdev_flow *flow) static bool flow_mark_has_no_ref(uint32_t mark) { + unsigned int tid = netdev_offload_thread_id(); struct dp_netdev_flow *flow; CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash_int(mark, 0), - &dp_offload_thread.mark_to_flow) { + &dp_offload_threads[tid].mark_to_flow) { if (flow->mark == mark) { return false; } @@ -2562,6 +2589,7 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, const char *dpif_type_str = dpif_normalize_type(pmd->dp->class->type); struct cmap_node *mark_node = CONST_CAST(struct cmap_node *, &flow->mark_node); + unsigned int tid = netdev_offload_thread_id(); uint32_t mark = flow->mark; int ret = 0; @@ -2571,7 +2599,8 @@ mark_to_flow_disassociate(struct dp_netdev_pmd_thread *pmd, return EINVAL; } - cmap_remove(&dp_offload_thread.mark_to_flow, mark_node, hash_int(mark, 0)); + cmap_remove(&dp_offload_threads[tid].mark_to_flow, + mark_node, hash_int(mark, 0)); flow->mark = INVALID_FLOW_MARK; /* @@ -2607,10 +2636,18 @@ static void flow_mark_flush(struct dp_netdev_pmd_thread *pmd) { struct dp_netdev_flow *flow; + unsigned int tid; - CMAP_FOR_EACH (flow, mark_node, &dp_offload_thread.mark_to_flow) { - if (flow->pmd_id == pmd->core_id) { - queue_netdev_flow_del(pmd, flow); + if (dp_offload_threads == NULL) { + return; + } + + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + CMAP_FOR_EACH (flow, mark_node, + &dp_offload_threads[tid].mark_to_flow) { + if (flow->pmd_id == pmd->core_id) { + queue_netdev_flow_del(pmd, flow); + } } } } @@ -2620,12 +2657,21 @@ mark_to_flow_find(const struct dp_netdev_pmd_thread *pmd, const uint32_t mark) { struct dp_netdev_flow *flow; + unsigned int tid; + size_t hash; - CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash_int(mark, 0), - &dp_offload_thread.mark_to_flow) { - if (flow->mark == mark && flow->pmd_id == pmd->core_id && - flow->dead == false) { - return flow; + if (dp_offload_threads == NULL) { + return NULL; + } + + hash = hash_int(mark, 0); + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + CMAP_FOR_EACH_WITH_HASH (flow, mark_node, hash, + &dp_offload_threads[tid].mark_to_flow) { + if (flow->mark == mark && flow->pmd_id == pmd->core_id && + flow->dead == false) { + return flow; + } } } @@ -2690,10 +2736,25 @@ dp_netdev_free_offload(struct dp_offload_thread_item *offload) } static void -dp_netdev_append_offload(struct dp_offload_thread_item *offload) +dp_netdev_append_offload(struct dp_offload_thread_item *offload, + unsigned int tid) +{ + dp_netdev_offload_init(); + + mpsc_queue_insert(&dp_offload_threads[tid].queue, &offload->node); + atomic_count_inc64(&dp_offload_threads[tid].enqueued_item); +} + +static void +dp_netdev_offload_flow_enqueue(struct dp_offload_thread_item *item) { - mpsc_queue_insert(&dp_offload_thread.queue, &offload->node); - atomic_count_inc64(&dp_offload_thread.enqueued_item); + struct dp_offload_flow_item *flow_offload = &item->data->flow; + unsigned int tid; + + ovs_assert(item->type == DP_OFFLOAD_FLOW); + + tid = netdev_offload_ufid_to_thread_id(flow_offload->flow->mega_ufid); + dp_netdev_append_offload(item, tid); } static int @@ -2831,8 +2892,8 @@ dp_offload_flush(struct dp_offload_thread_item *item) ovs_barrier_block(flush->barrier); - /* Allow the other thread to take again the port lock, before - * continuing offload operations in this thread. + /* Allow the initiator thread to take again the port lock, + * before continuing offload operations in this thread. */ ovs_barrier_block(flush->barrier); } @@ -2842,8 +2903,9 @@ dp_offload_flush(struct dp_offload_thread_item *item) #define DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US (10 * 1000) /* 10 ms */ static void * -dp_netdev_flow_offload_main(void *data OVS_UNUSED) +dp_netdev_flow_offload_main(void *arg) { + struct dp_offload_thread *ofl_thread = arg; struct dp_offload_thread_item *offload; struct mpsc_queue_node *node; struct mpsc_queue *queue; @@ -2852,7 +2914,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) long long int now; uint64_t backoff; - queue = &dp_offload_thread.queue; + queue = &ofl_thread->queue; mpsc_queue_acquire(queue); while (true) { @@ -2867,7 +2929,7 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) next_rcu = time_usec() + DP_NETDEV_OFFLOAD_QUIESCE_INTERVAL_US; MPSC_QUEUE_FOR_EACH_POP (node, queue) { offload = CONTAINER_OF(node, struct dp_offload_thread_item, node); - atomic_count_dec64(&dp_offload_thread.enqueued_item); + atomic_count_dec64(&ofl_thread->enqueued_item); switch (offload->type) { case DP_OFFLOAD_FLOW: @@ -2883,8 +2945,8 @@ dp_netdev_flow_offload_main(void *data OVS_UNUSED) now = time_usec(); latency_us = now - offload->timestamp; - mov_avg_cma_update(&dp_offload_thread.cma, latency_us); - mov_avg_ema_update(&dp_offload_thread.ema, latency_us); + mov_avg_cma_update(&ofl_thread->cma, latency_us); + mov_avg_ema_update(&ofl_thread->ema, latency_us); dp_netdev_free_offload(offload); @@ -2908,16 +2970,10 @@ queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, { struct dp_offload_thread_item *offload; - if (ovsthread_once_start(&offload_thread_once)) { - mpsc_queue_init(&dp_offload_thread.queue); - ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); - ovsthread_once_done(&offload_thread_once); - } - offload = dp_netdev_alloc_flow_offload(pmd, flow, DP_NETDEV_FLOW_OFFLOAD_OP_DEL); offload->timestamp = pmd->ctx.now; - dp_netdev_append_offload(offload); + dp_netdev_offload_flow_enqueue(offload); } static void @@ -2933,12 +2989,6 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, return; } - if (ovsthread_once_start(&offload_thread_once)) { - mpsc_queue_init(&dp_offload_thread.queue); - ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); - ovsthread_once_done(&offload_thread_once); - } - if (flow->mark != INVALID_FLOW_MARK) { op = DP_NETDEV_FLOW_OFFLOAD_OP_MOD; } else { @@ -2952,7 +3002,7 @@ queue_netdev_flow_put(struct dp_netdev_pmd_thread *pmd, flow_offload->actions_len = actions_len; item->timestamp = pmd->ctx.now; - dp_netdev_append_offload(item); + dp_netdev_offload_flow_enqueue(item); } static void @@ -2981,25 +3031,24 @@ dp_netdev_offload_flush_enqueue(struct dp_netdev *dp, struct netdev *netdev, struct ovs_barrier *barrier) { - struct dp_offload_thread_item *item; - struct dp_offload_flush_item *flush; + unsigned int tid; + long long int now_us = time_usec(); - if (ovsthread_once_start(&offload_thread_once)) { - mpsc_queue_init(&dp_offload_thread.queue); - ovs_thread_create("hw_offload", dp_netdev_flow_offload_main, NULL); - ovsthread_once_done(&offload_thread_once); - } + for (tid = 0; tid < netdev_offload_thread_nb(); tid++) { + struct dp_offload_thread_item *item; + struct dp_offload_flush_item *flush; - item = xmalloc(sizeof *item + sizeof *flush); - item->type = DP_OFFLOAD_FLUSH; - item->timestamp = time_usec(); + item = xmalloc(sizeof *item + sizeof *flush); + item->type = DP_OFFLOAD_FLUSH; + item->timestamp = now_us; - flush = &item->data->flush; - flush->dp = dp; - flush->netdev = netdev; - flush->barrier = barrier; + flush = &item->data->flush; + flush->dp = dp; + flush->netdev = netdev; + flush->barrier = barrier; - dp_netdev_append_offload(item); + dp_netdev_append_offload(item, tid); + } } /* Blocking call that will wait on the offload thread to @@ -3018,13 +3067,17 @@ dp_netdev_offload_flush(struct dp_netdev *dp, struct dp_netdev_port *port) OVS_REQ_WRLOCK(dp->port_rwlock) { - /* The flush mutex only serves to protect the static memory barrier. + /* The flush mutex serves to exclude mutual access to the static + * barrier, and to prevent multiple flush orders to several threads. + * * The memory barrier needs to go beyond the function scope as - * the other thread can resume from blocking after this function + * the other threads can resume from blocking after this function * already finished. - * As the barrier is made static, then it will be shared by - * calls to this function, and it needs to be protected from - * concurrent use. + * + * Additionally, because the flush operation is blocking, it would + * deadlock if multiple offload threads were blocking on several + * different barriers. Only allow a single flush order in the offload + * queue at a time. */ static struct ovs_mutex flush_mutex = OVS_MUTEX_INITIALIZER; static struct ovs_barrier barrier OVS_GUARDED_BY(flush_mutex); @@ -3037,8 +3090,8 @@ dp_netdev_offload_flush(struct dp_netdev *dp, ovs_rwlock_unlock(&dp->port_rwlock); ovs_mutex_lock(&flush_mutex); - /* This thread and the offload thread. */ - ovs_barrier_init(&barrier, 2); + /* This thread and the offload threads. */ + ovs_barrier_init(&barrier, 1 + netdev_offload_thread_nb()); netdev = netdev_ref(port->netdev); dp_netdev_offload_flush_enqueue(dp, netdev, &barrier); @@ -3046,7 +3099,7 @@ dp_netdev_offload_flush(struct dp_netdev *dp, netdev_close(netdev); /* Take back the datapath port lock before allowing the offload - * thread to proceed further. The port deletion must complete first, + * threads to proceed further. The port deletion must complete first, * to ensure no further offloads are inserted after the flush. * * Some offload provider (e.g. DPDK) keeps a netdev reference with @@ -4433,60 +4486,99 @@ dpif_netdev_offload_stats_get(struct dpif *dpif, DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN, DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV, }; - const char *names[] = { + struct { + const char *name; + uint64_t total; + } hwol_stats[] = { [DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED] = - " Enqueued offloads", + { " Enqueued offloads", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_INSERTED] = - " Inserted offloads", + { " Inserted offloads", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN] = - " Cumulative Average latency (us)", + { " Cumulative Average latency (us)", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV] = - " Cumulative Latency stddev (us)", + { " Cumulative Latency stddev (us)", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN] = - " Exponential Average latency (us)", + { " Exponential Average latency (us)", 0 }, [DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV] = - " Exponential Latency stddev (us)", + { " Exponential Latency stddev (us)", 0 }, }; struct dp_netdev *dp = get_dp_netdev(dpif); struct dp_netdev_port *port; - uint64_t nb_offloads; + unsigned int nb_thread; + uint64_t *port_nb_offloads; + uint64_t *nb_offloads; + unsigned int tid; size_t i; if (!netdev_is_flow_api_enabled()) { return EINVAL; } - stats->size = ARRAY_SIZE(names); + nb_thread = netdev_offload_thread_nb(); + /* nb_thread counters for the overall total as well. */ + stats->size = ARRAY_SIZE(hwol_stats) * (nb_thread + 1); stats->counters = xcalloc(stats->size, sizeof *stats->counters); - nb_offloads = 0; + nb_offloads = xcalloc(nb_thread, sizeof *nb_offloads); + port_nb_offloads = xcalloc(nb_thread, sizeof *port_nb_offloads); ovs_rwlock_rdlock(&dp->port_rwlock); HMAP_FOR_EACH (port, node, &dp->ports) { - uint64_t port_nb_offloads = 0; - + memset(port_nb_offloads, 0, nb_thread * sizeof *port_nb_offloads); /* Do not abort on read error from a port, just report 0. */ - if (!netdev_flow_get_n_flows(port->netdev, &port_nb_offloads)) { - nb_offloads += port_nb_offloads; + if (!netdev_flow_get_n_flows(port->netdev, port_nb_offloads)) { + for (i = 0; i < nb_thread; i++) { + nb_offloads[i] += port_nb_offloads[i]; + } } } ovs_rwlock_unlock(&dp->port_rwlock); - atomic_read_relaxed(&dp_offload_thread.enqueued_item, - &stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED].value); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED].value = nb_offloads; - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].value = - mov_avg_cma(&dp_offload_thread.cma); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV].value = - mov_avg_cma_std_dev(&dp_offload_thread.cma); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN].value = - mov_avg_ema(&dp_offload_thread.ema); - stats->counters[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV].value = - mov_avg_ema_std_dev(&dp_offload_thread.ema); - - for (i = 0; i < ARRAY_SIZE(names); i++) { + free(port_nb_offloads); + + for (tid = 0; tid < nb_thread; tid++) { + uint64_t counts[ARRAY_SIZE(hwol_stats)]; + size_t idx = ((tid + 1) * ARRAY_SIZE(hwol_stats)); + + memset(counts, 0, sizeof counts); + counts[DP_NETDEV_HW_OFFLOADS_STATS_INSERTED] = nb_offloads[tid]; + if (dp_offload_threads != NULL) { + atomic_read_relaxed(&dp_offload_threads[tid].enqueued_item, + &counts[DP_NETDEV_HW_OFFLOADS_STATS_ENQUEUED]); + + counts[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN] = + mov_avg_cma(&dp_offload_threads[tid].cma); + counts[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV] = + mov_avg_cma_std_dev(&dp_offload_threads[tid].cma); + + counts[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN] = + mov_avg_ema(&dp_offload_threads[tid].ema); + counts[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV] = + mov_avg_ema_std_dev(&dp_offload_threads[tid].ema); + } + + for (i = 0; i < ARRAY_SIZE(hwol_stats); i++) { + snprintf(stats->counters[idx + i].name, + sizeof(stats->counters[idx + i].name), + " [%3u] %s", tid, hwol_stats[i].name); + stats->counters[idx + i].value = counts[i]; + hwol_stats[i].total += counts[i]; + } + } + + free(nb_offloads); + + /* Do an average of the average for the aggregate. */ + hwol_stats[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_MEAN].total /= nb_thread; + hwol_stats[DP_NETDEV_HW_OFFLOADS_STATS_LAT_CMA_STDDEV].total /= nb_thread; + hwol_stats[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_MEAN].total /= nb_thread; + hwol_stats[DP_NETDEV_HW_OFFLOADS_STATS_LAT_EMA_STDDEV].total /= nb_thread; + + for (i = 0; i < ARRAY_SIZE(hwol_stats); i++) { snprintf(stats->counters[i].name, sizeof(stats->counters[i].name), - "%s", names[i]); + " Total %s", hwol_stats[i].name); + stats->counters[i].value = hwol_stats[i].total; } return 0; diff --git a/lib/netdev-offload-dpdk.c b/lib/netdev-offload-dpdk.c index 70e6843f5..f4452553b 100644 --- a/lib/netdev-offload-dpdk.c +++ b/lib/netdev-offload-dpdk.c @@ -59,6 +59,7 @@ struct ufid_to_rte_flow_data { bool actions_offloaded; struct dpif_flow_stats stats; struct ovs_mutex lock; + unsigned int creation_tid; bool dead; }; @@ -235,6 +236,7 @@ ufid_to_rte_flow_associate(struct netdev *netdev, const ovs_u128 *ufid, data->netdev = netdev_ref(netdev); data->rte_flow = rte_flow; data->actions_offloaded = actions_offloaded; + data->creation_tid = netdev_offload_thread_id(); ovs_mutex_init(&data->lock); cmap_insert(map, CONST_CAST(struct cmap_node *, &data->node), hash); @@ -1772,13 +1774,16 @@ netdev_offload_dpdk_flow_flush(struct netdev *netdev) { struct cmap *map = offload_data_map(netdev); struct ufid_to_rte_flow_data *data; + unsigned int tid = netdev_offload_thread_id(); if (!map) { return -1; } CMAP_FOR_EACH (data, node, map) { - netdev_offload_dpdk_flow_destroy(data); + if (data->creation_tid == tid) { + netdev_offload_dpdk_flow_destroy(data); + } } return 0; From patchwork Wed Jun 9 13:09:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gaetan Rivet X-Patchwork-Id: 1489882 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=u256.net header.i=@u256.net header.a=rsa-sha256 header.s=fm2 header.b=YU52z/Cn; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=messagingengine.com header.i=@messagingengine.com header.a=rsa-sha256 header.s=fm3 header.b=R0j4pZLd; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G0SFC0gYCz9sT6 for ; Wed, 9 Jun 2021 23:12:15 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id A5F746F829; Wed, 9 Jun 2021 13:12:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lcMByzAo9iIs; Wed, 9 Jun 2021 13:12:09 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 76EE26F817; Wed, 9 Jun 2021 13:12:04 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 34B14C000B; Wed, 9 Jun 2021 13:12:04 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 072E4C000B for ; Wed, 9 Jun 2021 13:12:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C72B54052D for ; Wed, 9 Jun 2021 13:10:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=u256.net header.b="YU52z/Cn"; dkim=pass (2048-bit key) header.d=messagingengine.com header.b="R0j4pZLd" Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ogUZnVqczSCv for ; Wed, 9 Jun 2021 13:10:32 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by smtp4.osuosl.org (Postfix) with ESMTPS id CFC7840586 for ; Wed, 9 Jun 2021 13:10:28 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 1DA1B1C0B; Wed, 9 Jun 2021 09:10:28 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 09 Jun 2021 09:10:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=u256.net; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=7yygEH8YeZuk3 IZW1P95i7SGo4gl/TishAwNIMNFykU=; b=YU52z/CnnciLAAPm+XlnwAc5PA9fB g/yI8YtYI9XJBnV+HjbFnzl+ypo9eP6rIcgjPIwukudo/t/ZTTyvwEfzZn23jMhX Tjnb/pQVqb0W62A32lyqB2d5QyfGhH3vb8tgNnMkg5Zt6zi6/NxkzMGJlnXD8JN1 oUYalAr7wMLuBR730wFyP6jya5MjkKNHW3aaXKXRfUr746Nz0hNEIQduiCvPSvHc QgCM/MY7oivTzodODiZeInSXd+yfPltNoILApaWxJNQKtaZ8TJaUI61MhYsu+1pM hEc8IzGydsP4NrVnl/tsynvXv8Z6fweHW5bM/j0+C/qCe5RKqF98oWLYA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=7yygEH8YeZuk3IZW1P95i7SGo4gl/TishAwNIMNFykU=; b=R0j4pZLd IdWubuslCHKRhNBgvISVsLmtYHOxn6H/c3gmiqwUpTPXXjFf210pqnywpLVT5h1n 16ueao0yIGCR3MZJCJThT5S3rHFyMPCgZExiGmMNE5tK1fzoCvjH/VK5s/YJQ7b8 FcwUzUFH/T71CiApp3zcx3XvsuWhX1rq9MrYgRQV5FyKa1/2J00gDmGC9ihlpENC /nltppmxx7aU2ydl5Idv8u5KuelGtelAEdxmo7QsipteCuAzKKmAkNKAUGuJ5nk2 17DoAqx7WpDI9z0Tp+gVvwYhtCwaSBcEGyJ2U5vrUp8Mt2b8Gqw5KHxg1Y905TAa y+dwTnhojeCmdw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrfeduuddgieefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefirggvthgr nhcutfhivhgvthcuoehgrhhivhgvsehuvdehiedrnhgvtheqnecuggftrfgrthhtvghrnh epgedtueehiefgieefffevhfehtedvuddtvdfhuefgueeffffhkeeigfejudfhgeeunecu ffhomhgrihhnpeguphgukhdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpehgrhhivhgvsehuvdehiedrnhgvth X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 9 Jun 2021 09:10:27 -0400 (EDT) From: Gaetan Rivet To: ovs-dev@openvswitch.org Date: Wed, 9 Jun 2021 15:09:35 +0200 Message-Id: <834a3efdeced3bec945892538b0f41db6c0f50df.1623234822.git.grive@u256.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Cc: Eli Britstein , Maxime Coquelin Subject: [ovs-dev] [PATCH v4 27/27] netdev-dpdk: Remove rte-flow API access locks X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The rte_flow DPDK API was made thread-safe [1] in release 20.11. Now that the DPDK offload provider in OVS is thread safe, remove the locks. [1]: http://mails.dpdk.org/archives/dev/2020-October/184251.html Signed-off-by: Gaetan Rivet Reviewed-by: Eli Britstein Reviewed-by: Maxime Coquelin --- lib/netdev-dpdk.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 9d8096668..c7ebeb4d5 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -5239,9 +5239,7 @@ netdev_dpdk_rte_flow_destroy(struct netdev *netdev, struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); int ret; - ovs_mutex_lock(&dev->mutex); ret = rte_flow_destroy(dev->port_id, rte_flow, error); - ovs_mutex_unlock(&dev->mutex); return ret; } @@ -5255,9 +5253,7 @@ netdev_dpdk_rte_flow_create(struct netdev *netdev, struct rte_flow *flow; struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); - ovs_mutex_lock(&dev->mutex); flow = rte_flow_create(dev->port_id, attr, items, actions, error); - ovs_mutex_unlock(&dev->mutex); return flow; } @@ -5285,9 +5281,7 @@ netdev_dpdk_rte_flow_query_count(struct netdev *netdev, } dev = netdev_dpdk_cast(netdev); - ovs_mutex_lock(&dev->mutex); ret = rte_flow_query(dev->port_id, rte_flow, actions, query, error); - ovs_mutex_unlock(&dev->mutex); return ret; }