From patchwork Tue Oct 6 14:54:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1377476 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4C5L7k2Ygfz9sSG for ; Wed, 7 Oct 2020 01:53:38 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id ED42B85CFF; Tue, 6 Oct 2020 14:53:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yaeDHgjnVUeF; Tue, 6 Oct 2020 14:53:31 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3B162858DA; Tue, 6 Oct 2020 14:53:31 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 015B6C1AD6; Tue, 6 Oct 2020 14:53:31 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 897BFC0051 for ; Tue, 6 Oct 2020 14:53:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7757E82346 for ; Tue, 6 Oct 2020 14:53:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rH06DqDqRBEs for ; Tue, 6 Oct 2020 14:53:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 3F13185910 for ; Tue, 6 Oct 2020 14:53:27 +0000 (UTC) IronPort-SDR: tEABpGruERcaJp5LPI3pL+Hz1eFZL3JaKr5Qr1lQoMnYPE5zr8BccuuBaYL8ju6fEDFJFkLtoF e/umwAiszaMg== X-IronPort-AV: E=McAfee;i="6000,8403,9765"; a="249228240" X-IronPort-AV: E=Sophos;i="5.77,343,1596524400"; d="scan'208";a="249228240" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Oct 2020 07:53:15 -0700 IronPort-SDR: e5NBXmc90O6D2jUluxYMmR2GueSsSg8s4lXau78ElodL8sXBdW2yAWgJm1hrmArzrsEDGFNGAx eIiG98xVxC/g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,343,1596524400"; d="scan'208";a="311246822" Received: from silpixa00399779.ir.intel.com (HELO silpixa00399779.ger.corp.intel.com) ([10.237.222.209]) by orsmga003.jf.intel.com with ESMTP; 06 Oct 2020 07:53:14 -0700 From: Harry van Haaren To: ovs-dev@openvswitch.org Date: Tue, 6 Oct 2020 15:54:31 +0100 Message-Id: <20201006145437.35124-3-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201006145437.35124-1-harry.van.haaren@intel.com> References: <20201006145437.35124-1-harry.van.haaren@intel.com> Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH 2/8] dpif-netdev: refactor emc and smc into dfc header X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit moves the EMC and SMC implementations and data- structures out of the very large dpif-netdev.c file, and into its own header file. No functional changes are made here, only movement of code between files. Signed-off-by: Harry van Haaren --- lib/automake.mk | 1 + lib/dpif-netdev-private-dfc.h | 204 ++++++++++++++++++++++++++++++++++ lib/dpif-netdev.c | 169 +--------------------------- 3 files changed, 206 insertions(+), 168 deletions(-) create mode 100644 lib/dpif-netdev-private-dfc.h diff --git a/lib/automake.mk b/lib/automake.mk index 380a67228..9b0cdeb54 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -111,6 +111,7 @@ lib_libopenvswitch_la_SOURCES = \ lib/dpif-netdev-lookup-generic.c \ lib/dpif-netdev.c \ lib/dpif-netdev.h \ + lib/dpif-netdev-private-dfc.h \ lib/dpif-netdev-private.h \ lib/dpif-netdev-perf.c \ lib/dpif-netdev-perf.h \ diff --git a/lib/dpif-netdev-private-dfc.h b/lib/dpif-netdev-private-dfc.h new file mode 100644 index 000000000..f3c1aa338 --- /dev/null +++ b/lib/dpif-netdev-private-dfc.h @@ -0,0 +1,204 @@ +/* + * Copyright (c) 2008, 2009, 2010, 2011, 2012, 2013, 2015 Nicira, Inc. + * Copyright (c) 2019, 2020 Intel Corperation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef DPIF_NETDEV_PRIVATE_DFC_H +#define DPIF_NETDEV_PRIVATE_DFC_H 1 + +#include +#include + +#include "dpif.h" +#include "cmap.h" + +#ifdef __cplusplus +extern "C" { +#endif + +/* EMC cache and SMC cache compose the datapath flow cache (DFC) + * + * Exact match cache for frequently used flows + * + * The cache uses a 32-bit hash of the packet (which can be the RSS hash) to + * search its entries for a miniflow that matches exactly the miniflow of the + * packet. It stores the 'dpcls_rule' (rule) that matches the miniflow. + * + * A cache entry holds a reference to its 'dp_netdev_flow'. + * + * A miniflow with a given hash can be in one of EM_FLOW_HASH_SEGS different + * entries. The 32-bit hash is split into EM_FLOW_HASH_SEGS values (each of + * them is EM_FLOW_HASH_SHIFT bits wide and the remainder is thrown away). Each + * value is the index of a cache entry where the miniflow could be. + * + * + * Signature match cache (SMC) + * + * This cache stores a 16-bit signature for each flow without storing keys, and + * stores the corresponding 16-bit flow_table index to the 'dp_netdev_flow'. + * Each flow thus occupies 32bit which is much more memory efficient than EMC. + * SMC uses a set-associative design that each bucket contains + * SMC_ENTRY_PER_BUCKET number of entries. + * Since 16-bit flow_table index is used, if there are more than 2^16 + * dp_netdev_flow, SMC will miss them that cannot be indexed by a 16-bit value. + * + * + * Thread-safety + * ============= + * + * Each pmd_thread has its own private exact match cache. + * If dp_netdev_input is not called from a pmd thread, a mutex is used. + */ + +#define EM_FLOW_HASH_SHIFT 13 +#define EM_FLOW_HASH_ENTRIES (1u << EM_FLOW_HASH_SHIFT) +#define EM_FLOW_HASH_MASK (EM_FLOW_HASH_ENTRIES - 1) +#define EM_FLOW_HASH_SEGS 2 + +/* SMC uses a set-associative design. A bucket contains a set of entries that + * a flow item can occupy. For now, it uses one hash function rather than two + * as for the EMC design. */ +#define SMC_ENTRY_PER_BUCKET 4 +#define SMC_ENTRIES (1u << 20) +#define SMC_BUCKET_CNT (SMC_ENTRIES / SMC_ENTRY_PER_BUCKET) +#define SMC_MASK (SMC_BUCKET_CNT - 1) + +/* Default EMC insert probability is 1 / DEFAULT_EM_FLOW_INSERT_INV_PROB */ +#define DEFAULT_EM_FLOW_INSERT_INV_PROB 100 +#define DEFAULT_EM_FLOW_INSERT_MIN (UINT32_MAX / \ + DEFAULT_EM_FLOW_INSERT_INV_PROB) + +struct emc_entry { + struct dp_netdev_flow *flow; + struct netdev_flow_key key; /* key.hash used for emc hash value. */ +}; + +struct emc_cache { + struct emc_entry entries[EM_FLOW_HASH_ENTRIES]; + int sweep_idx; /* For emc_cache_slow_sweep(). */ +}; + +struct smc_bucket { + uint16_t sig[SMC_ENTRY_PER_BUCKET]; + uint16_t flow_idx[SMC_ENTRY_PER_BUCKET]; +}; + +/* Signature match cache, differentiate from EMC cache */ +struct smc_cache { + struct smc_bucket buckets[SMC_BUCKET_CNT]; +}; + +struct dfc_cache { + struct emc_cache emc_cache; + struct smc_cache smc_cache; +}; + +/* Iterate in the exact match cache through every entry that might contain a + * miniflow with hash 'HASH'. */ +#define EMC_FOR_EACH_POS_WITH_HASH(EMC, CURRENT_ENTRY, HASH) \ + for (uint32_t i__ = 0, srch_hash__ = (HASH); \ + (CURRENT_ENTRY) = &(EMC)->entries[srch_hash__ & EM_FLOW_HASH_MASK], \ + i__ < EM_FLOW_HASH_SEGS; \ + i__++, srch_hash__ >>= EM_FLOW_HASH_SHIFT) + +/* These cannot be moved to .h yet as they dereference dp_netdev_flow */ +static inline bool emc_entry_alive(struct emc_entry *ce); +static void emc_clear_entry(struct emc_entry *ce); + +static void +smc_clear_entry(struct smc_bucket *b, int idx) +{ + b->flow_idx[idx] = UINT16_MAX; +} + +static void +emc_cache_init(struct emc_cache *flow_cache) +{ + int i; + + flow_cache->sweep_idx = 0; + for (i = 0; i < ARRAY_SIZE(flow_cache->entries); i++) { + flow_cache->entries[i].flow = NULL; + flow_cache->entries[i].key.hash = 0; + flow_cache->entries[i].key.len = sizeof(struct miniflow); + flowmap_init(&flow_cache->entries[i].key.mf.map); + } +} + +static void +smc_cache_init(struct smc_cache *smc_cache) +{ + int i, j; + for (i = 0; i < SMC_BUCKET_CNT; i++) { + for (j = 0; j < SMC_ENTRY_PER_BUCKET; j++) { + smc_cache->buckets[i].flow_idx[j] = UINT16_MAX; + } + } +} + +static void +dfc_cache_init(struct dfc_cache *flow_cache) +{ + emc_cache_init(&flow_cache->emc_cache); + smc_cache_init(&flow_cache->smc_cache); +} + +static void +emc_cache_uninit(struct emc_cache *flow_cache) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(flow_cache->entries); i++) { + emc_clear_entry(&flow_cache->entries[i]); + } +} + +static void +smc_cache_uninit(struct smc_cache *smc) +{ + int i, j; + + for (i = 0; i < SMC_BUCKET_CNT; i++) { + for (j = 0; j < SMC_ENTRY_PER_BUCKET; j++) { + smc_clear_entry(&(smc->buckets[i]), j); + } + } +} + +static void +dfc_cache_uninit(struct dfc_cache *flow_cache) +{ + smc_cache_uninit(&flow_cache->smc_cache); + emc_cache_uninit(&flow_cache->emc_cache); +} + +/* Check and clear dead flow references slowly (one entry at each + * invocation). */ +static void +emc_cache_slow_sweep(struct emc_cache *flow_cache) +{ + struct emc_entry *entry = &flow_cache->entries[flow_cache->sweep_idx]; + + if (!emc_entry_alive(entry)) { + emc_clear_entry(entry); + } + flow_cache->sweep_idx = (flow_cache->sweep_idx + 1) & EM_FLOW_HASH_MASK; +} + +#ifdef __cplusplus +} +#endif + +#endif /* dpif-netdev-private-dfc.h */ diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index dbbacfa36..c86a98870 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -44,6 +44,7 @@ #include "dpif.h" #include "dpif-netdev-lookup.h" #include "dpif-netdev-perf.h" +#include "dpif-netdev-private-dfc.h" #include "dpif-provider.h" #include "dummy.h" #include "fat-rwlock.h" @@ -142,90 +143,6 @@ static struct odp_support dp_netdev_support = { .ct_orig_tuple6 = true, }; -/* EMC cache and SMC cache compose the datapath flow cache (DFC) - * - * Exact match cache for frequently used flows - * - * The cache uses a 32-bit hash of the packet (which can be the RSS hash) to - * search its entries for a miniflow that matches exactly the miniflow of the - * packet. It stores the 'dpcls_rule' (rule) that matches the miniflow. - * - * A cache entry holds a reference to its 'dp_netdev_flow'. - * - * A miniflow with a given hash can be in one of EM_FLOW_HASH_SEGS different - * entries. The 32-bit hash is split into EM_FLOW_HASH_SEGS values (each of - * them is EM_FLOW_HASH_SHIFT bits wide and the remainder is thrown away). Each - * value is the index of a cache entry where the miniflow could be. - * - * - * Signature match cache (SMC) - * - * This cache stores a 16-bit signature for each flow without storing keys, and - * stores the corresponding 16-bit flow_table index to the 'dp_netdev_flow'. - * Each flow thus occupies 32bit which is much more memory efficient than EMC. - * SMC uses a set-associative design that each bucket contains - * SMC_ENTRY_PER_BUCKET number of entries. - * Since 16-bit flow_table index is used, if there are more than 2^16 - * dp_netdev_flow, SMC will miss them that cannot be indexed by a 16-bit value. - * - * - * Thread-safety - * ============= - * - * Each pmd_thread has its own private exact match cache. - * If dp_netdev_input is not called from a pmd thread, a mutex is used. - */ - -#define EM_FLOW_HASH_SHIFT 13 -#define EM_FLOW_HASH_ENTRIES (1u << EM_FLOW_HASH_SHIFT) -#define EM_FLOW_HASH_MASK (EM_FLOW_HASH_ENTRIES - 1) -#define EM_FLOW_HASH_SEGS 2 - -/* SMC uses a set-associative design. A bucket contains a set of entries that - * a flow item can occupy. For now, it uses one hash function rather than two - * as for the EMC design. */ -#define SMC_ENTRY_PER_BUCKET 4 -#define SMC_ENTRIES (1u << 20) -#define SMC_BUCKET_CNT (SMC_ENTRIES / SMC_ENTRY_PER_BUCKET) -#define SMC_MASK (SMC_BUCKET_CNT - 1) - -/* Default EMC insert probability is 1 / DEFAULT_EM_FLOW_INSERT_INV_PROB */ -#define DEFAULT_EM_FLOW_INSERT_INV_PROB 100 -#define DEFAULT_EM_FLOW_INSERT_MIN (UINT32_MAX / \ - DEFAULT_EM_FLOW_INSERT_INV_PROB) - -struct emc_entry { - struct dp_netdev_flow *flow; - struct netdev_flow_key key; /* key.hash used for emc hash value. */ -}; - -struct emc_cache { - struct emc_entry entries[EM_FLOW_HASH_ENTRIES]; - int sweep_idx; /* For emc_cache_slow_sweep(). */ -}; - -struct smc_bucket { - uint16_t sig[SMC_ENTRY_PER_BUCKET]; - uint16_t flow_idx[SMC_ENTRY_PER_BUCKET]; -}; - -/* Signature match cache, differentiate from EMC cache */ -struct smc_cache { - struct smc_bucket buckets[SMC_BUCKET_CNT]; -}; - -struct dfc_cache { - struct emc_cache emc_cache; - struct smc_cache smc_cache; -}; - -/* Iterate in the exact match cache through every entry that might contain a - * miniflow with hash 'HASH'. */ -#define EMC_FOR_EACH_POS_WITH_HASH(EMC, CURRENT_ENTRY, HASH) \ - for (uint32_t i__ = 0, srch_hash__ = (HASH); \ - (CURRENT_ENTRY) = &(EMC)->entries[srch_hash__ & EM_FLOW_HASH_MASK], \ - i__ < EM_FLOW_HASH_SEGS; \ - i__++, srch_hash__ >>= EM_FLOW_HASH_SHIFT) /* Simple non-wildcarding single-priority classifier. */ @@ -912,90 +829,12 @@ static inline struct dpcls * dp_netdev_pmd_lookup_dpcls(struct dp_netdev_pmd_thread *pmd, odp_port_t in_port); -static inline bool emc_entry_alive(struct emc_entry *ce); -static void emc_clear_entry(struct emc_entry *ce); -static void smc_clear_entry(struct smc_bucket *b, int idx); - static void dp_netdev_request_reconfigure(struct dp_netdev *dp); static inline bool pmd_perf_metrics_enabled(const struct dp_netdev_pmd_thread *pmd); static void queue_netdev_flow_del(struct dp_netdev_pmd_thread *pmd, struct dp_netdev_flow *flow); -static void -emc_cache_init(struct emc_cache *flow_cache) -{ - int i; - - flow_cache->sweep_idx = 0; - for (i = 0; i < ARRAY_SIZE(flow_cache->entries); i++) { - flow_cache->entries[i].flow = NULL; - flow_cache->entries[i].key.hash = 0; - flow_cache->entries[i].key.len = sizeof(struct miniflow); - flowmap_init(&flow_cache->entries[i].key.mf.map); - } -} - -static void -smc_cache_init(struct smc_cache *smc_cache) -{ - int i, j; - for (i = 0; i < SMC_BUCKET_CNT; i++) { - for (j = 0; j < SMC_ENTRY_PER_BUCKET; j++) { - smc_cache->buckets[i].flow_idx[j] = UINT16_MAX; - } - } -} - -static void -dfc_cache_init(struct dfc_cache *flow_cache) -{ - emc_cache_init(&flow_cache->emc_cache); - smc_cache_init(&flow_cache->smc_cache); -} - -static void -emc_cache_uninit(struct emc_cache *flow_cache) -{ - int i; - - for (i = 0; i < ARRAY_SIZE(flow_cache->entries); i++) { - emc_clear_entry(&flow_cache->entries[i]); - } -} - -static void -smc_cache_uninit(struct smc_cache *smc) -{ - int i, j; - - for (i = 0; i < SMC_BUCKET_CNT; i++) { - for (j = 0; j < SMC_ENTRY_PER_BUCKET; j++) { - smc_clear_entry(&(smc->buckets[i]), j); - } - } -} - -static void -dfc_cache_uninit(struct dfc_cache *flow_cache) -{ - smc_cache_uninit(&flow_cache->smc_cache); - emc_cache_uninit(&flow_cache->emc_cache); -} - -/* Check and clear dead flow references slowly (one entry at each - * invocation). */ -static void -emc_cache_slow_sweep(struct emc_cache *flow_cache) -{ - struct emc_entry *entry = &flow_cache->entries[flow_cache->sweep_idx]; - - if (!emc_entry_alive(entry)) { - emc_clear_entry(entry); - } - flow_cache->sweep_idx = (flow_cache->sweep_idx + 1) & EM_FLOW_HASH_MASK; -} - /* Updates the time in PMD threads context and should be called in three cases: * * 1. PMD structure initialization: @@ -3181,12 +3020,6 @@ smc_entry_get(struct dp_netdev_pmd_thread *pmd, const uint32_t hash) return NULL; } -static void -smc_clear_entry(struct smc_bucket *b, int idx) -{ - b->flow_idx[idx] = UINT16_MAX; -} - /* Insert the flow_table index into SMC. Insertion may fail when 1) SMC is * turned off, 2) the flow_table index is larger than uint16_t can handle. * If there is already an SMC entry having same signature, the index will be