From patchwork Wed Mar 13 05:27:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yanqin Wei X-Patchwork-Id: 1055905 X-Patchwork-Delegate: ian.stokes@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=arm.com Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44K0jY3c6zz9s47 for ; Wed, 13 Mar 2019 16:28:01 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 770E3AE7; Wed, 13 Mar 2019 05:27:59 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 392C5A59 for ; Wed, 13 Mar 2019 05:27:58 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from foss.arm.com (foss.arm.com [217.140.101.70]) by smtp1.linuxfoundation.org (Postfix) with ESMTP id B4765827 for ; Wed, 13 Mar 2019 05:27:57 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 77A9580D; Tue, 12 Mar 2019 22:27:57 -0700 (PDT) Received: from net-arm-thunderx2.shanghai.arm.com (net-arm-thunderx2.shanghai.arm.com [10.169.40.121]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 95CAB3F71D; Tue, 12 Mar 2019 22:27:56 -0700 (PDT) From: Yanqin Wei To: dev@openvswitch.org Date: Wed, 13 Mar 2019 13:27:48 +0800 Message-Id: <1552454868-183848-1-git-send-email-Yanqin.Wei@arm.com> X-Mailer: git-send-email 2.7.4 X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Cc: nd@arm.com, Gavin.Hu@arm.com Subject: [ovs-dev] [PATCH v3] dpif-netdev: dfc_process optimization by prefetching EMC entry. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org It is observed that the throughput of multi-flow is worse than single-flow in the EMC NIC to NIC cases. It is because CPU cache-miss increasing in EMC lookup. Each flow need load at least one EMC entry to CPU cache(several cache lines) and compare it with packet miniflow. This patch improve it by prefetching EMC entry in advance. Hash value can be obtained from dpdk rss hash, so this step can be advanced ahead of miniflow_extract() and prefetch EMC entry there. The prefetching size is defined as ROUND_UP(128,CACHE_LINE_SIZE), which can cover majority traffic including TCP/UDP protocol and need 2 cache lines in most modern CPU. Performance test was run in some arm platform. 1000/10000 flows NIC2NIC test achieved around 10% throughput improvement in thunderX2(aarch64 platform). Signed-off-by: Yanqin Wei Reviewed-by: Gavin Hu Signed-off-by: Yanqin Wei Reviewed-by: Gavin Hu --- lib/dpif-netdev.c | 80 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 52 insertions(+), 28 deletions(-) diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 4d6d0c3..982082c 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -189,6 +189,10 @@ struct netdev_flow_key { #define DEFAULT_EM_FLOW_INSERT_MIN (UINT32_MAX / \ DEFAULT_EM_FLOW_INSERT_INV_PROB) +/* DEFAULT_EMC_PREFETCH_SIZE can cover majority traffic including TCP/UDP + * protocol. */ +#define DEFAULT_EMC_PREFETCH_SIZE ROUND_UP(128,CACHE_LINE_SIZE) + struct emc_entry { struct dp_netdev_flow *flow; struct netdev_flow_key key; /* key.hash used for emc hash value. */ @@ -6166,15 +6170,20 @@ dp_netdev_upcall(struct dp_netdev_pmd_thread *pmd, struct dp_packet *packet_, } static inline uint32_t -dpif_netdev_packet_get_rss_hash_orig_pkt(struct dp_packet *packet, - const struct miniflow *mf) +dpif_netdev_packet_get_packet_rss_hash(struct dp_packet *packet, + bool md_is_valid) { - uint32_t hash; + uint32_t hash,recirc_depth; - if (OVS_LIKELY(dp_packet_rss_valid(packet))) { - hash = dp_packet_get_rss_hash(packet); - } else { - hash = miniflow_hash_5tuple(mf, 0); + hash = dp_packet_get_rss_hash(packet); + + if (md_is_valid) { + /* The RSS hash must account for the recirculation depth to avoid + * collisions in the exact match cache */ + recirc_depth = *recirc_depth_get_unsafe(); + if (OVS_UNLIKELY(recirc_depth)) { + hash = hash_finish(hash, recirc_depth); + } dp_packet_set_rss_hash(packet, hash); } @@ -6182,24 +6191,23 @@ dpif_netdev_packet_get_rss_hash_orig_pkt(struct dp_packet *packet, } static inline uint32_t -dpif_netdev_packet_get_rss_hash(struct dp_packet *packet, - const struct miniflow *mf) +dpif_netdev_packet_get_hash_5tuple(struct dp_packet *packet, + const struct miniflow *mf, + bool md_is_valid) { - uint32_t hash, recirc_depth; + uint32_t hash,recirc_depth; - if (OVS_LIKELY(dp_packet_rss_valid(packet))) { - hash = dp_packet_get_rss_hash(packet); - } else { - hash = miniflow_hash_5tuple(mf, 0); - dp_packet_set_rss_hash(packet, hash); - } + hash = miniflow_hash_5tuple(mf, 0); + dp_packet_set_rss_hash(packet, hash); - /* The RSS hash must account for the recirculation depth to avoid - * collisions in the exact match cache */ - recirc_depth = *recirc_depth_get_unsafe(); - if (OVS_UNLIKELY(recirc_depth)) { - hash = hash_finish(hash, recirc_depth); - dp_packet_set_rss_hash(packet, hash); + if (md_is_valid) { + /* The RSS hash must account for the recirculation depth to avoid + * collisions in the exact match cache */ + recirc_depth = *recirc_depth_get_unsafe(); + if (OVS_UNLIKELY(recirc_depth)) { + hash = hash_finish(hash, recirc_depth); + dp_packet_set_rss_hash(packet, hash); + } } return hash; } @@ -6390,6 +6398,7 @@ dfc_processing(struct dp_netdev_pmd_thread *pmd, bool smc_enable_db; size_t map_cnt = 0; bool batch_enable = true; + bool is_5tuple_hash_needed; atomic_read_relaxed(&pmd->dp->smc_enable_db, &smc_enable_db); pmd_perf_update_counter(&pmd->perf_stats, @@ -6436,16 +6445,31 @@ dfc_processing(struct dp_netdev_pmd_thread *pmd, } } - miniflow_extract(packet, &key->mf); - key->len = 0; /* Not computed yet. */ /* If EMC and SMC disabled skip hash computation */ if (smc_enable_db == true || cur_min != 0) { - if (!md_is_valid) { - key->hash = dpif_netdev_packet_get_rss_hash_orig_pkt(packet, - &key->mf); + if (OVS_LIKELY(dp_packet_rss_valid(packet))) { + is_5tuple_hash_needed = false; + key->hash = + dpif_netdev_packet_get_packet_rss_hash(packet,md_is_valid); + if (cur_min) { + ovs_prefetch_range( + &cache->emc_cache.entries[key->hash & EM_FLOW_HASH_MASK], + DEFAULT_EMC_PREFETCH_SIZE); + } } else { - key->hash = dpif_netdev_packet_get_rss_hash(packet, &key->mf); + is_5tuple_hash_needed = true; } + } else { + is_5tuple_hash_needed = false; + } + + miniflow_extract(packet, &key->mf); + key->len = 0; /* Not computed yet. */ + + /* If 5tuple hash is needed */ + if (is_5tuple_hash_needed) { + key->hash = dpif_netdev_packet_get_hash_5tuple(packet, &key->mf, + md_is_valid); } if (cur_min) { flow = emc_lookup(&cache->emc_cache, key);