From patchwork Thu Jul 7 15:38:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Emma Finn X-Patchwork-Id: 1653681 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=c+lKqXOr; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0wC345gz9s0w for ; Fri, 8 Jul 2022 01:39:55 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 3B0E24116C; Thu, 7 Jul 2022 15:39:53 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 3B0E24116C Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=c+lKqXOr X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wCtfHmQEHWng; Thu, 7 Jul 2022 15:39:50 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 3388941169; Thu, 7 Jul 2022 15:39:45 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 3388941169 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 10845C0070; Thu, 7 Jul 2022 15:39:44 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 52040C0080 for ; Thu, 7 Jul 2022 15:39:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 54FB4410CC for ; Thu, 7 Jul 2022 15:39:30 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 54FB4410CC X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uR6k6bYqhSzO for ; Thu, 7 Jul 2022 15:39:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 9C152410AE Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id 9C152410AE for ; Thu, 7 Jul 2022 15:39:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208367; x=1688744367; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PYDXkg/enDlUr1oEwGYtrQgW9jpYwLyS9vLNN5PhUps=; b=c+lKqXOr+E6Wr27CruNS34tM3wt0MvpgAFXj3AHEt4GhS25wEUH0SgiC 2AN7H7MZyBT7wYIl+X2saIP5ZsgIvMlfQyRQNndyTIsiyTnt2XIMW2e3S HrdJcL/3PhRQFpEo2kNPB+JKH05dhORGR2Ds5tXJuFIe+0tg68i6hiu/f prUJ7g0Xzz53ni1x+/29RlCZT7zUzoU7WmOynbm7xJtWLvve9tuG/omC+ ur7IJAQ/E/SC7w7qd6al0Jis0ZOojL4c1YwFSSbdj4Az9SA0QIquNt1q4 JzMkCJttUMDWDG1BcWcSktEtlnmkMyNXHv5lXb4HyVgAxU2cUrAEhgGZH A==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083995" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083995" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633496" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:25 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:59 +0000 Message-Id: <20220707153900.3147694-10-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 09/10] odp-execute: Add ISA implementation of set_masked ETH X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit includes infrastructure changes for enabling set_masked_X actions and also adds support for the AVX512 implementation of the eth_set_addrs action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 90 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.c | 14 ++++++ lib/odp-execute-private.h | 3 ++ lib/odp-execute.c | 49 +++++++++++---------- lib/odp-execute.h | 3 ++ 5 files changed, 137 insertions(+), 22 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 6252cd3d0..12eafc35e 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -24,6 +24,7 @@ #include "dp-packet.h" #include "immintrin.h" +#include "odp-execute.h" #include "odp-execute-private.h" #include "odp-netlink.h" #include "openvswitch/vlog.h" @@ -51,6 +52,16 @@ BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + BUILD_ASSERT_DECL(sizeof(struct dp_packet) - offsetof(struct dp_packet, l2_pad_size) >= sizeof(__m128i)); +/* The below build assert makes sure the order of the fields needed by + * the set masked functions shuffle operations do not change. This should not + * happen as these are defined under the Linux uapi. */ +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ethernet, eth_src) + + MEMBER_SIZEOF(struct ovs_key_ethernet, eth_src) == + offsetof(struct ovs_key_ethernet, eth_dst)); + +/* Array of callback functions, one for each masked operation. */ +odp_execute_action_cb impl_set_masked_funcs[__OVS_KEY_ATTR_MAX]; + static inline void ALWAYS_INLINE avx512_dp_packet_resize_l2(struct dp_packet *b, int resize_by_bytes) { @@ -208,6 +219,80 @@ action_avx512_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) } } +/* This function performs the same operation on each packet in the batch as + * the scalar odp_eth_set_addrs() function. */ +static void +action_avx512_eth_set_addrs(struct dp_packet_batch *batch, + const struct nlattr *a) +{ + const struct ovs_key_ethernet *key, *mask; + struct dp_packet *packet; + + a = nl_attr_get(a); + key = nl_attr_get(a); + mask = odp_get_key_mask(a, struct ovs_key_ethernet); + + /* Read the content of the key(src) and mask in the respective registers. + * We only load the src and dest addresses, which is only 96-bits and not + * 128-bits. */ + __m128i v_src = _mm_maskz_loadu_epi32(0x7,(void *) key); + __m128i v_mask = _mm_maskz_loadu_epi32(0x7, (void *) mask); + + + /* These shuffle masks are used below, and each position tells where to + * move the bytes to. So here, the fourth sixth byte in + * ovs_key_ethernet is moved to byte location 0 in v_src/v_mask. + * The seventh is moved to 1, etc., etc. + * This swap is needed to move the src and dest MAC addresses in the + * same order as in the ethernet packet. */ + static const uint8_t eth_shuffle[16] = { + 6, 7, 8, 9, 10, 11, 0, 1, + 2, 3, 4, 5, 0xFF, 0xFF, 0xFF, 0xFF + }; + + /* Load the shuffle mask in v_shuf. */ + __m128i v_shuf = _mm_loadu_si128((void *) eth_shuffle); + + /* Swap the key/mask src and dest addresses to the ethernet order. */ + v_src = _mm_shuffle_epi8(v_src, v_shuf); + v_mask = _mm_shuffle_epi8(v_mask, v_shuf); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + + struct eth_header *eh = dp_packet_eth(packet); + + if (!eh) { + continue; + } + + /* Load the first 128-bits of the packet into the v_ether register. */ + __m128i v_dst = _mm_loadu_si128((void *) eh); + + /* AND the v_mask to the packet data (v_dst). */ + __m128i dst_masked = _mm_andnot_si128(v_mask, v_dst); + + /* OR the new addresses (v_src) with the masked packet addresses + * (dst_masked). */ + __m128i res = _mm_or_si128(v_src, dst_masked); + + /* Write back the modified ethernet addresses. */ + _mm_storeu_si128((void *) eh, res); + } +} + +static void +action_avx512_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) +{ + const struct nlattr *mask = nl_attr_get(a); + enum ovs_key_attr attr_type = nl_attr_type(mask); + + if (attr_type <= OVS_KEY_ATTR_MAX && impl_set_masked_funcs[attr_type]) { + impl_set_masked_funcs[attr_type](batch, a); + } else { + odp_execute_scalar_action(batch, a); + } +} + int action_avx512_init(struct odp_execute_action_impl *self) { @@ -215,6 +300,11 @@ action_avx512_init(struct odp_execute_action_impl *self) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_avx512_push_vlan; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_avx512_set_masked; + + /* Set function pointers for the individual operations supported by the + * SET_MASKED action. */ + impl_set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_avx512_eth_set_addrs; return 0; } diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index d99a94a93..a6df84c90 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -22,6 +22,7 @@ #include "cpu.h" #include "dpdk.h" #include "dp-packet.h" +#include "odp-execute.h" #include "odp-execute-private.h" #include "odp-netlink.h" #include "odp-util.h" @@ -246,6 +247,19 @@ action_autoval_generic(struct dp_packet_batch *batch, const struct nlattr *a) dp_packet_delete_batch(&original_batch, true); } +void +odp_execute_scalar_action(struct dp_packet_batch *batch, + const struct nlattr *action) +{ + enum ovs_action_attr type = nl_attr_type(action); + + if (action_impls[ACTION_IMPL_SCALAR].funcs[type] && + type <= OVS_ACTION_ATTR_MAX) { + + action_impls[ACTION_IMPL_SCALAR].funcs[type](batch, action); + } +} + int action_autoval_init(struct odp_execute_action_impl *self) { diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index b3a10cd82..3fdcdc95b 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -94,4 +94,7 @@ int action_avx512_init(struct odp_execute_action_impl *self); void odp_execute_action_get_info(struct ds *name); +void odp_execute_scalar_action(struct dp_packet_batch *batch, + const struct nlattr *action); + #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 67d1e914b..09c8e4401 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -562,8 +562,6 @@ odp_execute_set_action(struct dp_packet *packet, const struct nlattr *a) } } -#define get_mask(a, type) ((const type *)(const void *)(a + 1) + 1) - static void odp_execute_masked_set_action(struct dp_packet *packet, const struct nlattr *a) @@ -575,17 +573,17 @@ odp_execute_masked_set_action(struct dp_packet *packet, switch (type) { case OVS_KEY_ATTR_PRIORITY: md->skb_priority = nl_attr_get_u32(a) - | (md->skb_priority & ~*get_mask(a, uint32_t)); + | (md->skb_priority & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_SKB_MARK: md->pkt_mark = nl_attr_get_u32(a) - | (md->pkt_mark & ~*get_mask(a, uint32_t)); + | (md->pkt_mark & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_ETHERNET: odp_eth_set_addrs(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ethernet)); + odp_get_key_mask(a, struct ovs_key_ethernet)); break; case OVS_KEY_ATTR_NSH: { @@ -595,27 +593,27 @@ odp_execute_masked_set_action(struct dp_packet *packet, case OVS_KEY_ATTR_IPV4: odp_set_ipv4(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ipv4)); + odp_get_key_mask(a, struct ovs_key_ipv4)); break; case OVS_KEY_ATTR_IPV6: odp_set_ipv6(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ipv6)); + odp_get_key_mask(a, struct ovs_key_ipv6)); break; case OVS_KEY_ATTR_TCP: odp_set_tcp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_tcp)); + odp_get_key_mask(a, struct ovs_key_tcp)); break; case OVS_KEY_ATTR_UDP: odp_set_udp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_udp)); + odp_get_key_mask(a, struct ovs_key_udp)); break; case OVS_KEY_ATTR_SCTP: odp_set_sctp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_sctp)); + odp_get_key_mask(a, struct ovs_key_sctp)); break; case OVS_KEY_ATTR_MPLS: @@ -623,33 +621,33 @@ odp_execute_masked_set_action(struct dp_packet *packet, if (mh) { put_16aligned_be32(&mh->mpls_lse, nl_attr_get_be32(a) | (get_16aligned_be32(&mh->mpls_lse) - & ~*get_mask(a, ovs_be32))); + & ~*odp_get_key_mask(a, ovs_be32))); } break; case OVS_KEY_ATTR_ARP: set_arp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_arp)); + odp_get_key_mask(a, struct ovs_key_arp)); break; case OVS_KEY_ATTR_ND: odp_set_nd(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_nd)); + odp_get_key_mask(a, struct ovs_key_nd)); break; case OVS_KEY_ATTR_ND_EXTENSIONS: odp_set_nd_ext(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_nd_extensions)); + odp_get_key_mask(a, struct ovs_key_nd_extensions)); break; case OVS_KEY_ATTR_DP_HASH: md->dp_hash = nl_attr_get_u32(a) - | (md->dp_hash & ~*get_mask(a, uint32_t)); + | (md->dp_hash & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_RECIRC_ID: md->recirc_id = nl_attr_get_u32(a) - | (md->recirc_id & ~*get_mask(a, uint32_t)); + | (md->recirc_id & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_TUNNEL: /* Masked data not supported for tunnel. */ @@ -857,6 +855,17 @@ action_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) } } +static void +action_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) +{ + const struct nlattr *key = nl_attr_get(a); + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + odp_execute_masked_set_action(packet, key); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -867,6 +876,7 @@ odp_action_scalar_init(struct odp_execute_action_impl *self) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_push_vlan; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_set_masked; return 0; } @@ -1078,12 +1088,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, } break; - case OVS_ACTION_ATTR_SET_MASKED: - DP_PACKET_BATCH_FOR_EACH(i, packet, batch) { - odp_execute_masked_set_action(packet, nl_attr_get(a)); - } - break; - case OVS_ACTION_ATTR_SAMPLE: DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { odp_execute_sample(dp, packet, steal && last_action, a, @@ -1210,6 +1214,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, /* The following actions are handled by the scalar implementation. */ case OVS_ACTION_ATTR_POP_VLAN: case OVS_ACTION_ATTR_PUSH_VLAN: + case OVS_ACTION_ATTR_SET_MASKED: OVS_NOT_REACHED(); } diff --git a/lib/odp-execute.h b/lib/odp-execute.h index 0921ee924..2ba1ec5d2 100644 --- a/lib/odp-execute.h +++ b/lib/odp-execute.h @@ -46,4 +46,7 @@ void odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, const struct nlattr *actions, size_t actions_len, odp_execute_cb dp_execute_action); + +#define odp_get_key_mask(a, type) ((const type *)(const void *)(a + 1) + 1) + #endif