From patchwork Fri May 6 05:23:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Amber X-Patchwork-Id: 1627396 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=OTX1dNXR; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Kvfh00j3Rz9sG6 for ; Fri, 6 May 2022 15:46:23 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 81F84840A5; Fri, 6 May 2022 05:46:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bITZ1CDaFXFu; Fri, 6 May 2022 05:46:20 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 998D483FBB; Fri, 6 May 2022 05:46:19 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C61BDC0083; Fri, 6 May 2022 05:46:17 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5EA54C002D for ; Fri, 6 May 2022 05:46:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 5A58483F91 for ; Fri, 6 May 2022 05:46:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FFxuR2yIjVA3 for ; Fri, 6 May 2022 05:46:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by smtp1.osuosl.org (Postfix) with ESMTPS id C5A5283F86 for ; Fri, 6 May 2022 05:46:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651815975; x=1683351975; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PZby0D+USv0YyrQmhJedatKX7pRMrjfEuDu6pxxbyqQ=; b=OTX1dNXR/shaCJuDDaPp1TwQj9Lp0BV+km9OMDmezvhYN9Pd7FBYK5bw 574yo03KV72++fZSgVpvqBa98t40NhsyU0l9z49f4kbZxhRQ0P7MVYdiV 1rIgxt4/XFRDipT51OXeUq5bZZPQMxbnNv8dvamUxb1t8OY74SXZovGBO HOrWmYKfXRn5AYAUDc0h1z+4IJTTDP8Bsr/PDfsvMuZ2T2gW/s0m6Oga5 hIlJlwJIwzoFEmfxaHP5x84l3pQi02iYALVJjWCTWIwNQg4JxA+bTqs01 Myzotf9KjH1g8eRazHa8ct1VNDZoZiwBu2Sse2PH2x0TAjphAhHfrWn8v g==; X-IronPort-AV: E=McAfee;i="6400,9594,10338"; a="268503767" X-IronPort-AV: E=Sophos;i="5.91,203,1647327600"; d="scan'208";a="268503767" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 May 2022 22:46:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,203,1647327600"; d="scan'208";a="537727376" Received: from unknown (HELO localhost.localdomain) ([10.190.213.111]) by orsmga006.jf.intel.com with ESMTP; 05 May 2022 22:46:12 -0700 From: Kumar Amber To: ovs-dev@openvswitch.org Date: Fri, 6 May 2022 10:53:23 +0530 Message-Id: <20220506052326.3191931-2-kumar.amber@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506052326.3191931-1-kumar.amber@intel.com> References: <20220506052326.3191931-1-kumar.amber@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, fbl@sysclose.org, Kumar Amber , david.marchand@redhat.com Subject: [ovs-dev] [PATCH v7 1/4] mfex_avx512: Calculate pkt offsets at compile time. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The patch removes magic numbers pkt offsets and minimum packet lenght and instead calculate it at compile time. Signed-off-by: Kumar Amber --- lib/dpif-netdev-extract-avx512.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/lib/dpif-netdev-extract-avx512.c b/lib/dpif-netdev-extract-avx512.c index 6b6fe07db..6ae15a4db 100644 --- a/lib/dpif-netdev-extract-avx512.c +++ b/lib/dpif-netdev-extract-avx512.c @@ -246,6 +246,16 @@ _mm512_maskz_permutexvar_epi8_wrap(__mmask64 kmask, __m512i idx, __m512i a) NC, NC, NC, NC, 0xBF, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, \ NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC +#define PKT_OFFSET_L2 (ETH_HEADER_LEN) +#define PKT_OFFSET_L3_VLAN (ETH_HEADER_LEN + VLAN_HEADER_LEN) +#define PKT_OFFSET_L4_IPv4 (ETH_HEADER_LEN + IP_HEADER_LEN) +#define PKT_OFFSET_L4_VLAN_IPv4 (PKT_OFFSET_L4_IPv4 + VLAN_HEADER_LEN) + +#define PKT_MIN_ETH_IPv4_UDP (PKT_OFFSET_L4_IPv4 + UDP_HEADER_LEN) +#define PKT_MIN_ETH_VLAN_IPv4_UDP (PKT_OFFSET_L4_VLAN_IPv4 + UDP_HEADER_LEN) +#define PKT_MIN_ETH_IPv4_TCP (PKT_OFFSET_L4_IPv4 + TCP_HEADER_LEN) +#define PKT_MIN_ETH_VLAN_IPv4_TCP (PKT_OFFSET_L4_VLAN_IPv4 + TCP_HEADER_LEN) + /* This union allows initializing static data as u8, but easily loading it * into AVX512 registers too. The union ensures proper alignment for the zmm. */ @@ -345,9 +355,9 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = .mf_bits = { 0x18a0000000000000, 0x0000000000040401}, .dp_pkt_offs = { - 0, UINT16_MAX, 14, 34, + 0, UINT16_MAX, PKT_OFFSET_L2, PKT_OFFSET_L4_IPv4, }, - .dp_pkt_min_size = 42, + .dp_pkt_min_size = PKT_MIN_ETH_IPv4_UDP, }, [PROFILE_ETH_IPV4_TCP] = { @@ -368,9 +378,9 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = .mf_bits = { 0x18a0000000000000, 0x0000000000044401}, .dp_pkt_offs = { - 0, UINT16_MAX, 14, 34, + 0, UINT16_MAX, PKT_OFFSET_L2, PKT_OFFSET_L4_IPv4, }, - .dp_pkt_min_size = 54, + .dp_pkt_min_size = PKT_MIN_ETH_IPv4_TCP, }, [PROFILE_ETH_VLAN_IPV4_UDP] = { @@ -387,9 +397,10 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = .mf_bits = { 0x38a0000000000000, 0x0000000000040401}, .dp_pkt_offs = { - 14, UINT16_MAX, 18, 38, + PKT_OFFSET_L2, UINT16_MAX, PKT_OFFSET_L3_VLAN, + PKT_OFFSET_L4_VLAN_IPv4, }, - .dp_pkt_min_size = 46, + .dp_pkt_min_size = PKT_MIN_ETH_VLAN_IPv4_UDP, }, [PROFILE_ETH_VLAN_IPV4_TCP] = { @@ -412,9 +423,10 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = .mf_bits = { 0x38a0000000000000, 0x0000000000044401}, .dp_pkt_offs = { - 14, UINT16_MAX, 18, 38, + PKT_OFFSET_L2, UINT16_MAX, PKT_OFFSET_L3_VLAN, + PKT_OFFSET_L4_VLAN_IPv4, }, - .dp_pkt_min_size = 58, + .dp_pkt_min_size = PKT_MIN_ETH_VLAN_IPv4_TCP, }, }; From patchwork Fri May 6 05:23:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Amber X-Patchwork-Id: 1627398 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=E2ZwJHYW; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Kvfh35fnBz9sBF for ; Fri, 6 May 2022 15:46:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 6C22081761; Fri, 6 May 2022 05:46:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j5lFAA1W6_ua; Fri, 6 May 2022 05:46:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id BC4E8840A7; Fri, 6 May 2022 05:46:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6DFABC0081; Fri, 6 May 2022 05:46:21 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id B80D3C007B for ; Fri, 6 May 2022 05:46:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 97A8783F86 for ; Fri, 6 May 2022 05:46:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c88ODQ82Bpzn for ; Fri, 6 May 2022 05:46:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by smtp1.osuosl.org (Postfix) with ESMTPS id 0B9B883FA3 for ; Fri, 6 May 2022 05:46:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651815979; x=1683351979; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LDrsASZY2UVvQsGDE9m+XMcbEKzNOTU0hVYXHaIFXCE=; b=E2ZwJHYWBrzA/OuDA2+YivOhflN1vKkCTgXfv/Bn4lCsyL7vOJUSOL9j E+2q7moNM72Sl0e+BdgT81xa6M/gci/QWKcLEi/p41pYRUiCFcGMB1i59 YBkiA4C/tnuhNdysjcY+jkGDj4fQSx13ZMRPko4dVI5wXA3NgnU8PMYrD sIY4PywS6lboLfrwGLz8J9Ecq/tpXWn21S124EvOEMSpfv6gT6bdCi/2i xMUASAtmuZR3xO3ERDEHuCI18hyRM6tGQxCuQdF2jNwlANIP8swlcMfVo YGfPGQB8iHETZT0M7C7iEm+dUsz/xMONjNTd65abcKsTox2foqAnUSxgu Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10338"; a="268503771" X-IronPort-AV: E=Sophos;i="5.91,203,1647327600"; d="scan'208";a="268503771" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 May 2022 22:46:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,203,1647327600"; d="scan'208";a="537727392" Received: from unknown (HELO localhost.localdomain) ([10.190.213.111]) by orsmga006.jf.intel.com with ESMTP; 05 May 2022 22:46:15 -0700 From: Kumar Amber To: ovs-dev@openvswitch.org Date: Fri, 6 May 2022 10:53:24 +0530 Message-Id: <20220506052326.3191931-3-kumar.amber@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506052326.3191931-1-kumar.amber@intel.com> References: <20220506052326.3191931-1-kumar.amber@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, fbl@sysclose.org, Kumar Amber , david.marchand@redhat.com Subject: [ovs-dev] [PATCH v7 2/4] mfex_avx512: Calculate miniflow_bits at compile time. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The patch removes magic numbers from miniflow_bits and calculates the bits at compile time. This also makes it easier to handle any ABI changes. Signed-off-by: Kumar Amber --- lib/dpif-netdev-extract-avx512.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/lib/dpif-netdev-extract-avx512.c b/lib/dpif-netdev-extract-avx512.c index 6ae15a4db..e77bb3214 100644 --- a/lib/dpif-netdev-extract-avx512.c +++ b/lib/dpif-netdev-extract-avx512.c @@ -256,6 +256,19 @@ _mm512_maskz_permutexvar_epi8_wrap(__mmask64 kmask, __m512i idx, __m512i a) #define PKT_MIN_ETH_IPv4_TCP (PKT_OFFSET_L4_IPv4 + TCP_HEADER_LEN) #define PKT_MIN_ETH_VLAN_IPv4_TCP (PKT_OFFSET_L4_VLAN_IPv4 + TCP_HEADER_LEN) +/* MF bits. */ +#define MF_BIT(field) (MAP_1 << ((offsetof(struct flow, field) / 8) % \ + MAP_T_BITS)) + +#define MF_ETH (MF_BIT(dp_hash) | MF_BIT(in_port) | MF_BIT(packet_type)\ + | MF_BIT(dl_dst) | MF_BIT(dl_src)| MF_BIT(dl_type)) + +#define MF_ETH_VLAN (MF_ETH | MF_BIT(vlans)) +#define MF_IPV4_UDP (MF_BIT(nw_src) | MF_BIT(ipv6_label) | MF_BIT(tp_src) | \ + MF_BIT(tp_dst)) + +#define MF_IPV4_TCP (MF_IPV4_UDP | MF_BIT(tcp_flags) | MF_BIT(arp_tha.ea[2])) + /* This union allows initializing static data as u8, but easily loading it * into AVX512 registers too. The union ensures proper alignment for the zmm. */ @@ -353,7 +366,7 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = .strip_mask.u8_data = { PATTERN_STRIP_IPV4_MASK }, .store_kmsk = PATTERN_IPV4_UDP_KMASK, - .mf_bits = { 0x18a0000000000000, 0x0000000000040401}, + .mf_bits = { MF_ETH, MF_IPV4_UDP}, .dp_pkt_offs = { 0, UINT16_MAX, PKT_OFFSET_L2, PKT_OFFSET_L4_IPv4, }, @@ -376,7 +389,7 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = .strip_mask.u8_data = { PATTERN_STRIP_IPV4_MASK }, .store_kmsk = PATTERN_IPV4_TCP_KMASK, - .mf_bits = { 0x18a0000000000000, 0x0000000000044401}, + .mf_bits = { MF_ETH, MF_IPV4_TCP}, .dp_pkt_offs = { 0, UINT16_MAX, PKT_OFFSET_L2, PKT_OFFSET_L4_IPv4, }, @@ -395,7 +408,7 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = .strip_mask.u8_data = { PATTERN_STRIP_DOT1Q_IPV4_MASK }, .store_kmsk = PATTERN_DT1Q_IPV4_UDP_KMASK, - .mf_bits = { 0x38a0000000000000, 0x0000000000040401}, + .mf_bits = { MF_ETH_VLAN, MF_IPV4_UDP}, .dp_pkt_offs = { PKT_OFFSET_L2, UINT16_MAX, PKT_OFFSET_L3_VLAN, PKT_OFFSET_L4_VLAN_IPv4, @@ -421,7 +434,7 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = .strip_mask.u8_data = { PATTERN_STRIP_DOT1Q_IPV4_MASK }, .store_kmsk = PATTERN_DT1Q_IPV4_TCP_KMASK, - .mf_bits = { 0x38a0000000000000, 0x0000000000044401}, + .mf_bits = { MF_ETH_VLAN, MF_IPV4_TCP}, .dp_pkt_offs = { PKT_OFFSET_L2, UINT16_MAX, PKT_OFFSET_L3_VLAN, PKT_OFFSET_L4_VLAN_IPv4, From patchwork Fri May 6 05:23:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Amber X-Patchwork-Id: 1627399 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=ma244Ioa; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KvfhB1hpbz9sBF for ; Fri, 6 May 2022 15:46:33 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1C3BD60071; Fri, 6 May 2022 05:46:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0AyiraubSPOE; Fri, 6 May 2022 05:46:29 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id AB95B61138; Fri, 6 May 2022 05:46:28 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 767E0C0032; Fri, 6 May 2022 05:46:28 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1ABFFC0081 for ; Fri, 6 May 2022 05:46:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 461FB82726 for ; Fri, 6 May 2022 05:46:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uaQh9yzmJb2i for ; Fri, 6 May 2022 05:46:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by smtp1.osuosl.org (Postfix) with ESMTPS id D708F840B2 for ; Fri, 6 May 2022 05:46:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651815982; x=1683351982; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lVjWsd43UzaT/V0A3IAeJpH+vyO2HCro2mVTgoMiL2A=; b=ma244IoaK0FEaoBqOL8xY2e3D2lwzHrG6lJDAV9TRb6b9WbGnXtF8I1F 9jLznzkYFzvkwriyLjvP8/DpHF9uy2nc2nkPSpFKz/8Ip0ojl+36+E5TE 7HF4QxatpiTl6rfqNM4b35TA1rfhg5rnTHMs8XKT444e2COVuz+toYFAr hFKo4dZK/VYhB5OdeH96DDZvEAgqabZT9NohnmcFr6a+yEP70RCFJXbwa +UIFBQB5VqmeEOrgPUwp5sKfomeLFPKCI5RXbYUzj4tnUlRpVMgsigFI0 C8A1IRYcyYCBWQdTSPAFJ63kbHmFU8ZHnes67IXtgHa1Wv7y5v1FILxCt g==; X-IronPort-AV: E=McAfee;i="6400,9594,10338"; a="268503827" X-IronPort-AV: E=Sophos;i="5.91,203,1647327600"; d="scan'208";a="268503827" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 May 2022 22:46:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,203,1647327600"; d="scan'208";a="537727408" Received: from unknown (HELO localhost.localdomain) ([10.190.213.111]) by orsmga006.jf.intel.com with ESMTP; 05 May 2022 22:46:18 -0700 From: Kumar Amber To: ovs-dev@openvswitch.org Date: Fri, 6 May 2022 10:53:25 +0530 Message-Id: <20220506052326.3191931-4-kumar.amber@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506052326.3191931-1-kumar.amber@intel.com> References: <20220506052326.3191931-1-kumar.amber@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, fbl@sysclose.org, Kumar Amber , david.marchand@redhat.com Subject: [ovs-dev] [PATCH v7 3/4] dpif-netdev/mfex: Add AVX512 ipv6 traffic profiles X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add AVX512 Ipv6 optimized profile for vlan/IPv6/UDP and vlan/IPv6/TCP, IPv6/UDP and IPv6/TCP. MFEX autovalidaton test-case already has the IPv6 support for validating against the scalar mfex. Signed-off-by: Kumar Amber Signed-off-by: Harry van Haaren Co-authored-by: Harry van Haaren --- v7: - Fix Lenght checks for plen. v5: - Add variable length checks for IPv6 and TCP. v4: - Rebase to master. v2: - Fix CI build error. - Fix check-patch sign-offs. --- --- NEWS | 6 +- lib/automake.mk | 1 + lib/dpif-netdev-extract-avx512.c | 306 +++++++++++++++++++++++++++++- lib/dpif-netdev-private-extract.c | 51 ++++- lib/dpif-netdev-private-extract.h | 12 ++ 5 files changed, 371 insertions(+), 5 deletions(-) diff --git a/NEWS b/NEWS index eece0d0b2..f963967d8 100644 --- a/NEWS +++ b/NEWS @@ -29,7 +29,11 @@ Post-v2.17.0 - Windows: * Conntrack support for TCPv6, UDPv6, ICMPv6, FTPv6. * IPv6 Geneve tunnel support. - + - Userspace datapath: + * Add AVX512 optimized profiles to miniflow extract for IPv6/UDP and + IPv6/TCP. + * Add AVX512 optimized profiles to miniflow extract for VLAN/IPv6/UDP + and VLAN/IPv6/TCP. v2.17.0 - 17 Feb 2022 --------------------- diff --git a/lib/automake.mk b/lib/automake.mk index a23cdc4ad..139220c23 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -33,6 +33,7 @@ lib_libopenvswitchavx512_la_CFLAGS = \ -mavx512f \ -mavx512bw \ -mavx512dq \ + -mavx512vl \ -mbmi \ -mbmi2 \ -fPIC \ diff --git a/lib/dpif-netdev-extract-avx512.c b/lib/dpif-netdev-extract-avx512.c index e77bb3214..708cf657a 100644 --- a/lib/dpif-netdev-extract-avx512.c +++ b/lib/dpif-netdev-extract-avx512.c @@ -49,6 +49,7 @@ #include "dpif-netdev-private-extract.h" #include "dpif-netdev-private-flow.h" #include "dp-packet.h" +#include "packets.h" /* AVX512-BW level permutex2var_epi8 emulation. */ static inline __m512i @@ -138,6 +139,7 @@ _mm512_maskz_permutexvar_epi8_wrap(__mmask64 kmask, __m512i idx, __m512i a) #define PATTERN_ETHERTYPE_MASK PATTERN_ETHERTYPE_GEN(0xFF, 0xFF) #define PATTERN_ETHERTYPE_IPV4 PATTERN_ETHERTYPE_GEN(0x08, 0x00) #define PATTERN_ETHERTYPE_DT1Q PATTERN_ETHERTYPE_GEN(0x81, 0x00) +#define PATTERN_ETHERTYPE_IPV6 PATTERN_ETHERTYPE_GEN(0x86, 0xDD) /* VLAN (Dot1Q) patterns and masks. */ #define PATTERN_DT1Q_MASK \ @@ -202,6 +204,40 @@ _mm512_maskz_permutexvar_epi8_wrap(__mmask64 kmask, __m512i idx, __m512i a) NU, NU, NU, NU, NU, NU, NU, NU, 38, 39, 40, 41, NU, NU, NU, NU, /* TCP */ \ NU, NU, NU, NU, NU, NU, NU, NU, /* Unused. */ +/* Generator for checking IPv6 ver. */ +#define PATTERN_IPV6_GEN(VER_TRC, PROTO) \ + VER_TRC, /* Version: 4bits and Traffic class: 4bits. */ \ + 0, 0, 0, /* Traffic class: 4bits and Flow Label: 24bits. */ \ + 0, 0, /* Payload length 16bits. */ \ + PROTO, 0, /* Next Header 8bits and Hop limit 8bits. */ \ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* Src IP: 128bits. */ \ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* Dst IP: 128bits. */ + +#define PATTERN_IPV6_MASK PATTERN_IPV6_GEN(0xF0, 0xFF) +#define PATTERN_IPV6_UDP PATTERN_IPV6_GEN(0x60, 0x11) +#define PATTERN_IPV6_TCP PATTERN_IPV6_GEN(0x60, 0x06) + +#define PATTERN_IPV6_SHUFFLE \ + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, NU, NU, /* Ether */ \ + 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, /* IPv6 */ \ + 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, /* IPv6 */ \ + NU, NU, NU, NU, NU, NU, NU, NU, NU, NU, NU, NU, NU, NU, NU, NU, /* Unused */ + +/* VLAN (Dot1Q) patterns and masks. */ +#define PATTERN_DT1Q_MASK \ + 0x00, 0x00, 0xFF, 0xFF, +#define PATTERN_DT1Q_IPV6 \ + 0x00, 0x00, 0x86, 0xDD, + +#define PATTERN_DT1Q_IPV6_SHUFFLE \ + /* Ether (2 blocks): Note that *VLAN* type is written here. */ \ + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 16, 17, 0, 0, \ + /* VLAN (1 block): Note that the *EtherHdr->Type* is written here. */ \ + 12, 13, 14, 15, 0, 0, 0, 0, \ + 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, /* IPv6 */ \ + 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, /* IPv6 */ \ + NU, NU, NU, NU, NU, NU, NU, NU, /* Unused */ + /* Generation of K-mask bitmask values, to zero out data in result. Note that * these correspond 1:1 to the above "*_SHUFFLE" values, and bit used must be * set in this K-mask, and "NU" values must be zero in the k-mask. Each mask @@ -214,6 +250,10 @@ _mm512_maskz_permutexvar_epi8_wrap(__mmask64 kmask, __m512i idx, __m512i a) #define KMASK_IPV4 0xF0FFULL #define KMASK_UDP 0x000FULL #define KMASK_TCP 0x0F00ULL +#define KMASK_IPV6 0xFFFFULL +#define KMASK_ETHER_IPV6 0x3FFFULL +#define KMASK_DT1Q_IPV6 0xFF0FULL +#define KMASK_IPV6_NOHDR 0x00FFULL #define PATTERN_IPV4_UDP_KMASK \ (KMASK_ETHER | (KMASK_IPV4 << 16) | (KMASK_UDP << 32)) @@ -227,6 +267,13 @@ _mm512_maskz_permutexvar_epi8_wrap(__mmask64 kmask, __m512i idx, __m512i a) #define PATTERN_DT1Q_IPV4_TCP_KMASK \ (KMASK_ETHER | (KMASK_DT1Q << 16) | (KMASK_IPV4 << 24) | (KMASK_TCP << 40)) +#define PATTERN_IPV6_KMASK \ + (KMASK_ETHER_IPV6 | (KMASK_IPV6 << 16) | (KMASK_IPV6 << 32)) + +#define PATTERN_DT1Q_IPV6_KMASK \ + (KMASK_ETHER_IPV6 | (KMASK_DT1Q_IPV6 << 16) | (KMASK_IPV6 << 32) | \ + (KMASK_IPV6_NOHDR << 48)) + /* Miniflow Strip post-processing masks. * This allows unsetting specific bits from the resulting miniflow. It is used * for e.g. IPv4 where the "DF" bit is never pushed to the miniflow itself. @@ -250,25 +297,44 @@ _mm512_maskz_permutexvar_epi8_wrap(__mmask64 kmask, __m512i idx, __m512i a) #define PKT_OFFSET_L3_VLAN (ETH_HEADER_LEN + VLAN_HEADER_LEN) #define PKT_OFFSET_L4_IPv4 (ETH_HEADER_LEN + IP_HEADER_LEN) #define PKT_OFFSET_L4_VLAN_IPv4 (PKT_OFFSET_L4_IPv4 + VLAN_HEADER_LEN) +#define PKT_OFFSET_L4_VLAN_IPv6 (PKT_OFFSET_L3_VLAN + IPV6_HEADER_LEN) +#define PKT_OFFSET_L4_IPv6 (PKT_OFFSET_L2 + IPV6_HEADER_LEN) #define PKT_MIN_ETH_IPv4_UDP (PKT_OFFSET_L4_IPv4 + UDP_HEADER_LEN) #define PKT_MIN_ETH_VLAN_IPv4_UDP (PKT_OFFSET_L4_VLAN_IPv4 + UDP_HEADER_LEN) #define PKT_MIN_ETH_IPv4_TCP (PKT_OFFSET_L4_IPv4 + TCP_HEADER_LEN) #define PKT_MIN_ETH_VLAN_IPv4_TCP (PKT_OFFSET_L4_VLAN_IPv4 + TCP_HEADER_LEN) +#define PKT_MIN_ETH_IPv6_UDP (PKT_OFFSET_L4_IPv6 + UDP_HEADER_LEN) +#define PKT_MIN_ETH_VLAN_IPv6_UDP (PKT_OFFSET_L4_VLAN_IPv6 + UDP_HEADER_LEN) +#define PKT_MIN_ETH_IPv6_TCP (PKT_OFFSET_L4_IPv6 + TCP_HEADER_LEN) +#define PKT_MIN_ETH_VLAN_IPv6_TCP (PKT_OFFSET_L4_VLAN_IPv6 + TCP_HEADER_LEN) + /* MF bits. */ #define MF_BIT(field) (MAP_1 << ((offsetof(struct flow, field) / 8) % \ MAP_T_BITS)) +#define MF_WORD(field, n_word) \ + (((MAP_1 << n_word) - 1) << ((offsetof(struct flow, field) / 8) % \ + MAP_T_BITS)) #define MF_ETH (MF_BIT(dp_hash) | MF_BIT(in_port) | MF_BIT(packet_type)\ | MF_BIT(dl_dst) | MF_BIT(dl_src)| MF_BIT(dl_type)) - #define MF_ETH_VLAN (MF_ETH | MF_BIT(vlans)) + #define MF_IPV4_UDP (MF_BIT(nw_src) | MF_BIT(ipv6_label) | MF_BIT(tp_src) | \ MF_BIT(tp_dst)) - #define MF_IPV4_TCP (MF_IPV4_UDP | MF_BIT(tcp_flags) | MF_BIT(arp_tha.ea[2])) +#define MF_IPV6_UDP (MF_BIT(ipv6_label) | MF_WORD(ipv6_src, 2) | \ + MF_WORD(ipv6_dst, 2) | MF_BIT(tp_src) | MF_BIT(tp_dst)) +#define MF_IPV6_TCP (MF_IPV6_UDP | MF_BIT(tcp_flags) | MF_BIT(arp_tha.ea[2])) + +#define PATTERN_STRIP_IPV6_MASK \ + NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, \ + NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, \ + NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, \ + NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC, NC + /* This union allows initializing static data as u8, but easily loading it * into AVX512 registers too. The union ensures proper alignment for the zmm. */ @@ -348,6 +414,10 @@ enum MFEX_PROFILES { PROFILE_ETH_IPV4_TCP, PROFILE_ETH_VLAN_IPV4_UDP, PROFILE_ETH_VLAN_IPV4_TCP, + PROFILE_ETH_IPV6_UDP, + PROFILE_ETH_IPV6_TCP, + PROFILE_ETH_VLAN_IPV6_TCP, + PROFILE_ETH_VLAN_IPV6_UDP, PROFILE_COUNT, }; @@ -441,8 +511,139 @@ static const struct mfex_profile mfex_profiles[PROFILE_COUNT] = }, .dp_pkt_min_size = PKT_MIN_ETH_VLAN_IPv4_TCP, }, + + [PROFILE_ETH_IPV6_UDP] = { + .probe_mask.u8_data = { PATTERN_ETHERTYPE_MASK PATTERN_IPV6_MASK }, + .probe_data.u8_data = { PATTERN_ETHERTYPE_IPV6 PATTERN_IPV6_UDP }, + + .store_shuf.u8_data = { PATTERN_IPV6_SHUFFLE }, + .strip_mask.u8_data = { PATTERN_STRIP_IPV6_MASK }, + .store_kmsk = PATTERN_IPV6_KMASK, + + .mf_bits = { MF_ETH, MF_IPV6_UDP}, + .dp_pkt_offs = { + 0, UINT16_MAX, PKT_OFFSET_L2, PKT_OFFSET_L4_IPv6, + }, + .dp_pkt_min_size = PKT_MIN_ETH_IPv6_UDP, + }, + + [PROFILE_ETH_IPV6_TCP] = { + .probe_mask.u8_data = { PATTERN_ETHERTYPE_MASK PATTERN_IPV6_MASK }, + .probe_data.u8_data = { PATTERN_ETHERTYPE_IPV6 PATTERN_IPV6_TCP }, + + .store_shuf.u8_data = { PATTERN_IPV6_SHUFFLE }, + .strip_mask.u8_data = { PATTERN_STRIP_IPV6_MASK }, + .store_kmsk = PATTERN_IPV6_KMASK, + + .mf_bits = { MF_ETH, MF_IPV6_TCP}, + .dp_pkt_offs = { + 0, UINT16_MAX, PKT_OFFSET_L2, PKT_OFFSET_L4_IPv6, + }, + .dp_pkt_min_size = PKT_MIN_ETH_IPv6_TCP, + }, + + [PROFILE_ETH_VLAN_IPV6_TCP] = { + .probe_mask.u8_data = { + PATTERN_ETHERTYPE_MASK PATTERN_DT1Q_MASK PATTERN_IPV6_MASK }, + .probe_data.u8_data = { + PATTERN_ETHERTYPE_DT1Q PATTERN_DT1Q_IPV6 PATTERN_IPV6_TCP }, + + .store_shuf.u8_data = { PATTERN_DT1Q_IPV6_SHUFFLE }, + .strip_mask.u8_data = { PATTERN_STRIP_IPV6_MASK }, + .store_kmsk = PATTERN_DT1Q_IPV6_KMASK, + + .mf_bits = { MF_ETH_VLAN, MF_IPV6_TCP}, + .dp_pkt_offs = { + PKT_OFFSET_L2, UINT16_MAX, PKT_OFFSET_L3_VLAN, + PKT_OFFSET_L4_VLAN_IPv6, + }, + .dp_pkt_min_size = PKT_MIN_ETH_VLAN_IPv6_TCP, + }, + + [PROFILE_ETH_VLAN_IPV6_UDP] = { + .probe_mask.u8_data = { + PATTERN_ETHERTYPE_MASK PATTERN_DT1Q_MASK PATTERN_IPV6_MASK }, + .probe_data.u8_data = { + PATTERN_ETHERTYPE_DT1Q PATTERN_DT1Q_IPV6 PATTERN_IPV6_UDP }, + + .store_shuf.u8_data = { PATTERN_DT1Q_IPV6_SHUFFLE }, + .strip_mask.u8_data = { PATTERN_STRIP_IPV6_MASK }, + .store_kmsk = PATTERN_DT1Q_IPV6_KMASK, + + .mf_bits = { MF_ETH_VLAN, MF_IPV6_UDP}, + .dp_pkt_offs = { + PKT_OFFSET_L2, UINT16_MAX, PKT_OFFSET_L3_VLAN, + PKT_OFFSET_L4_VLAN_IPv6, + }, + .dp_pkt_min_size = PKT_MIN_ETH_VLAN_IPv6_UDP, + }, }; +/* IPv6 header helper function to fix TC, flow label and next header. */ +static inline void ALWAYS_INLINE +mfex_handle_ipv6_hdr_block(const uint8_t *ipv6, uint64_t *block) +{ + static const uint8_t data_shuf[16] = { + 0, 1, 2, 3, /* copy IPv6 label in place, it is masked later. */ + 1, 0, /* Byte-swap TC fields for LE usage. */ + 7, 6, /* Move TTL and next proto to MF required locations. */ + }; + + /* BE mask for IPv6 label, and mask to strip away unwanted TC bits. */ + const uint64_t mask = 0xffff0f00 | (UINT64_MAX << 40); + uint64_t mask_data[2] = { mask, mask }; + + /* Load constant data. Is lifted to occur 1x per burst, not per packet. */ + __m128i ipv6_hdr = _mm_loadu_si128((void *) ipv6); + __m128i v_mask = _mm_loadu_si128((void *) mask_data); + __m128i v_shuf_mask = _mm_loadu_si128((void *) data_shuf); + + /* Shuffle data layout, shift 16-bits to get TC fixed, mask to cleanup. */ + __m128i v_ipv6 = _mm_shuffle_epi8(ipv6_hdr, v_shuf_mask); + __m128i v_tc_shift = _mm_mask_slli_epi16(v_ipv6, 0b100, v_ipv6, 4); + __m128i v_ipv6_m = _mm_and_si128(v_tc_shift, v_mask); + + *block = _mm_extract_epi64(v_ipv6_m, 0); +} + +/* IPv6 Protocol specific helper functions, for handling L4 UDP/TCP. */ +static inline void +mfex_handle_ipv6_l4(const uint8_t *ports, uint64_t *block) +{ + void *ptr_ports = (void *) ports; + uint32_t l4_ports = *(uint32_t *) ptr_ports; + *block = l4_ports; +} + +/* IPv6 specific helper functions, for calculating offsets/lengths. */ +static int +mfex_ipv6_set_l2_pad_size(struct dp_packet *pkt, + struct ovs_16aligned_ip6_hdr *nh, + uint32_t len_from_ipv6, + uint32_t next_hdr_size) +{ + /* Handle dynamic l2_pad_size. */ + uint16_t p_len = ntohs(nh->ip6_ctlun.ip6_un1.ip6_un1_plen); + + /* Error if IP total length is greater than remaining packet size. */ + bool err_ipv6_len_too_high = p_len + IPV6_HEADER_LEN > len_from_ipv6; + + /* Plen must be greater then the l4 packet header. */ + bool err_ipv6_len_too_low = p_len < next_hdr_size; + + bool err_packet_size_low = len_from_ipv6 < sizeof *nh; + + /* Ensure the l2 pad size will not overflow. */ + bool err_len_u16_overflow = (len_from_ipv6 - (p_len + IPV6_HEADER_LEN)) + > UINT16_MAX; + + if (OVS_UNLIKELY(err_ipv6_len_too_high || err_ipv6_len_too_low || + err_len_u16_overflow || err_packet_size_low)) { + return -1; + } + dp_packet_set_l2_pad_size(pkt, len_from_ipv6 - (p_len + IPV6_HEADER_LEN)); + return 0; +} /* Protocol specific helper functions, for calculating offsets/lenghts. */ static int32_t @@ -493,6 +694,14 @@ mfex_handle_tcp_flags(const struct tcp_header *tcp, uint64_t *block) *block = ctl_u64 << 32; } +static int +mfex_check_tcp_data_offset(const struct tcp_header *tcp) +{ + /* we dont support TCP options, offset must be 5. */ + bool ret = TCP_OFFSET(tcp->tcp_ctl) == 5; + return ret; +} + /* Generic loop to process any mfex profile. This code is specialized into * multiple actual MFEX implementation functions. Its marked ALWAYS_INLINE * to ensure the compiler specializes each instance. The code is marked "hot" @@ -643,6 +852,94 @@ mfex_avx512_process(struct dp_packet_batch *packets, } dp_packet_update_rss_hash_ipv4_tcp_udp(packet); } break; + + case PROFILE_ETH_IPV6_UDP: { + /* Handle dynamic l2_pad_size. */ + uint32_t size_from_ipv6 = size - sizeof(struct eth_header); + struct ovs_16aligned_ip6_hdr *nh = (void *)&pkt[sizeof + (struct eth_header)]; + if (mfex_ipv6_set_l2_pad_size(packet, nh, size_from_ipv6, + UDP_HEADER_LEN)) { + continue; + } + + /* Process IPv6 header for TC, flow Label and next header. */ + mfex_handle_ipv6_hdr_block(&pkt[ETH_HEADER_LEN], &blocks[8]); + + /* Process UDP header. */ + mfex_handle_ipv6_l4((void *)&pkt[54], &blocks[9]); + + } break; + + case PROFILE_ETH_IPV6_TCP: { + /* Handle dynamic l2_pad_size. */ + uint32_t size_from_ipv6 = size - sizeof(struct eth_header); + struct ovs_16aligned_ip6_hdr *nh = (void *)&pkt[sizeof + (struct eth_header)]; + if (mfex_ipv6_set_l2_pad_size(packet, nh, size_from_ipv6, + TCP_HEADER_LEN)) { + continue; + } + + /* Process IPv6 header for TC, flow Label and next header. */ + mfex_handle_ipv6_hdr_block(&pkt[ETH_HEADER_LEN], &blocks[8]); + + /* Process TCP header. */ + mfex_handle_ipv6_l4((void *)&pkt[54], &blocks[10]); + const struct tcp_header *tcp = (void *)&pkt[54]; + if (!mfex_check_tcp_data_offset(tcp)) { + continue; + } + mfex_handle_tcp_flags(tcp, &blocks[9]); + + } break; + + case PROFILE_ETH_VLAN_IPV6_TCP: { + mfex_vlan_pcp(pkt[14], &keys[i].buf[4]); + + /* Handle dynamic l2_pad_size. */ + uint32_t size_from_ipv6 = size - VLAN_ETH_HEADER_LEN; + struct ovs_16aligned_ip6_hdr *nh = (void *)&pkt + [VLAN_ETH_HEADER_LEN]; + if (mfex_ipv6_set_l2_pad_size(packet, nh, size_from_ipv6, + TCP_HEADER_LEN)) { + continue; + } + + /* Process IPv6 header for TC, flow Label and next header. */ + mfex_handle_ipv6_hdr_block(&pkt[VLAN_ETH_HEADER_LEN], + &blocks[9]); + + /* Process TCP header. */ + mfex_handle_ipv6_l4((void *)&pkt[58], &blocks[11]); + const struct tcp_header *tcp = (void *)&pkt[58]; + if (!mfex_check_tcp_data_offset(tcp)) { + continue; + } + mfex_handle_tcp_flags(tcp, &blocks[10]); + + } break; + + case PROFILE_ETH_VLAN_IPV6_UDP: { + mfex_vlan_pcp(pkt[14], &keys[i].buf[4]); + + /* Handle dynamic l2_pad_size. */ + uint32_t size_from_ipv6 = size - VLAN_ETH_HEADER_LEN; + struct ovs_16aligned_ip6_hdr *nh = (void *)&pkt + [VLAN_ETH_HEADER_LEN]; + if (mfex_ipv6_set_l2_pad_size(packet, nh, size_from_ipv6, + UDP_HEADER_LEN)) { + continue; + } + + /* Process IPv6 header for TC, flow Label and next header. */ + mfex_handle_ipv6_hdr_block(&pkt[VLAN_ETH_HEADER_LEN], + &blocks[9]); + + /* Process UDP header. */ + mfex_handle_ipv6_l4((void *)&pkt[58], &blocks[10]); + + } break; default: break; }; @@ -688,7 +985,10 @@ DECLARE_MFEX_FUNC(ip_udp, PROFILE_ETH_IPV4_UDP) DECLARE_MFEX_FUNC(ip_tcp, PROFILE_ETH_IPV4_TCP) DECLARE_MFEX_FUNC(dot1q_ip_udp, PROFILE_ETH_VLAN_IPV4_UDP) DECLARE_MFEX_FUNC(dot1q_ip_tcp, PROFILE_ETH_VLAN_IPV4_TCP) - +DECLARE_MFEX_FUNC(ipv6_udp, PROFILE_ETH_IPV6_UDP) +DECLARE_MFEX_FUNC(ipv6_tcp, PROFILE_ETH_IPV6_TCP) +DECLARE_MFEX_FUNC(dot1q_ipv6_tcp, PROFILE_ETH_VLAN_IPV6_TCP) +DECLARE_MFEX_FUNC(dot1q_ipv6_udp, PROFILE_ETH_VLAN_IPV6_UDP) static int32_t avx512_isa_probe(uint32_t needs_vbmi) diff --git a/lib/dpif-netdev-private-extract.c b/lib/dpif-netdev-private-extract.c index 4b2f12015..c11309def 100644 --- a/lib/dpif-netdev-private-extract.c +++ b/lib/dpif-netdev-private-extract.c @@ -93,7 +93,56 @@ static struct dpif_miniflow_extract_impl mfex_impls[] = { [MFEX_IMPL_DOT1Q_IPv4_TCP] = { .probe = mfex_avx512_probe, .extract_func = mfex_avx512_dot1q_ip_tcp, - .name = "avx512_dot1q_ipv4_tcp", }, + .name = "avx512_dot1q_ipv4_tcp", + }, + + [MFEX_IMPL_VMBI_IPv6_UDP] = { + .probe = mfex_avx512_vbmi_probe, + .extract_func = mfex_avx512_vbmi_ipv6_udp, + .name = "avx512_vbmi_ipv6_udp", + }, + + [MFEX_IMPL_IPv6_UDP] = { + .probe = mfex_avx512_probe, + .extract_func = mfex_avx512_ipv6_udp, + .name = "avx512_ipv6_udp", + }, + + [MFEX_IMPL_VMBI_IPv6_TCP] = { + .probe = mfex_avx512_vbmi_probe, + .extract_func = mfex_avx512_vbmi_ipv6_tcp, + .name = "avx512_vbmi_ipv6_tcp", + }, + + [MFEX_IMPL_IPv6_TCP] = { + .probe = mfex_avx512_probe, + .extract_func = mfex_avx512_ipv6_tcp, + .name = "avx512_ipv6_tcp", + }, + + [MFEX_IMPL_VMBI_DOT1Q_IPv6_TCP] = { + .probe = mfex_avx512_vbmi_probe, + .extract_func = mfex_avx512_vbmi_dot1q_ipv6_tcp, + .name = "avx512_vbmi_avx512_dot1q_ipv6_tcp", + }, + + [MFEX_IMPL_DOT1Q_IPv6_TCP] = { + .probe = mfex_avx512_probe, + .extract_func = mfex_avx512_dot1q_ipv6_tcp, + .name = "avx512_dot1q_ipv6_tcp", + }, + + [MFEX_IMPL_VMBI_DOT1Q_IPv6_UDP] = { + .probe = mfex_avx512_vbmi_probe, + .extract_func = mfex_avx512_vbmi_dot1q_ipv6_udp, + .name = "avx512_vbmi_avx512_dot1q_ipv6_udp", + }, + + [MFEX_IMPL_DOT1Q_IPv6_UDP] = { + .probe = mfex_avx512_probe, + .extract_func = mfex_avx512_dot1q_ipv6_udp, + .name = "avx512_dot1q_ipv6_udp", + }, #endif }; diff --git a/lib/dpif-netdev-private-extract.h b/lib/dpif-netdev-private-extract.h index f9a757ba4..e10d840f3 100644 --- a/lib/dpif-netdev-private-extract.h +++ b/lib/dpif-netdev-private-extract.h @@ -90,6 +90,14 @@ enum dpif_miniflow_extract_impl_idx { MFEX_IMPL_DOT1Q_IPv4_UDP, MFEX_IMPL_VMBI_DOT1Q_IPv4_TCP, MFEX_IMPL_DOT1Q_IPv4_TCP, + MFEX_IMPL_VMBI_IPv6_UDP, + MFEX_IMPL_IPv6_UDP, + MFEX_IMPL_VMBI_IPv6_TCP, + MFEX_IMPL_IPv6_TCP, + MFEX_IMPL_VMBI_DOT1Q_IPv6_TCP, + MFEX_IMPL_DOT1Q_IPv6_TCP, + MFEX_IMPL_VMBI_DOT1Q_IPv6_UDP, + MFEX_IMPL_DOT1Q_IPv6_UDP, #endif MFEX_IMPL_MAX }; @@ -197,6 +205,10 @@ DECLARE_AVX512_MFEX_PROTOTYPE(ip_udp); DECLARE_AVX512_MFEX_PROTOTYPE(ip_tcp); DECLARE_AVX512_MFEX_PROTOTYPE(dot1q_ip_udp); DECLARE_AVX512_MFEX_PROTOTYPE(dot1q_ip_tcp); +DECLARE_AVX512_MFEX_PROTOTYPE(ipv6_udp); +DECLARE_AVX512_MFEX_PROTOTYPE(ipv6_tcp); +DECLARE_AVX512_MFEX_PROTOTYPE(dot1q_ipv6_tcp); +DECLARE_AVX512_MFEX_PROTOTYPE(dot1q_ipv6_udp); #endif /* __x86_64__ */ From patchwork Fri May 6 05:23:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Amber X-Patchwork-Id: 1627400 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=KQB9xHBZ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KvfhH27wfz9sBF for ; Fri, 6 May 2022 15:46:39 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 36AF06112F; Fri, 6 May 2022 05:46:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OVBS8OQpn4nW; Fri, 6 May 2022 05:46:35 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id EE72561158; Fri, 6 May 2022 05:46:33 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 96F01C007B; Fri, 6 May 2022 05:46:33 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3F962C0032 for ; Fri, 6 May 2022 05:46:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id C25BC84003 for ; Fri, 6 May 2022 05:46:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N08lkk29YB4C for ; Fri, 6 May 2022 05:46:26 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by smtp1.osuosl.org (Postfix) with ESMTPS id E033980B1F for ; Fri, 6 May 2022 05:46:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651815985; x=1683351985; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CRYqz4+rnuACfSdeWLkwVyQoqTbsvYwK/yl7x4tOtgU=; b=KQB9xHBZaJI7Ld4nNW5L/TpZyfwZpAOa9SpJwDiwhpgQW5Yv00QVhfnG 2OPqrdVFNQSFNemqbnkfU3BGpady89uaOGdcG4X9zVWgHlco3EYt7i253 FOxGm1SnqCSWsm2eOPTO+XS1xSLdYNhedZmAfDd3k7pL6ylhdrElNH7Td NduMPiImmFxv4MZ+m/qGC/7dLzygfWtBaXiuABwG/ZTX7Tlo8/0eHOv/G 4qIZbN+NSTGjVxxDsOZKcffRM4xKEJjN76Ir2sb/bVoygiFmObMkMqhjg IY+S0+P2bXkWNo48/KgZ8UFfT1xyZOqNNN2M5BG2i2gwqBrDrvDRIyi5g w==; X-IronPort-AV: E=McAfee;i="6400,9594,10338"; a="268503860" X-IronPort-AV: E=Sophos;i="5.91,203,1647327600"; d="scan'208";a="268503860" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 May 2022 22:46:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,203,1647327600"; d="scan'208";a="537727417" Received: from unknown (HELO localhost.localdomain) ([10.190.213.111]) by orsmga006.jf.intel.com with ESMTP; 05 May 2022 22:46:22 -0700 From: Kumar Amber To: ovs-dev@openvswitch.org Date: Fri, 6 May 2022 10:53:26 +0530 Message-Id: <20220506052326.3191931-5-kumar.amber@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220506052326.3191931-1-kumar.amber@intel.com> References: <20220506052326.3191931-1-kumar.amber@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, fbl@sysclose.org, Kumar Amber , david.marchand@redhat.com Subject: [ovs-dev] [PATCH v7 4/4] dpif-netdev/mfex: Add ipv6 profile based hashing. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" For packets which don't already have a hash calculated, miniflow_hash_5tuple() calculates the hash of a packet using the previously built miniflow. This commit adds IPv6 profile specific hashing which uses fixed offsets into the packet to improve hashing performance. Signed-off-by: Kumar Amber Acked-by: Harry van Haaren --- lib/dp-packet.h | 43 ++++++++++++++++++++++++++++++++ lib/dpif-netdev-extract-avx512.c | 8 +++--- lib/flow.c | 4 +++ 3 files changed, 51 insertions(+), 4 deletions(-) diff --git a/lib/dp-packet.h b/lib/dp-packet.h index 7c5da258a..277bb51b8 100644 --- a/lib/dp-packet.h +++ b/lib/dp-packet.h @@ -1117,6 +1117,49 @@ dp_packet_update_rss_hash_ipv4_tcp_udp(struct dp_packet *packet) dp_packet_set_rss_hash(packet, hash); } +static inline void ALWAYS_INLINE +dp_packet_update_rss_hash_ipv6_tcp_udp(struct dp_packet *packet) +{ + if (dp_packet_rss_valid(packet)) { + return; + } + + const uint8_t *pkt = dp_packet_data(packet); + const uint16_t l3_ofs = packet->l3_ofs; + uint32_t ipv6_src_off = offsetof(struct ovs_16aligned_ip6_hdr, ip6_src); + uint32_t ipv6_dst_off = offsetof(struct ovs_16aligned_ip6_hdr, ip6_dst); + uint32_t ipv6_proto_off = offsetof(struct ovs_16aligned_ip6_hdr, + ip6_ctlun.ip6_un1.ip6_un1_nxt); + const void *ipv6_src_l = &pkt[l3_ofs + ipv6_src_off]; + const void *ipv6_src_h = &pkt[l3_ofs + ipv6_src_off + 8]; + const void *ipv6_dst_l = &pkt[l3_ofs + ipv6_dst_off]; + const void *ipv6_dst_h = &pkt[l3_ofs + ipv6_dst_off + 8]; + const void *l4_ports = &pkt[packet->l4_ofs]; + uint64_t ipv6_src_lo, ipv6_src_hi; + uint64_t ipv6_dst_lo, ipv6_dst_hi; + uint32_t ports; + uint32_t hash = 0; + + memcpy(&ipv6_src_lo, ipv6_src_l, sizeof ipv6_src_lo); + memcpy(&ipv6_src_hi, ipv6_src_h, sizeof ipv6_src_hi); + memcpy(&ipv6_dst_lo, ipv6_dst_l, sizeof ipv6_dst_lo); + memcpy(&ipv6_dst_hi, ipv6_dst_h, sizeof ipv6_dst_hi); + memcpy(&ports, l4_ports, sizeof ports); + + /* IPv6 Src and Dst. */ + hash = hash_add64(hash, ipv6_src_lo); + hash = hash_add64(hash, ipv6_src_hi); + hash = hash_add64(hash, ipv6_dst_lo); + hash = hash_add64(hash, ipv6_dst_hi); + /* IPv6 proto. */ + hash = hash_add(hash, pkt[l3_ofs + ipv6_proto_off]); + /* L4 ports. */ + hash = hash_add(hash, ports); + hash = hash_finish(hash, 42); + + dp_packet_set_rss_hash(packet, hash); +} + #ifdef __cplusplus } #endif diff --git a/lib/dpif-netdev-extract-avx512.c b/lib/dpif-netdev-extract-avx512.c index 708cf657a..838136e45 100644 --- a/lib/dpif-netdev-extract-avx512.c +++ b/lib/dpif-netdev-extract-avx512.c @@ -868,7 +868,7 @@ mfex_avx512_process(struct dp_packet_batch *packets, /* Process UDP header. */ mfex_handle_ipv6_l4((void *)&pkt[54], &blocks[9]); - + dp_packet_update_rss_hash_ipv6_tcp_udp(packet); } break; case PROFILE_ETH_IPV6_TCP: { @@ -891,7 +891,7 @@ mfex_avx512_process(struct dp_packet_batch *packets, continue; } mfex_handle_tcp_flags(tcp, &blocks[9]); - + dp_packet_update_rss_hash_ipv6_tcp_udp(packet); } break; case PROFILE_ETH_VLAN_IPV6_TCP: { @@ -917,7 +917,7 @@ mfex_avx512_process(struct dp_packet_batch *packets, continue; } mfex_handle_tcp_flags(tcp, &blocks[10]); - + dp_packet_update_rss_hash_ipv6_tcp_udp(packet); } break; case PROFILE_ETH_VLAN_IPV6_UDP: { @@ -938,7 +938,7 @@ mfex_avx512_process(struct dp_packet_batch *packets, /* Process UDP header. */ mfex_handle_ipv6_l4((void *)&pkt[58], &blocks[10]); - + dp_packet_update_rss_hash_ipv6_tcp_udp(packet); } break; default: break; diff --git a/lib/flow.c b/lib/flow.c index 8ab9df3fc..b1e1fb34d 100644 --- a/lib/flow.c +++ b/lib/flow.c @@ -1019,6 +1019,8 @@ miniflow_extract(struct dp_packet *packet, struct miniflow *dst) miniflow_push_be16(mf, ct_tp_dst, ct_tp_dst); if (dl_type == htons(ETH_TYPE_IP)) { dp_packet_update_rss_hash_ipv4_tcp_udp(packet); + } else if (dl_type == htons(ETH_TYPE_IPV6)) { + dp_packet_update_rss_hash_ipv6_tcp_udp(packet); } } } @@ -1032,6 +1034,8 @@ miniflow_extract(struct dp_packet *packet, struct miniflow *dst) miniflow_push_be16(mf, ct_tp_dst, ct_tp_dst); if (dl_type == htons(ETH_TYPE_IP)) { dp_packet_update_rss_hash_ipv4_tcp_udp(packet); + } else if (dl_type == htons(ETH_TYPE_IPV6)) { + dp_packet_update_rss_hash_ipv6_tcp_udp(packet); } } } else if (OVS_LIKELY(nw_proto == IPPROTO_SCTP)) {