From patchwork Fri Jul 14 06:24:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Jiang, Haochen" X-Patchwork-Id: 1807555 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=RFvM9yfW; dkim-atps=neutral Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4R2Lzt5ryKz20bt for ; Fri, 14 Jul 2023 16:24:42 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8FA9C3858017 for ; Fri, 14 Jul 2023 06:24:40 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8FA9C3858017 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1689315880; bh=Wb5UEFj7QOJCkKOGOkOzGtOYHSlIkEbbLPuLZOEOOAc=; h=To:Cc:Subject:Date:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=RFvM9yfW+QRK++S/kDUccYqT+BndCdUhNAt95bdBQuP6FWEgYEUmLITedlIzWn/9Q EJmLDVRiSEq0uMX3KvcXDYigTYxkNN8Tn6LGRdXQ/SCoRxUDGFovP2CHIdBpN/uU/S W2+UIaNQ02xCAUfhjZLdlBdRMjYDd/hXqCyO+kH8= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by sourceware.org (Postfix) with ESMTPS id 2E0E53858CDB for ; Fri, 14 Jul 2023 06:24:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2E0E53858CDB X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="451766896" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="451766896" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2023 23:24:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10770"; a="792331087" X-IronPort-AV: E=Sophos;i="6.01,204,1684825200"; d="scan'208";a="792331087" Received: from shvmail03.sh.intel.com ([10.239.245.20]) by fmsmga004.fm.intel.com with ESMTP; 13 Jul 2023 23:24:14 -0700 Received: from shliclel4217.sh.intel.com (shliclel4217.sh.intel.com [10.239.240.127]) by shvmail03.sh.intel.com (Postfix) with ESMTP id 5B79110056F3; Fri, 14 Jul 2023 14:24:13 +0800 (CST) To: gcc-patches@gcc.gnu.org Cc: ubizjak@gmail.com, hongtao.liu@intel.com Subject: [PATCH] i386: Auto vectorize usdot_prod, udot_prod with AVXVNNIINT16 instruction. Date: Fri, 14 Jul 2023 14:24:13 +0800 Message-Id: <20230714062413.2277485-1-haochen.jiang@intel.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Haochen Jiang via Gcc-patches From: "Jiang, Haochen" Reply-To: Haochen Jiang Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" Hi all, This patch aims to auto vectorize usdot_prod and udot_prod with newly introduced AVX-VNNI-INT16. Also I refined the redundant mode iterator in the patch. Regtested on x86_64-pc-linux-gnu. Ok for trunk after AVX-VNNI-INT16 patch checked in? BRs, Haochen gcc/ChangeLog: * config/i386/sse.md (VI2_AVX2): Delete V32HI since we actually have the same iterator. Also renaming all the occurence to VI2_AVX2_AVX512BW. (usdot_prod): New define_expand. (udot_prod): Ditto. gcc/testsuite/ChangeLog: * gcc.target/i386/vnniint16-auto-vectorize-1.c: New test. * gcc.target/i386/vnniint16-auto-vectorize-2.c: Ditto. --- gcc/config/i386/sse.md | 98 +++++++++++++------ .../i386/vnniint16-auto-vectorize-1.c | 28 ++++++ .../i386/vnniint16-auto-vectorize-2.c | 76 ++++++++++++++ 3 files changed, 172 insertions(+), 30 deletions(-) create mode 100644 gcc/testsuite/gcc.target/i386/vnniint16-auto-vectorize-1.c create mode 100644 gcc/testsuite/gcc.target/i386/vnniint16-auto-vectorize-2.c diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md index 7471932b27e..98e7f9334bc 100644 --- a/gcc/config/i386/sse.md +++ b/gcc/config/i386/sse.md @@ -545,6 +545,9 @@ V32HI (V16HI "TARGET_AVX512VL")]) (define_mode_iterator VI2_AVX2 + [(V16HI "TARGET_AVX2") V8HI]) + +(define_mode_iterator VI2_AVX2_AVX512BW [(V32HI "TARGET_AVX512BW") (V16HI "TARGET_AVX2") V8HI]) (define_mode_iterator VI2_AVX512F @@ -637,9 +640,6 @@ (V16HI "TARGET_AVX2") V8HI (V8SI "TARGET_AVX2") V4SI]) -(define_mode_iterator VI2_AVX2_AVX512BW - [(V32HI "TARGET_AVX512BW") (V16HI "TARGET_AVX2") V8HI]) - (define_mode_iterator VI248_AVX512VL [V32HI V16SI V8DI (V16HI "TARGET_AVX512VL") (V8SI "TARGET_AVX512VL") @@ -15298,16 +15298,16 @@ }) (define_expand "mul3" - [(set (match_operand:VI2_AVX2 0 "register_operand") - (mult:VI2_AVX2 (match_operand:VI2_AVX2 1 "vector_operand") - (match_operand:VI2_AVX2 2 "vector_operand")))] + [(set (match_operand:VI2_AVX2_AVX512BW 0 "register_operand") + (mult:VI2_AVX2_AVX512BW (match_operand:VI2_AVX2_AVX512BW 1 "vector_operand") + (match_operand:VI2_AVX2_AVX512BW 2 "vector_operand")))] "TARGET_SSE2 && && " "ix86_fixup_binary_operands_no_copy (MULT, mode, operands);") (define_insn "*mul3" - [(set (match_operand:VI2_AVX2 0 "register_operand" "=x,") - (mult:VI2_AVX2 (match_operand:VI2_AVX2 1 "vector_operand" "%0,") - (match_operand:VI2_AVX2 2 "vector_operand" "xBm,m")))] + [(set (match_operand:VI2_AVX2_AVX512BW 0 "register_operand" "=x,") + (mult:VI2_AVX2_AVX512BW (match_operand:VI2_AVX2_AVX512BW 1 "vector_operand" "%0,") + (match_operand:VI2_AVX2_AVX512BW 2 "vector_operand" "xBm,m")))] "TARGET_SSE2 && !(MEM_P (operands[1]) && MEM_P (operands[2])) && && " "@ @@ -15320,28 +15320,28 @@ (set_attr "mode" "")]) (define_expand "mul3_highpart" - [(set (match_operand:VI2_AVX2 0 "register_operand") - (truncate:VI2_AVX2 + [(set (match_operand:VI2_AVX2_AVX512BW 0 "register_operand") + (truncate:VI2_AVX2_AVX512BW (lshiftrt: (mult: (any_extend: - (match_operand:VI2_AVX2 1 "vector_operand")) + (match_operand:VI2_AVX2_AVX512BW 1 "vector_operand")) (any_extend: - (match_operand:VI2_AVX2 2 "vector_operand"))) + (match_operand:VI2_AVX2_AVX512BW 2 "vector_operand"))) (const_int 16))))] "TARGET_SSE2 && && " "ix86_fixup_binary_operands_no_copy (MULT, mode, operands);") (define_insn "*mul3_highpart" - [(set (match_operand:VI2_AVX2 0 "register_operand" "=x,") - (truncate:VI2_AVX2 + [(set (match_operand:VI2_AVX2_AVX512BW 0 "register_operand" "=x,") + (truncate:VI2_AVX2_AVX512BW (lshiftrt: (mult: (any_extend: - (match_operand:VI2_AVX2 1 "vector_operand" "%0,")) + (match_operand:VI2_AVX2_AVX512BW 1 "vector_operand" "%0,")) (any_extend: - (match_operand:VI2_AVX2 2 "vector_operand" "xBm,m"))) + (match_operand:VI2_AVX2_AVX512BW 2 "vector_operand" "xBm,m"))) (const_int 16))))] "TARGET_SSE2 && !(MEM_P (operands[1]) && MEM_P (operands[2])) && && " @@ -15591,8 +15591,8 @@ (define_insn "avx512bw_pmaddwd512" [(set (match_operand: 0 "register_operand" "=v") (unspec: - [(match_operand:VI2_AVX2 1 "register_operand" "v") - (match_operand:VI2_AVX2 2 "nonimmediate_operand" "vm")] + [(match_operand:VI2_AVX2_AVX512BW 1 "register_operand" "v") + (match_operand:VI2_AVX2_AVX512BW 2 "nonimmediate_operand" "vm")] UNSPEC_PMADDWD512))] "TARGET_AVX512BW && " "vpmaddwd\t{%2, %1, %0|%0, %1, %2}"; @@ -21569,16 +21569,16 @@ }) (define_expand "smulhrs3" - [(set (match_operand:VI2_AVX2 0 "register_operand") - (truncate:VI2_AVX2 + [(set (match_operand:VI2_AVX2_AVX512BW 0 "register_operand") + (truncate:VI2_AVX2_AVX512BW (lshiftrt: (plus: (lshiftrt: (mult: (sign_extend: - (match_operand:VI2_AVX2 1 "nonimmediate_operand")) + (match_operand:VI2_AVX2_AVX512BW 1 "nonimmediate_operand")) (sign_extend: - (match_operand:VI2_AVX2 2 "nonimmediate_operand"))) + (match_operand:VI2_AVX2_AVX512BW 2 "nonimmediate_operand"))) (const_int 14)) (match_dup 3)) (const_int 1))))] @@ -21589,18 +21589,18 @@ }) (define_insn "*_pmulhrsw3" - [(set (match_operand:VI2_AVX2 0 "register_operand" "=x,") - (truncate:VI2_AVX2 + [(set (match_operand:VI2_AVX2_AVX512BW 0 "register_operand" "=x,") + (truncate:VI2_AVX2_AVX512BW (lshiftrt: (plus: (lshiftrt: (mult: (sign_extend: - (match_operand:VI2_AVX2 1 "vector_operand" "%0,")) + (match_operand:VI2_AVX2_AVX512BW 1 "vector_operand" "%0,")) (sign_extend: - (match_operand:VI2_AVX2 2 "vector_operand" "xBm,m"))) + (match_operand:VI2_AVX2_AVX512BW 2 "vector_operand" "xBm,m"))) (const_int 14)) - (match_operand:VI2_AVX2 3 "const1_operand")) + (match_operand:VI2_AVX2_AVX512BW 3 "const1_operand")) (const_int 1))))] "TARGET_SSSE3 && && && !(MEM_P (operands[1]) && MEM_P (operands[2]))" @@ -22327,8 +22327,8 @@ (set_attr "mode" "")]) (define_insn "_packusdw" - [(set (match_operand:VI2_AVX2 0 "register_operand" "=Yr,*x,") - (unspec:VI2_AVX2 + [(set (match_operand:VI2_AVX2_AVX512BW 0 "register_operand" "=Yr,*x,") + (unspec:VI2_AVX2_AVX512BW [(match_operand: 1 "register_operand" "0,0,") (match_operand: 2 "vector_operand" "YrBm,*xBm,m")] UNSPEC_US_TRUNCATE))] @@ -30340,6 +30340,44 @@ (UNSPEC_VPDPWSUD "wsud") (UNSPEC_VPDPWSUDS "wsuds") (UNSPEC_VPDPWUUD "wuud") (UNSPEC_VPDPWUUDS "wuuds")]) +(define_expand "usdot_prod" + [(match_operand: 0 "register_operand") + (match_operand:VI2_AVX2 1 "register_operand") + (match_operand:VI2_AVX2 2 "register_operand") + (match_operand: 3 "register_operand")] + "TARGET_AVXVNNIINT16" +{ + operands[1] = lowpart_subreg (mode, + force_reg (mode, operands[1]), + mode); + operands[2] = lowpart_subreg (mode, + force_reg (mode, operands[2]), + mode); + emit_insn (gen_rtx_SET (operands[0], operands[3])); + emit_insn (gen_vpdpwusd_ (operands[0], operands[3], + operands[1], operands[2])); + DONE; +}) + +(define_expand "udot_prod" + [(match_operand: 0 "register_operand") + (match_operand:VI2_AVX2 1 "register_operand") + (match_operand:VI2_AVX2 2 "register_operand") + (match_operand: 3 "register_operand")] + "TARGET_AVXVNNIINT16" +{ + operands[1] = lowpart_subreg (mode, + force_reg (mode, operands[1]), + mode); + operands[2] = lowpart_subreg (mode, + force_reg (mode, operands[2]), + mode); + emit_insn (gen_rtx_SET (operands[0], operands[3])); + emit_insn (gen_vpdpwuud_ (operands[0], operands[3], + operands[1], operands[2])); + DONE; +}) + (define_insn "vpdp_" [(set (match_operand:VI4_AVX 0 "register_operand" "=x") (unspec:VI4_AVX diff --git a/gcc/testsuite/gcc.target/i386/vnniint16-auto-vectorize-1.c b/gcc/testsuite/gcc.target/i386/vnniint16-auto-vectorize-1.c new file mode 100644 index 00000000000..73f0d3296aa --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/vnniint16-auto-vectorize-1.c @@ -0,0 +1,28 @@ +/* { dg-do compile } */ +/* { dg-options "-mavxvnniint16 -O2" } */ +/* { dg-final { scan-assembler "vpdpwusd\t" } } */ +/* { dg-final { scan-assembler "vpdpwuud\t" } } */ + +int __attribute__((noinline, noclone, optimize("tree-vectorize"))) +usdot_prod_hi (unsigned short * restrict a, short * restrict b, + int c, int n) +{ + int i; + for (i = 0; i < n; i++) + { + c += ((int) a[i] * (int) b[i]); + } + return c; +} + +int __attribute__((noinline, noclone, optimize("tree-vectorize"))) +udot_prod_hi (unsigned short * restrict a, unsigned short *restrict b, + int c, int n) +{ + int i; + for (i = 0; i < n; i++) + { + c += ((int) a[i] * (int) b[i]); + } + return c; +} diff --git a/gcc/testsuite/gcc.target/i386/vnniint16-auto-vectorize-2.c b/gcc/testsuite/gcc.target/i386/vnniint16-auto-vectorize-2.c new file mode 100644 index 00000000000..90dc0eade7e --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/vnniint16-auto-vectorize-2.c @@ -0,0 +1,76 @@ +/* { dg-do run } */ +/* { dg-options "-O2 -mavxvnniint16" } */ +/* { dg-require-effective-target avxvnniint16 } */ + +#define AVXVNNIINT16 +#ifndef CHECK +#define CHECK "avx-check.h" +#endif + +#ifndef TEST +#define TEST avx_test +#endif + +#include CHECK +#include "vnniint16-auto-vectorize-1.c" + +#define N 256 + +short a_i16[N]; +unsigned short b_u16[N], c_u16[N], d_u16[N]; +int i16_exp, i16_ref; + +int __attribute__((noinline, noclone, optimize("no-tree-vectorize"))) +udot_prod_hi_scalar (unsigned short * restrict a, unsigned short * restrict b, + int c, int n) +{ + int i; + for (i = 0; i < n; i++) + { + c += ((int) a[i] * (int) b[i]); + } + return c; +} + +int __attribute__((noinline, noclone, optimize("no-tree-vectorize"))) +usdot_prod_hi_scalar (unsigned short * restrict a, short *restrict b, + int c, int n) +{ + int i; + for (i = 0; i < n; i++) + { + c += ((int) a[i] * (int) b[i]); + } + return c; +} + +void init () +{ + int i; + + i16_exp = i16_ref = 65535; + + for (i = 0; i < N; i++) + { + a_i16[i] = -i + 2; + b_u16[i] = i * 2; + c_u16[i] = i * 3; + d_u16[i] = i * 4; + } +} + +void +TEST (void) +{ + init (); + i16_exp = usdot_prod_hi (a_i16, b_u16, i16_exp, N); + i16_ref = usdot_prod_hi_scalar (a_i16, b_u16, i16_ref, N); + if (i16_exp != i16_ref) + abort (); + + init (); + i16_exp = udot_prod_hi (c_u16, d_u16, i16_exp, N); + i16_ref = udot_prod_hi_scalar (c_u16, d_u16, i16_ref, N); + if (i16_exp != i16_ref) + abort (); +}