From patchwork Mon Dec 4 07:01:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: liuhongt X-Patchwork-Id: 1871395 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=iWskxna8; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SkF4n73R3z23mf for ; Mon, 4 Dec 2023 18:03:37 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E979E385734D for ; Mon, 4 Dec 2023 07:03:35 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.9]) by sourceware.org (Postfix) with ESMTPS id B8B0C3857711 for ; Mon, 4 Dec 2023 07:03:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B8B0C3857711 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org B8B0C3857711 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=198.175.65.9 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1701673404; cv=none; b=YvINzU+ol/HOQnI0tHaupWEAEvomcO1TgN+5y3334zDZaNkcb9dX5FoYlt7gsRxsnhodVMXptxvoJaiRWyhKy/zkIiTNVUxTIQ2uKEyU5Go3XPOdzKTsw8CiZ0tn23t6U2fsNOjkEBSwykQsZogu9k7jH0yqg7hzRr2V2fkWL+k= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1701673404; c=relaxed/simple; bh=CqNkHSBcafjXNcmkePy3dUKQrRVCXgi8Rk5sVFb9QtA=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=X65h3xZV9h2l2Xj5lC8cUDwD7120ZEK07IJh5BVOhFTMue5kp5T0MLteR+q9EHe7Q4ocmKPUf7Yp46K1Z0cSj6R4Kg5ykrdMGUZ/mEASUibhU6akxuoo2/89IP1hBE2XQHt60o+6sKfv1bJclVYa7ZN2ykJXC8u2FLe7inUXORo= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1701673403; x=1733209403; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=CqNkHSBcafjXNcmkePy3dUKQrRVCXgi8Rk5sVFb9QtA=; b=iWskxna8HBRIlqVMxBuC5tQ1WJwKUpVTQYPIfm6JyD7QNbisvnlVanLp xngv9cu/o6YkEGIPy6K/Oa+dsXUwy6pJ6L5ZHeorkOAt43Yngfb4xNAdE 6R3GYeSqVVI+AQ6WFgFtKKH7QTR2f/KYPXg7cf52j4CQ+Tw3Mlbghi0Mm abWhrhXt0M1ihvspDYqvxeJRMvstgJbrPNgRtQnsyFrzkxis+zonVnXWp dBSCiiofgwiFwdFev2/eD5skU0DrVsvGYfmBOtxB5Fj0CTV2HRwI6e+ZX Jv0G9lQQw+8X31ebmooFr34XX3G+DW7C8UlyrmU4lsis7g06hvo8dMvwX Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10913"; a="12407437" X-IronPort-AV: E=Sophos;i="6.04,249,1695711600"; d="scan'208";a="12407437" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orvoesa101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Dec 2023 23:03:22 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10913"; a="763847610" X-IronPort-AV: E=Sophos;i="6.04,249,1695711600"; d="scan'208";a="763847610" Received: from shvmail03.sh.intel.com ([10.239.245.20]) by orsmga007.jf.intel.com with ESMTP; 03 Dec 2023 23:03:19 -0800 Received: from shliclel4217.sh.intel.com (shliclel4217.sh.intel.com [10.239.240.127]) by shvmail03.sh.intel.com (Postfix) with ESMTP id C4B9F100568B; Mon, 4 Dec 2023 15:03:18 +0800 (CST) From: liuhongt To: gcc-patches@gcc.gnu.org Cc: crazylht@gmail.com, hjl.tools@gmail.com Subject: [PATCH] Support udot_prodv*qi with emulation sdot_prodv*hi Date: Mon, 4 Dec 2023 15:01:18 +0800 Message-Id: <20231204070118.1092995-1-hongtao.liu@intel.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Like r14-5990-gb4a7c1c8c59d19, but the patch optimized for udot_prod. Since (zero_extend) (unsigned char)-> int is equal to (zero_extend)(unsigned char) -> short + (sign_extend) (short) -> int Bootstrapped and regtested on x86_64-pc-linux-gnu{-m32,}. Ready push to trunk. It should be safe to emulate udot_prodv*qi with vec_unpacku_lo_v32qi vec_unpacku_lo_v32qi vec_unpacku_hi_v32qi vec_unpacku_hi_v32qi sdot_prodv16hi sdot_prodv16hi add3v8si gcc/ChangeLog: * config/i386/sse.md (udot_prodv64qi): New expander. (udot_prod): Emulates with VEC_UNPACKU_EXPR + DOT_PROD (short, int). gcc/testsuite/ChangeLog: * gcc.target/i386/udotprodint8_emulate.c: New test. --- gcc/config/i386/sse.md | 82 ++++++++++++++++--- .../gcc.target/i386/udotprodint8_emulate.c | 15 ++++ 2 files changed, 85 insertions(+), 12 deletions(-) create mode 100644 gcc/testsuite/gcc.target/i386/udotprodint8_emulate.c diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md index a1d4fec42a2..3244cef483a 100644 --- a/gcc/config/i386/sse.md +++ b/gcc/config/i386/sse.md @@ -30835,20 +30835,78 @@ (define_expand "sdot_prodv64qi" (define_expand "udot_prod" [(match_operand: 0 "register_operand") - (match_operand:VI1 1 "register_operand") - (match_operand:VI1 2 "register_operand") + (match_operand:VI1_AVX2 1 "register_operand") + (match_operand:VI1_AVX2 2 "register_operand") (match_operand: 3 "register_operand")] - "TARGET_AVXVNNIINT8" + "TARGET_SSE2" { - operands[1] = lowpart_subreg (mode, - force_reg (mode, operands[1]), - mode); - operands[2] = lowpart_subreg (mode, - force_reg (mode, operands[2]), - mode); - emit_insn (gen_rtx_SET (operands[0], operands[3])); - emit_insn (gen_vpdpbuud_ (operands[0], operands[3], - operands[1], operands[2])); + if (TARGET_AVXVNNIINT8) + { + operands[1] = lowpart_subreg (mode, + force_reg (mode, operands[1]), + mode); + operands[2] = lowpart_subreg (mode, + force_reg (mode, operands[2]), + mode); + emit_insn (gen_rtx_SET (operands[0], operands[3])); + emit_insn (gen_vpdpbuud_ (operands[0], operands[3], + operands[1], operands[2])); + } + else + { + /* Emulate with vpdpwssd. */ + rtx op1_lo = gen_reg_rtx (mode); + rtx op1_hi = gen_reg_rtx (mode); + rtx op2_lo = gen_reg_rtx (mode); + rtx op2_hi = gen_reg_rtx (mode); + + emit_insn (gen_vec_unpacku_lo_ (op1_lo, operands[1])); + emit_insn (gen_vec_unpacku_lo_ (op2_lo, operands[2])); + emit_insn (gen_vec_unpacku_hi_ (op1_hi, operands[1])); + emit_insn (gen_vec_unpacku_hi_ (op2_hi, operands[2])); + + rtx res1 = gen_reg_rtx (mode); + rtx res2 = gen_reg_rtx (mode); + rtx sum = gen_reg_rtx (mode); + + emit_move_insn (sum, CONST0_RTX (mode)); + emit_insn (gen_sdot_prod (res1, op1_lo, + op2_lo, sum)); + emit_insn (gen_sdot_prod (res2, op1_hi, + op2_hi, operands[3])); + emit_insn (gen_add3 (operands[0], res1, res2)); + } + + DONE; +}) + +(define_expand "udot_prodv64qi" + [(match_operand:V16SI 0 "register_operand") + (match_operand:V64QI 1 "register_operand") + (match_operand:V64QI 2 "register_operand") + (match_operand:V16SI 3 "register_operand")] + "(TARGET_AVX512VNNI || TARGET_AVX512BW) && TARGET_EVEX512" +{ + /* Emulate with vpdpwssd. */ + rtx op1_lo = gen_reg_rtx (V32HImode); + rtx op1_hi = gen_reg_rtx (V32HImode); + rtx op2_lo = gen_reg_rtx (V32HImode); + rtx op2_hi = gen_reg_rtx (V32HImode); + + emit_insn (gen_vec_unpacku_lo_v64qi (op1_lo, operands[1])); + emit_insn (gen_vec_unpacku_lo_v64qi (op2_lo, operands[2])); + emit_insn (gen_vec_unpacku_hi_v64qi (op1_hi, operands[1])); + emit_insn (gen_vec_unpacku_hi_v64qi (op2_hi, operands[2])); + + rtx res1 = gen_reg_rtx (V16SImode); + rtx res2 = gen_reg_rtx (V16SImode); + rtx sum = gen_reg_rtx (V16SImode); + + emit_move_insn (sum, CONST0_RTX (V16SImode)); + emit_insn (gen_sdot_prodv32hi (res1, op1_lo, op2_lo, sum)); + emit_insn (gen_sdot_prodv32hi (res2, op1_hi, op2_hi, operands[3])); + + emit_insn (gen_addv16si3 (operands[0], res1, res2)); DONE; }) diff --git a/gcc/testsuite/gcc.target/i386/udotprodint8_emulate.c b/gcc/testsuite/gcc.target/i386/udotprodint8_emulate.c new file mode 100644 index 00000000000..1e8f2cfe521 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/udotprodint8_emulate.c @@ -0,0 +1,15 @@ +/* { dg-do compile } */ +/* { dg-options "-mavxvnni -O2 -fdump-tree-optimized" } */ +/* { dg-final { scan-tree-dump-times "DOT_PROD_EXPR" 1 "optimized" } } */ +/* { dg-final { scan-assembler-times "vpdpwssd" 2 } } */ + +int +foo (unsigned char* a, unsigned char* b) +{ + int sum = 0; + for (int i = 0; i != 16; i++) + { + sum += a[i] * b[i]; + } + return sum; +}