From patchwork Fri Nov 15 18:04:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Joel Hutton X-Patchwork-Id: 1195794 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=gcc-patches-return-513652-incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b="RYzuO+Ua"; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=armh.onmicrosoft.com header.i=@armh.onmicrosoft.com header.b="FmsVHspc"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=armh.onmicrosoft.com header.i=@armh.onmicrosoft.com header.b="FmsVHspc"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47F5pJ13qdz9sPT for ; Sat, 16 Nov 2019 05:04:23 +1100 (AEDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:content-type:mime-version; q=dns; s=default; b=F+tP1mlb+PiAqN6e2+7s28AG+NX+hpb3UXuATFzRVf2DoXbgmR n5oGDxl8FDc7KG7YgKmdFyoENqaaoFKrQwRqKDldX0mQRLx+IZqZevdRjKaLVwJu eiPPAjChZfeytL0LwdyggqtnM25OfR3dyagN6+gA8tLdJ07p59hAgWB9Q= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:content-type:mime-version; s= default; bh=kaCLVNB+TR2w+R3ePRvjHM7h5Lg=; b=RYzuO+UaWrusF1P62Rss SVqYyAgeBo2F6dwLJpZacblr9lggQYTYVZlh2L5M2a+ND3LhR9MPBEkwtdwRq5Om Tt0ORPV8B2XhZZX3ra7Y0kkA4BAahg+egTTSjTOfRAoWuStOuuStQZdktUsXENSB RIEinZ5o5TiOlQznCPpD3vA= Received: (qmail 26519 invoked by alias); 15 Nov 2019 18:04:16 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 26511 invoked by uid 89); 15 Nov 2019 18:04:16 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-23.4 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_PASS autolearn=ham version=3.3.1 spammy= X-HELO: EUR04-DB3-obe.outbound.protection.outlook.com Received: from mail-eopbgr60041.outbound.protection.outlook.com (HELO EUR04-DB3-obe.outbound.protection.outlook.com) (40.107.6.41) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 15 Nov 2019 18:04:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P37L+jAtZS9aN4/0NBSi5TlKI59gZrU7iqVsFBQUBCI=; b=FmsVHspctcEchDevKg/n08bKu/mCtp101dJdo3/OkIuIHExnQb3NF9eaME8Bbc70FT0CLl6z78d19Ud0Zsc+zKHsaQKJd8wjkuuOsiJa7/gjcmnx38EoJeyqU2ClMWFkbUchei7PnWEHQUa7P+1ODBTaSMVntq6ag+C4+pp7xmU= Received: from VI1PR0802CA0045.eurprd08.prod.outlook.com (2603:10a6:800:a9::31) by AM6PR08MB4755.eurprd08.prod.outlook.com (2603:10a6:20b:c2::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.29; Fri, 15 Nov 2019 18:04:10 +0000 Received: from DB5EUR03FT058.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e0a::205) by VI1PR0802CA0045.outlook.office365.com (2603:10a6:800:a9::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23 via Frontend Transport; Fri, 15 Nov 2019 18:04:09 +0000 Authentication-Results: spf=fail (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; gcc.gnu.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; gcc.gnu.org; dmarc=none action=none header.from=arm.com; Received-SPF: Fail (protection.outlook.com: domain of arm.com does not designate 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT058.mail.protection.outlook.com (10.152.20.255) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.23 via Frontend Transport; Fri, 15 Nov 2019 18:04:09 +0000 Received: ("Tessian outbound e4042aced47b:v33"); Fri, 15 Nov 2019 18:04:09 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: a2aec155b54a9b92 X-CR-MTA-TID: 64aa7808 Received: from d6162ed02202.2 (cr-mta-lb-1.cr-mta-net [104.47.5.58]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id B8FEFCF2-61DF-4EAB-9887-72B3F207F923.1; Fri, 15 Nov 2019 18:04:04 +0000 Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-he1eur02lp2058.outbound.protection.outlook.com [104.47.5.58]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id d6162ed02202.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 15 Nov 2019 18:04:04 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ytnwchbs8DWtGlR56V9GafZI7P8b6WSwyQbfAFaGsyO2CkWj4drJ8dpyFY3SKhmy8t93Ow2Gh1N7lfpJS2U7/mcQNYtfaN+QRGJmIJFkJIByRaUhDSVlaEtvGYHmt22IkbwLcvA8t8TPGS/lIg2Q9lEdankQcFucBA85v3TxBnF2SngzI/p6LK1n4czLjGOdIHDiQ9ki/mzeIxO9aeEdQtcRskVa1ORpPsUy9cU02wfgxgCbAte7YvH8Mitapja4Rj0Zp60KVSA3omsr8S6NnleTNLPxBh3QpKjVUS3p+akgKv7kh0zUzIgN3RWPPN7FBZD9b4Srt7xNO/Q6YuybPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P37L+jAtZS9aN4/0NBSi5TlKI59gZrU7iqVsFBQUBCI=; b=VCA0oPo2rF1FC03xXCWinAeOhyN/ae1QEz8Nkc+f8Jv+m8m0/h6sGPMmMMbPCBQR0li1rK5277Y3sERrMYEXk7lYjeaXJI7PUXc2ygEGFJyoLtxFlqiPF861llWXu3LtVPchExjTGqe4HhtHbcy791kghVhJivVcU3aMXDLCyYKdulZcnvA4n1YBrNeJ4VTjLULpHSdBOT3RP3VgGdOJhAqRPsKYqL2BGJNZh9iDwBOSEAXZvKa06zASfzAh0IZNGqNCJtTGsG4hUIFjqwfZ35Sb55/7y/uPHqATPn2dvQbetI+gFnpVpxQ73gjn5y7E25eiL2nyzMAj8qFtcYWx8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P37L+jAtZS9aN4/0NBSi5TlKI59gZrU7iqVsFBQUBCI=; b=FmsVHspctcEchDevKg/n08bKu/mCtp101dJdo3/OkIuIHExnQb3NF9eaME8Bbc70FT0CLl6z78d19Ud0Zsc+zKHsaQKJd8wjkuuOsiJa7/gjcmnx38EoJeyqU2ClMWFkbUchei7PnWEHQUa7P+1ODBTaSMVntq6ag+C4+pp7xmU= Received: from DB6PR0801MB2054.eurprd08.prod.outlook.com (10.168.86.135) by DB6PR0801MB1797.eurprd08.prod.outlook.com (10.169.222.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2451.28; Fri, 15 Nov 2019 18:04:02 +0000 Received: from DB6PR0801MB2054.eurprd08.prod.outlook.com ([fe80::3163:8919:3dc9:d6ff]) by DB6PR0801MB2054.eurprd08.prod.outlook.com ([fe80::3163:8919:3dc9:d6ff%7]) with mapi id 15.20.2451.029; Fri, 15 Nov 2019 18:04:02 +0000 From: Joel Hutton To: GCC Patches CC: Tamar Christina , nd Subject: [RFC][GCC][AArch64] Add minmax phi-reduction pattern Date: Fri, 15 Nov 2019 18:04:02 +0000 Message-ID: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Joel.Hutton@arm.com; x-ms-exchange-transport-forked: True x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(346002)(376002)(136003)(366004)(396003)(39860400002)(53754006)(189003)(199004)(54534003)(71200400001)(71190400001)(6116002)(4001150100001)(66616009)(486006)(476003)(2616005)(64756008)(66446008)(66556008)(4326008)(65806001)(65956001)(66066001)(6486002)(6512007)(6436002)(31686004)(66946007)(5660300002)(66476007)(316002)(2906002)(58126008)(81166006)(36756003)(52116002)(478600001)(54906003)(25786009)(102836004)(256004)(26005)(6916009)(31696002)(81156014)(8676002)(186003)(99286004)(386003)(305945005)(8936002)(86362001)(3846002)(7736002)(6506007)(14454004); DIR:OUT; SFP:1101; SCL:1; SRVR:DB6PR0801MB1797; H:DB6PR0801MB2054.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: DD1lxn7jF4qU/EJK5bkWHUeVx4YcTH031R2fF+oCxKKK1wWNCFrEADTrO/PDqjZ1cwyJOBw2WcKAUOr5DQ9PblWNkqLIPvQUBaFeI0A/Tf/xcRNahe22dohNHLnCqDlDcNkBxYF2q7XJNkua9MGjmbRtpTRgvZbrA7Sw9ArtNwRTmzKD5vv4c+tO81tSgcke03C2mzWvmmMcr9IpP1MdPTvDO2jYtQJmn6dt5C4QEy0Ix9/boWtfUC/v710J/BlwZeFMTnBWz1UaC6NqDnwOFaV6xKjOve0NkrAtQRuyhN1BTCAQZVu8J7t0kjGidyS/oVQLBc23Zl5Bja1JC27BcGYDyoflD5F1AZHYDK6diMJXhqFezB7wbj4DiOAmZIV/Xlkz3dUdnqrGy0mvBUQVMI/yxhmiwuPm1OY71r8DwAyCTeFb5BGwxCh+IMp+mH9X MIME-Version: 1.0 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Joel.Hutton@arm.com; X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT058.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 4c797a19-7af0-4598-8d1a-08d769f6321d X-IsSubscribed: yes Hi all, Just looking for some feedback on the approach. Currently the loop vectorizer can't vectorize the following typical loop for getting max value and index from an array: void test_vec(int *data, int n) {         int best_i, best = 0;         for (int i = 0; i < n; i++) {                 if (data[i] > best) {                         best = data[i];                         best_i = i;                 }         }         data[best_i] = data[0];         data[0] = best; } This patch adds some support for this pattern. This patch addresses Bug 88259. Regression testing is still in progress, This patch does not work correctly with vect-epilogues-nomask, and the reason for that is still being investigated. gcc/ChangeLog: 2019-11-15  Joel Hutton          Tamar Christina      PR tree-optimization/88259     * tree-vect-loop.c (vect_reassociating_reduction_simple_p): New function.     (vect_recog_minmax_index_pattern): New function.     (vect_is_simple_reduction): Add check for minmax pattern.     (vect_model_reduction_cost): Add case for minmax pattern.     (vect_create_epilog_for_reduction): Add fixup for minmax epilog.     * tree-vectorizer.h (enum vect_reduction_type): Add INDEX_MINMAX_REDUCTION reduction type. diff --git a/gcc/tree-vect-loop.c b/gcc/tree-vect-loop.c index b600d3157457c3180d0456c4f66cbc57012e3c71..dc97dea38a504e8f9391e6d138aad0a2e3872b50 100644 --- a/gcc/tree-vect-loop.c +++ b/gcc/tree-vect-loop.c @@ -387,6 +387,83 @@ vect_determine_vectorization_factor (loop_vec_info loop_vinfo) return opt_result::success (); } +static bool +vect_reassociating_reduction_simple_p (stmt_vec_info stmt_info, tree_code code, + tree *op0_out, tree *op1_out) +{ + loop_vec_info loop_info = STMT_VINFO_LOOP_VINFO (stmt_info); + if (!loop_info) + return false; + + gassign *assign = dyn_cast (stmt_info->stmt); + if (!assign || gimple_assign_rhs_code (assign) != code) + return false; + + /* We don't allow changing the order of the computation in the inner-loop + when doing outer-loop vectorization. */ + class loop *loop = LOOP_VINFO_LOOP (loop_info); + if (loop && nested_in_vect_loop_p (loop, stmt_info)) + return false; + + *op0_out = gimple_assign_rhs1 (assign); + *op1_out = gimple_assign_rhs2 (assign); + return true; +} + + +static bool +vect_recog_minmax_index_pattern (stmt_vec_info stmt_vinfo, loop_vec_info loop_info) +{ + tree oprnd0, oprnd1; + gimple *last_stmt = stmt_vinfo->stmt; + vec_info *vinfo = stmt_vinfo->vinfo; + gimple *use_stmt; + use_operand_p use_p; + imm_use_iterator iter; + + /* Starting from LAST_STMT, follow the defs of its uses in search + of the above pattern. */ + + if (!vect_reassociating_reduction_simple_p (stmt_vinfo, MAX_EXPR, + &oprnd0, &oprnd1)) + return NULL; + + if (!is_a (SSA_NAME_DEF_STMT (oprnd1))) + return NULL; + + stmt_vec_info phy_vinfo = vinfo->lookup_def (oprnd1); + if (!phy_vinfo) + return NULL; + + basic_block top_bb = gimple_bb (last_stmt); + + FOR_EACH_IMM_USE_FAST (use_p, iter, oprnd1) + { + use_stmt = USE_STMT (use_p); + if (is_gimple_assign (use_stmt) + && gimple_assign_rhs_code (use_stmt) == COND_EXPR) + { + basic_block bb = gimple_bb (use_stmt); + + if (bb == top_bb + && gimple_uid (use_stmt) < gimple_uid (last_stmt)) + { + tree cond = gimple_assign_rhs1 (use_stmt); + if (TREE_CODE (cond) != LE_EXPR) + continue; + + stmt_vec_info ind_stmt = loop_info->lookup_stmt (use_stmt); + ind_stmt->reduc_related_stmt = stmt_vinfo; + stmt_vinfo->reduc_related_stmt = ind_stmt; + return true; + } + } + } + + return false; +} + + /* Function vect_is_simple_iv_evolution. @@ -2771,20 +2848,6 @@ pop: fail = true; break; } - /* Check there's only a single stmt the op is used on inside - of the loop. */ - imm_use_iterator imm_iter; - gimple *op_use_stmt; - unsigned cnt = 0; - FOR_EACH_IMM_USE_STMT (op_use_stmt, imm_iter, op) - if (!is_gimple_debug (op_use_stmt) - && flow_bb_inside_loop_p (loop, gimple_bb (op_use_stmt))) - cnt++; - if (cnt != 1) - { - fail = true; - break; - } tree_code use_code = gimple_assign_rhs_code (use_stmt); if (use_code == MINUS_EXPR) { @@ -2963,10 +3026,22 @@ vect_is_simple_reduction (loop_vec_info loop_info, stmt_vec_info phi_info, return def_stmt_info; } + /* Detect if this is a reduction that we have special code to handle the use + of the reduction multiple times. */ + bool supported_multi_use_reduction + = vect_recog_minmax_index_pattern (def_stmt_info, loop_info); + + /* Update the reduction type for multi use reduction. */ + if (supported_multi_use_reduction) + STMT_VINFO_REDUC_TYPE (phi_info) = INDEX_MINMAX_REDUCTION; + + def_stmt_info->reduc_phi_node = phi_info; + /* If this isn't a nested cycle or if the nested cycle reduction value is used ouside of the inner loop we cannot handle uses of the reduction value. */ - if (nlatch_def_loop_uses > 1 || nphi_def_loop_uses > 1) + if (!supported_multi_use_reduction + && (nlatch_def_loop_uses > 1 || nphi_def_loop_uses > 1)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -3737,7 +3812,8 @@ vect_model_reduction_cost (stmt_vec_info stmt_info, internal_fn reduc_fn, { if (reduc_fn != IFN_LAST) { - if (reduction_type == COND_REDUCTION) + if (reduction_type == COND_REDUCTION + || reduction_type == INDEX_MINMAX_REDUCTION) { /* An EQ stmt and an COND_EXPR stmt. */ epilogue_cost += record_stmt_cost (cost_vec, 2, @@ -4764,6 +4840,14 @@ vect_create_epilog_for_reduction (stmt_vec_info stmt_info, new_temp = make_ssa_name (new_scalar_dest, epilog_stmt); gimple_set_lhs (epilog_stmt, new_temp); + + if (stmt_info->reduc_related_stmt + && stmt_info->reduc_related_stmt->reduc_phi_node + && stmt_info->reduc_related_stmt->reduc_phi_node->reduc_type == INDEX_MINMAX_REDUCTION) + { + stmt_info->reduc_minmax_epilog_stmt = epilog_stmt; + } + gsi_insert_before (&exit_gsi, epilog_stmt, GSI_SAME_STMT); if ((STMT_VINFO_REDUC_TYPE (reduc_info) == INTEGER_INDUC_COND_REDUCTION) @@ -5008,6 +5092,7 @@ vect_create_epilog_for_reduction (stmt_vec_info stmt_info, gimple_seq stmts = NULL; new_temp = gimple_convert (&stmts, vectype1, new_temp); + tree maxresultvec = new_temp; for (elt_offset = nelements / 2; elt_offset >= 1; elt_offset /= 2) @@ -5034,7 +5119,54 @@ vect_create_epilog_for_reduction (stmt_vec_info stmt_info, epilog_stmt = gimple_build_assign (new_scalar_dest, rhs); new_temp = make_ssa_name (new_scalar_dest, epilog_stmt); gimple_assign_set_lhs (epilog_stmt, new_temp); + + stmt_info->reduc_minmax_scalar_result = epilog_stmt; gsi_insert_before (&exit_gsi, epilog_stmt, GSI_SAME_STMT); + + /* Fixup the minmax epilog. */ + if (stmt_info->reduc_related_stmt + && stmt_info->reduc_related_stmt->reduc_minmax_epilog_stmt) + { + gimple_stmt_iterator gsi = gsi_for_stmt (stmt_info->reduc_related_stmt->reduc_minmax_epilog_stmt); + gimple *pre_stmt = NULL; + gimple *mm_i_ep_stmt = stmt_info->reduc_related_stmt->reduc_minmax_epilog_stmt; + tree mm_i_lhs = gimple_get_lhs (mm_i_ep_stmt); + tree mm_i_rhs = gimple_call_arg (mm_i_ep_stmt, 0); + tree vectype = TREE_TYPE (mm_i_rhs); + /* Scalar result of min/max in minmax reduction. */ + tree maxval = gimple_get_lhs (stmt_info->reduc_minmax_scalar_result); + tree maxvec = build_vector_from_val (vectype, maxval); + gimple* maxvec_decl = gimple_build_assign (make_ssa_name (vectype), + maxvec); + gsi_insert_before (&gsi, maxvec_decl, GSI_SAME_STMT); + tree maxvec_lhs = gimple_get_lhs (maxvec_decl); + enum tree_code cond_code = EQ_EXPR; + tree vec_comp_type = build_same_sized_truth_vector_type (vectype); + /* Produce a mask that gives the position of max element in max + vector. */ + tree mask = build2 (cond_code, vec_comp_type, maxvec_lhs, + maxresultvec); + tree _mask_ssa_name = make_ssa_name (vec_comp_type); + gimple* _mask_decl = gimple_build_assign (_mask_ssa_name, + mask); + gsi_insert_before (&gsi, _mask_decl, GSI_SAME_STMT); + tree mask_ssa_name = make_ssa_name (vectype); + gimple* mask_decl = gimple_build_assign (mask_ssa_name, CONVERT_EXPR, _mask_ssa_name); + gsi_insert_before (&gsi, mask_decl, GSI_SAME_STMT); + /* Apply the mask to the index vector to select index value in + corresponding position. */ + tree new_ssa = make_ssa_name (vectype); + pre_stmt = gimple_build_assign (new_ssa, BIT_AND_EXPR, mask_ssa_name, mm_i_rhs); + gsi_insert_before (&gsi, pre_stmt, GSI_SAME_STMT); + /* Reduce the index vector to a scalar result. */ + gimple* new_stmt = gimple_build_call_internal (IFN_REDUC_MAX, + 1, + gimple_get_lhs (pre_stmt)); + gimple_set_lhs (new_stmt, mm_i_lhs); + gsi = gsi_for_stmt (mm_i_ep_stmt); + gsi_replace (&gsi, new_stmt, 1); + } + scalar_results.safe_push (new_temp); } else diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index e9575a184ad02787cbdc6ea9059ef1dc35fbca94..8ff6266ecf08213a0f19191c4091bf10b4d6ae37 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -85,7 +85,17 @@ enum vect_reduction_type { res = res OP val[i]; (with no reassocation). */ - FOLD_LEFT_REDUCTION + FOLD_LEFT_REDUCTION, + + /* Reduction of an if selecting both the maximum/minimum element and the + corresponding index. + + for (int i = 0; i < n; i++) { + if (data[i] > best) { + best = data[i]; + best_i = i; + } */ + INDEX_MINMAX_REDUCTION }; #define VECTORIZABLE_CYCLE_DEF(D) (((D) == vect_reduction_def) \ @@ -967,6 +977,15 @@ public: pattern). */ stmt_vec_info related_stmt; + + stmt_vec_info reduc_related_stmt; + + gimple* reduc_minmax_epilog_stmt; + + gimple* reduc_minmax_scalar_result; + + stmt_vec_info reduc_phi_node; + /* Used to keep a sequence of def stmts of a pattern stmt if such exists. The sequence is attached to the original statement rather than the pattern statement. */