From patchwork Wed Nov 8 15:03:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1861640 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=CdgBCf/c; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=D9fF2zZ4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SQSyg6Fhjz1yRF for ; Thu, 9 Nov 2023 02:03:39 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 888CC385C6FF for ; Wed, 8 Nov 2023 15:03:32 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by sourceware.org (Postfix) with ESMTPS id 686E4385773C for ; Wed, 8 Nov 2023 15:03:10 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 686E4385773C Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 686E4385773C Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.220.28 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699455800; cv=none; b=PUizkzh6NS7ufD2pK1yTaBLXWtoq/E5Qm5gpYQa7vcaLm34noXMsIvDTmAVxtvD9JNuj3lqvAvqRevEnrYvgxzIFQ4F3MiA3eI/vxGA0hup9AZFY6k9LG1MnrOZFNXPNqmpsFiZIZLukh7Cj6mADGjlTqA+8fqMLdmc38XODxWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699455800; c=relaxed/simple; bh=XePhruUaUtS+BGQ6zeruf/8DH8WIt07jRaxp5GO3fIY=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:MIME-Version: Message-Id; b=g6n8vR7uU4i8ETQPPh098wS2Uq0ri2rqYgP7sb4uaYdmPgCEYQMjcfiMtumONaXJ74xaUnDTqPbQaRtP2GMqk6AZEPAPEN5UsircsOF/2wifh9KqUH8p4d3s5hC31lJEBJ/k2B6IJ4/1RJTnpeSj7BdVLq26Vcuxe46WEcWT/xY= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 9832621961 for ; Wed, 8 Nov 2023 15:03:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1699455789; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=T7gyEh3R6K2Z+tVnY8joBV9TgIGqBFjaiclXRgdAXYQ=; b=CdgBCf/cEsxSOmWbVTYvORSeiYYgZmwyYzc2AF5HaXVlibmtPGSTahUQhK335C7a5EaecT +1MCyVXWmTcxpui1Jvbx1VggKi5266jHu+ls1iE4PgFjWWiPyddR9Gwq3I9a/I3ZXG5GVl LxDGM+pfPt3Ppjz5YzFGuW5nI0bh5KY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1699455789; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=T7gyEh3R6K2Z+tVnY8joBV9TgIGqBFjaiclXRgdAXYQ=; b=D9fF2zZ4t/u0xdjKW3MNbap+FAAhMDMGvOFzEJFJvRpeCvk3denO5oLa1LfGrvSOvg6FZ4 /CnFSyTo23BfqYCQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 85118133F5 for ; Wed, 8 Nov 2023 15:03:09 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id GrFPHy2jS2X8dwAAMHmgww (envelope-from ) for ; Wed, 08 Nov 2023 15:03:09 +0000 Date: Wed, 8 Nov 2023 16:03:09 +0100 (CET) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH 2/4] TLC to vect_check_store_rhs and vect_slp_child_index_for_operand MIME-Version: 1.0 Message-Id: <20231108150309.85118133F5@imap2.suse-dmz.suse.de> X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org This prepares us for the SLP of scatters. We have to tell vect_slp_child_index_for_operand whether we are dealing with a scatter/gather stmt so this adds an argument similar to the one we have for vect_get_operand_map. This also refactors vect_check_store_rhs to get the actual rhs and the associated SLP node instead of leaving that to the caller. Bootstrap and regtest running on x86_64-unknown-linux-gnu. * tree-vectorizer.h (vect_slp_child_index_for_operand): Add gatherscatter_p argument. * tree-vect-slp.cc (vect_slp_child_index_for_operand): Likewise. Pass it on. * tree-vect-stmts.cc (vect_check_store_rhs): Turn the rhs argument into an output, also output the SLP node associated with it. (vectorizable_simd_clone_call): Adjust. (vectorizable_store): Likewise. (vectorizable_load): Likewise. --- gcc/tree-vect-slp.cc | 5 ++-- gcc/tree-vect-stmts.cc | 52 ++++++++++++++++++++++-------------------- gcc/tree-vectorizer.h | 2 +- 3 files changed, 31 insertions(+), 28 deletions(-) diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 13137ede8d4..176aaf270f4 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -589,9 +589,10 @@ vect_get_operand_map (const gimple *stmt, bool gather_scatter_p = false, /* Return the SLP node child index for operand OP of STMT. */ int -vect_slp_child_index_for_operand (const gimple *stmt, int op) +vect_slp_child_index_for_operand (const gimple *stmt, int op, + bool gather_scatter_p) { - const int *opmap = vect_get_operand_map (stmt); + const int *opmap = vect_get_operand_map (stmt, gather_scatter_p); if (!opmap) return op; for (int i = 1; i < 1 + opmap[0]; ++i) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 096a857f2dd..61e23b29516 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -2486,42 +2486,33 @@ vect_check_scalar_mask (vec_info *vinfo, stmt_vec_info stmt_info, return true; } -/* Return true if stored value RHS is suitable for vectorizing store - statement STMT_INFO. When returning true, store the type of the - definition in *RHS_DT_OUT, the type of the vectorized store value in +/* Return true if stored value is suitable for vectorizing store + statement STMT_INFO. When returning true, store the scalar stored + in *RHS and *RHS_NODE, the type of the definition in *RHS_DT_OUT, + the type of the vectorized store value in *RHS_VECTYPE_OUT and the type of the store in *VLS_TYPE_OUT. */ static bool vect_check_store_rhs (vec_info *vinfo, stmt_vec_info stmt_info, - slp_tree slp_node, tree rhs, + slp_tree slp_node, tree *rhs, slp_tree *rhs_node, vect_def_type *rhs_dt_out, tree *rhs_vectype_out, vec_load_store_type *vls_type_out) { - /* In the case this is a store from a constant make sure - native_encode_expr can handle it. */ - if (CONSTANT_CLASS_P (rhs) && native_encode_expr (rhs, NULL, 64) == 0) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "cannot encode constant as a byte sequence.\n"); - return false; - } - int op_no = 0; if (gcall *call = dyn_cast (stmt_info->stmt)) { if (gimple_call_internal_p (call) && internal_store_fn_p (gimple_call_internal_fn (call))) op_no = internal_fn_stored_value_index (gimple_call_internal_fn (call)); - if (slp_node) - op_no = vect_slp_child_index_for_operand (call, op_no); } + if (slp_node) + op_no = vect_slp_child_index_for_operand + (stmt_info->stmt, op_no, STMT_VINFO_GATHER_SCATTER_P (stmt_info)); enum vect_def_type rhs_dt; tree rhs_vectype; - slp_tree slp_op; if (!vect_is_simple_use (vinfo, stmt_info, slp_node, op_no, - &rhs, &slp_op, &rhs_dt, &rhs_vectype)) + rhs, rhs_node, &rhs_dt, &rhs_vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -2529,6 +2520,16 @@ vect_check_store_rhs (vec_info *vinfo, stmt_vec_info stmt_info, return false; } + /* In the case this is a store from a constant make sure + native_encode_expr can handle it. */ + if (CONSTANT_CLASS_P (*rhs) && native_encode_expr (*rhs, NULL, 64) == 0) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "cannot encode constant as a byte sequence.\n"); + return false; + } + tree vectype = STMT_VINFO_VECTYPE (stmt_info); if (rhs_vectype && !useless_type_conversion_p (vectype, rhs_vectype)) { @@ -4052,7 +4053,7 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info, int op_no = i + masked_call_offset; if (slp_node) - op_no = vect_slp_child_index_for_operand (stmt, op_no); + op_no = vect_slp_child_index_for_operand (stmt, op_no, false); if (!vect_is_simple_use (vinfo, stmt_info, slp_node, op_no, &op, &slp_op[i], &thisarginfo.dt, &thisarginfo.vectype) @@ -8173,7 +8174,6 @@ vectorizable_store (vec_info *vinfo, stmt_vector_for_cost *cost_vec) { tree data_ref; - tree op; tree vec_oprnd = NULL_TREE; tree elem_type; loop_vec_info loop_vinfo = dyn_cast (vinfo); @@ -8236,15 +8236,14 @@ vectorizable_store (vec_info *vinfo, int mask_index = internal_fn_mask_index (ifn); if (mask_index >= 0 && slp_node) - mask_index = vect_slp_child_index_for_operand (call, mask_index); + mask_index = vect_slp_child_index_for_operand + (call, mask_index, STMT_VINFO_GATHER_SCATTER_P (stmt_info)); if (mask_index >= 0 && !vect_check_scalar_mask (vinfo, stmt_info, slp_node, mask_index, &mask, NULL, &mask_dt, &mask_vectype)) return false; } - op = vect_get_store_rhs (stmt_info); - /* Cannot have hybrid store SLP -- that would mean storing to the same location twice. */ gcc_assert (slp == PURE_SLP_STMT (stmt_info)); @@ -8279,8 +8278,10 @@ vectorizable_store (vec_info *vinfo, return false; } + tree op; + slp_tree op_node; if (!vect_check_store_rhs (vinfo, stmt_info, slp_node, - op, &rhs_dt, &rhs_vectype, &vls_type)) + &op, &op_node, &rhs_dt, &rhs_vectype, &vls_type)) return false; elem_type = TREE_TYPE (vectype); @@ -9855,7 +9856,8 @@ vectorizable_load (vec_info *vinfo, mask_index = internal_fn_mask_index (ifn); if (mask_index >= 0 && slp_node) - mask_index = vect_slp_child_index_for_operand (call, mask_index); + mask_index = vect_slp_child_index_for_operand + (call, mask_index, STMT_VINFO_GATHER_SCATTER_P (stmt_info)); if (mask_index >= 0 && !vect_check_scalar_mask (vinfo, stmt_info, slp_node, mask_index, &mask, &slp_op, &mask_dt, &mask_vectype)) diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index d2ddc2e4ad5..e4d7ab4567c 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -2462,7 +2462,7 @@ extern int vect_get_place_in_interleaving_chain (stmt_vec_info, stmt_vec_info); extern slp_tree vect_create_new_slp_node (unsigned, tree_code); extern void vect_free_slp_tree (slp_tree); extern bool compatible_calls_p (gcall *, gcall *); -extern int vect_slp_child_index_for_operand (const gimple *, int op); +extern int vect_slp_child_index_for_operand (const gimple *, int op, bool); /* In tree-vect-patterns.cc. */ extern void