From patchwork Fri Oct 25 11:10:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1184033 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=gcc-patches-return-511745-incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b="b0hOVPZu"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4701c73VVcz9sPK for ; Fri, 25 Oct 2019 22:10:13 +1100 (AEDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:date :from:to:subject:message-id:mime-version:content-type; q=dns; s= default; b=AoI7vxW/4Z8cYg8a/NNoPuKwvxOtOXWqyRWFO1PNoBc7P2aTeD/F4 zX0vKXy71piPJ3mmW4Bf7Z++5Xdof/ERAqrvzEvLtNeQLIiG1+e3O12h3cOoRtBA uMzyErfL8qBJu061z00sOedkk2DKHkjRon6FsenZSNwz40i0yW3+9Y= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:date :from:to:subject:message-id:mime-version:content-type; s= default; bh=GKfybZXuRDzf8DEdix7mRfrb9Gk=; b=b0hOVPZuVHnWs6Wwm/8F nQD0i/2AqEv8N4W3iQskvH/4g6ganp//GotyJ2mZTrVvY6aQoPYNw94CJeNFMkXV GdkgcUJFuH4ERq3z0nWouhD/aodYZH98rEofxyXBfwaKWgbmmiAxTWOPL8foJuMk MqtuxlrKamhOkrQkaIaJPrY= Received: (qmail 17867 invoked by alias); 25 Oct 2019 11:10:05 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 17859 invoked by uid 89); 25 Oct 2019 11:10:05 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-13.4 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, SPF_PASS autolearn=ham version=3.3.1 spammy=participating, !vectype_in X-HELO: mx1.suse.de Received: from mx2.suse.de (HELO mx1.suse.de) (195.135.220.15) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 25 Oct 2019 11:10:03 +0000 Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 554E4B34B for ; Fri, 25 Oct 2019 11:10:01 +0000 (UTC) Date: Fri, 25 Oct 2019 13:10:01 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH] Use STMT_VINFO_REDUC_IDX instead of recomputing it Message-ID: User-Agent: Alpine 2.21 (LSU 202 2017-01-01) MIME-Version: 1.0 This is a cleanup. The cond-reduction restriction can go, the fold-left one stays (it cannot handle more than one stmt in the cycle - in the future when we get partial loop vectorization generic code would handle duplicating of scalar code parts, they'd simply stay single-lane SLP graph parts). Bootstrap and regtest running on x86_64-unknown-linux-gnu. Richard. 2019-10-25 Richard Biener * tree-vect-loop.c (vect_create_epilog_for_reduction): Use STMT_VINFO_REDUC_IDX from the actual stmt. (vect_transform_reduction): Likewise. (vectorizable_reduction): Compute the reduction chain length, do not recompute the reduction operand index. Remove no longer necessary restriction for condition reduction chains. Index: gcc/tree-vect-loop.c =================================================================== --- gcc/tree-vect-loop.c (revision 277441) +++ gcc/tree-vect-loop.c (working copy) @@ -4263,9 +4263,9 @@ vect_create_epilog_for_reduction (stmt_v (CCOMPARE). The then and else values mirror the main VEC_COND_EXPR: the reduction phi corresponds to NEW_PHI_TREE and the new values correspond to INDEX_BEFORE_INCR. */ - gcc_assert (STMT_VINFO_REDUC_IDX (reduc_info) >= 1); + gcc_assert (STMT_VINFO_REDUC_IDX (stmt_info) >= 1); tree index_cond_expr; - if (STMT_VINFO_REDUC_IDX (reduc_info) == 2) + if (STMT_VINFO_REDUC_IDX (stmt_info) == 2) index_cond_expr = build3 (VEC_COND_EXPR, cr_index_vector_type, ccompare, indx_before_incr, new_phi_tree); else @@ -5720,19 +5720,21 @@ vectorizable_reduction (stmt_vec_info st gcc_assert (!STMT_VINFO_RELATED_STMT (phi_info)); gphi *reduc_def_phi = as_a (phi_info->stmt); - /* Verify following REDUC_IDX from the latch def leads us back to the PHI. */ + /* Verify following REDUC_IDX from the latch def leads us back to the PHI + and compute the reduction chain length. */ tree reduc_def = PHI_ARG_DEF_FROM_EDGE (reduc_def_phi, loop_latch_edge (loop)); + unsigned reduc_chain_length = 0; while (reduc_def != PHI_RESULT (reduc_def_phi)) { stmt_vec_info def = loop_vinfo->lookup_def (reduc_def); def = vect_stmt_to_vectorize (def); gcc_assert (STMT_VINFO_REDUC_IDX (def) != -1); reduc_def = gimple_op (def->stmt, 1 + STMT_VINFO_REDUC_IDX (def)); + reduc_chain_length++; } reduc_def = PHI_RESULT (reduc_def_phi); - int reduc_index = -1; for (i = 0; i < op_type; i++) { tree op = gimple_op (stmt, i + 1); @@ -5753,7 +5755,6 @@ vectorizable_reduction (stmt_vec_info st if ((dt == vect_reduction_def || dt == vect_nested_cycle) && op == reduc_def) { - reduc_index = i; continue; } @@ -5792,10 +5793,6 @@ vectorizable_reduction (stmt_vec_info st if (!vectype_in) vectype_in = vectype_out; STMT_VINFO_REDUC_VECTYPE_IN (reduc_info) = vectype_in; - /* For the SSA cycle we store on each participating stmt the operand index - where the cycle continues. Store the one relevant for the actual - operation in the reduction meta. */ - STMT_VINFO_REDUC_IDX (reduc_info) = reduc_index; enum vect_reduction_type v_reduc_type = STMT_VINFO_REDUC_TYPE (phi_info); STMT_VINFO_REDUC_TYPE (reduc_info) = v_reduc_type; @@ -5805,28 +5802,8 @@ vectorizable_reduction (stmt_vec_info st if (slp_node) return false; - /* TODO: We can't yet handle reduction chains, since we need to treat - each COND_EXPR in the chain specially, not just the last one. - E.g. for: - - x_1 = PHI - x_2 = a_2 ? ... : x_1; - x_3 = a_3 ? ... : x_2; - - we're interested in the last element in x_3 for which a_2 || a_3 - is true, whereas the current reduction chain handling would - vectorize x_2 as a normal VEC_COND_EXPR and only treat x_3 - as a reduction operation. */ - if (reduc_index == -1) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "conditional reduction chains not supported\n"); - return false; - } - /* When the condition uses the reduction value in the condition, fail. */ - if (reduc_index == 0) + if (STMT_VINFO_REDUC_IDX (stmt_info) == 0) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -5995,17 +5972,17 @@ vectorizable_reduction (stmt_vec_info st outer-loop vectorization is safe. */ if (needs_fold_left_reduction_p (scalar_type, orig_code)) { - STMT_VINFO_REDUC_TYPE (reduc_info) - = reduction_type = FOLD_LEFT_REDUCTION; - /* When vectorizing a reduction chain w/o SLP the reduction PHI is not - directy used in stmt. */ - if (reduc_index == -1) + /* When vectorizing a reduction chain w/o SLP the reduction PHI + is not directy used in stmt. */ + if (reduc_chain_length != 1) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, "in-order reduction chain without SLP.\n"); return false; } + STMT_VINFO_REDUC_TYPE (reduc_info) + = reduction_type = FOLD_LEFT_REDUCTION; } else if (!commutative_tree_code (orig_code) || !associative_tree_code (orig_code)) @@ -6410,7 +6387,7 @@ vect_transform_reduction (stmt_vec_info reduction variable. */ stmt_vec_info phi_info = STMT_VINFO_REDUC_DEF (vect_orig_stmt (stmt_info)); gphi *reduc_def_phi = as_a (phi_info->stmt); - int reduc_index = STMT_VINFO_REDUC_IDX (reduc_info); + int reduc_index = STMT_VINFO_REDUC_IDX (stmt_info); tree vectype_in = STMT_VINFO_REDUC_VECTYPE_IN (reduc_info); if (slp_node)