From patchwork Mon Jan 11 11:50:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1424569 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=gcc-patches-bounces@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=suse.de Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DDsVJ0nWXz9sVy for ; Mon, 11 Jan 2021 22:51:01 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 828063843856; Mon, 11 Jan 2021 11:50:57 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by sourceware.org (Postfix) with ESMTPS id 4F8E338618B8 for ; Mon, 11 Jan 2021 11:50:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 4F8E338618B8 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rguenther@suse.de X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 0F1C4AB3E for ; Mon, 11 Jan 2021 11:50:54 +0000 (UTC) Date: Mon, 11 Jan 2021 12:50:53 +0100 (CET) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH] tree-optimization/98526 - fix vectorizer reduction cost Message-ID: User-Agent: Alpine 2.21 (LSU 202 2017-01-01) MIME-Version: 1.0 X-Spam-Status: No, score=-11.2 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces@gcc.gnu.org Sender: "Gcc-patches" This fixes a double-counting in the reduction cost when vectorizing the reduction through the regular vectorizable_* functions. Bootstrapped / tested on x86_64-unknown-linux-gnu, pushed. 2021-01-11 Richard Biener PR tree-optimization/98526 * tree-vect-loop.c (vect_model_reduction_cost): Remove costing of the actual reduction op for the regular case. (vectorizable_reduction): Cost the stmts vect_transform_reduction produces here. --- gcc/tree-vect-loop.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/gcc/tree-vect-loop.c b/gcc/tree-vect-loop.c index 965cc164f6e..acfd1952e3b 100644 --- a/gcc/tree-vect-loop.c +++ b/gcc/tree-vect-loop.c @@ -4452,8 +4452,8 @@ have_whole_vector_shift (machine_mode mode) /* Function vect_model_reduction_cost. Models cost for a reduction operation, including the vector ops - generated within the strip-mine loop, the initial definition before - the loop, and the epilogue code that must be generated. */ + generated within the strip-mine loop in some cases, the initial + definition before the loop, and the epilogue code that must be generated. */ static void vect_model_reduction_cost (loop_vec_info loop_vinfo, @@ -4516,10 +4516,6 @@ vect_model_reduction_cost (loop_vec_info loop_vinfo, prologue_cost += record_stmt_cost (cost_vec, prologue_stmts, scalar_to_vec, stmt_info, 0, vect_prologue); - - /* Cost of reduction op inside loop. */ - inside_cost = record_stmt_cost (cost_vec, ncopies, vector_stmt, - stmt_info, 0, vect_body); } /* Determine cost of epilogue code. @@ -7268,6 +7264,15 @@ vectorizable_reduction (loop_vec_info loop_vinfo, vect_model_reduction_cost (loop_vinfo, stmt_info, reduc_fn, reduction_type, ncopies, cost_vec); + /* Cost the reduction op inside the loop if transformed via + vect_transform_reduction. Otherwise this is costed by the + separate vectorizable_* routines. */ + if (single_defuse_cycle + || code == DOT_PROD_EXPR + || code == WIDEN_SUM_EXPR + || code == SAD_EXPR) + record_stmt_cost (cost_vec, ncopies, vector_stmt, stmt_info, 0, vect_body); + if (dump_enabled_p () && reduction_type == FOLD_LEFT_REDUCTION) dump_printf_loc (MSG_NOTE, vect_location,