From patchwork Fri Jun  7 11:58:56 2024
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Richard Biener <rguenther@suse.de>
X-Patchwork-Id: 1945077
Return-Path: <gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org>
X-Original-To: incoming@patchwork.ozlabs.org
Delivered-To: patchwork-incoming@legolas.ozlabs.org
Authentication-Results: legolas.ozlabs.org;
	dkim=pass (1024-bit key;
 unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256
 header.s=susede2_rsa header.b=JVH3y9ef;
	dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256
 header.s=susede2_ed25519 header.b=9U463RCY;
	dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de
 header.a=rsa-sha256 header.s=susede2_rsa header.b=JVH3y9ef;
	dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256
 header.s=susede2_ed25519 header.b=9U463RCY;
	dkim-atps=neutral
Authentication-Results: legolas.ozlabs.org;
 spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org
 (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org;
 envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org;
 receiver=patchwork.ozlabs.org)
Received: from server2.sourceware.org (server2.sourceware.org
 [IPv6:2620:52:3:1:0:246e:9693:128c])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384)
	(No client certificate requested)
	by legolas.ozlabs.org (Postfix) with ESMTPS id 4VwfrM5xMfz20Q5
	for <incoming@patchwork.ozlabs.org>; Fri,  7 Jun 2024 21:59:30 +1000 (AEST)
Received: from server2.sourceware.org (localhost [IPv6:::1])
	by sourceware.org (Postfix) with ESMTP id 6B6C63A711EA
	for <incoming@patchwork.ozlabs.org>; Fri,  7 Jun 2024 11:59:28 +0000 (GMT)
X-Original-To: gcc-patches@gcc.gnu.org
Delivered-To: gcc-patches@gcc.gnu.org
Received: from smtp-out2.suse.de (smtp-out2.suse.de
 [IPv6:2a07:de40:b251:101:10:150:64:2])
 by sourceware.org (Postfix) with ESMTPS id 72E0F3A69D1C
 for <gcc-patches@gcc.gnu.org>; Fri,  7 Jun 2024 11:58:58 +0000 (GMT)
DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 72E0F3A69D1C
Authentication-Results: sourceware.org;
 dmarc=pass (p=none dis=none) header.from=suse.de
Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de
ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 72E0F3A69D1C
Authentication-Results: server2.sourceware.org;
 arc=none smtp.remote-ip=2a07:de40:b251:101:10:150:64:2
ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1717761540; cv=none;
 b=KpPqfQkuWWjsq+oFc3sHIFuOHRsCBJCVuE5KFVePdAWAbeNVAcNjGx0VeZ/EFABeF8qn4+J6pNON38z2OjYVOBP8QMwUTEWpZcpNBW/xTesC9/vPB7sMzooGgPpvTiYc1gO39o5RtKZiSVJs1OTijCDm/1rw6pg/G3tALXTtCmk=
ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key;
 t=1717761540; c=relaxed/simple;
 bh=Lq1dz6xu0M2UkFhKKVtJDgWBes4Dc1auUtnEbOIKIwY=;
 h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date:
 From:To:Subject:MIME-Version:Message-Id;
 b=hWpx15UMdb2kdP8EPru3n33N2QzUJeEmy6SYbriDgsyLuhiUjo3aDF/zqL3Lz58jEASBfHY6IyCXotE9rQKcz7tjt+DQVjAkwwB5qweeq+7K334Xj/WDccocbwzrChzOsJoLCykRxeNa1mifp4TYZ3X+eT8INAVkoUPewP0gUIo=
ARC-Authentication-Results: i=1; server2.sourceware.org
Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest
 SHA256)
 (No client certificate requested)
 by smtp-out2.suse.de (Postfix) with ESMTPS id 6947E1FB92
 for <gcc-patches@gcc.gnu.org>; Fri,  7 Jun 2024 11:58:57 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de;
 s=susede2_rsa;
 t=1717761537; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:
 mime-version:mime-version:content-type:content-type;
 bh=T1CcFgBtfSMTVLKTesbz14pEMNMMM3G1F+7akqu+jnw=;
 b=JVH3y9efK+MWRU5nn02+7RAiWxT18AKCpf5ULdrGRgL8yXnWEAT4pbCU3GWnoVxFA+Cegn
 SkItwg0AR1x4dNCvXwpLV4slVVa3j5Sie3Iv530CnrQkSz9QJFGUgkC5DYih1I1z7K2fmA
 AkwRkNc6ZcXUs6kS+Mj8rLvgTuq1JN8=
DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de;
 s=susede2_ed25519; t=1717761537;
 h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:
 mime-version:mime-version:content-type:content-type;
 bh=T1CcFgBtfSMTVLKTesbz14pEMNMMM3G1F+7akqu+jnw=;
 b=9U463RCYQGE9DxnMH3PXg73CwIyIVYeFQf6sUyAd0dEntyMMm2Dxy5aBxXhV5wG/EkUsft
 xaiSVOQmXX2dA1Dg==
Authentication-Results: smtp-out2.suse.de;
	none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de;
 s=susede2_rsa;
 t=1717761537; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:
 mime-version:mime-version:content-type:content-type;
 bh=T1CcFgBtfSMTVLKTesbz14pEMNMMM3G1F+7akqu+jnw=;
 b=JVH3y9efK+MWRU5nn02+7RAiWxT18AKCpf5ULdrGRgL8yXnWEAT4pbCU3GWnoVxFA+Cegn
 SkItwg0AR1x4dNCvXwpLV4slVVa3j5Sie3Iv530CnrQkSz9QJFGUgkC5DYih1I1z7K2fmA
 AkwRkNc6ZcXUs6kS+Mj8rLvgTuq1JN8=
DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de;
 s=susede2_ed25519; t=1717761537;
 h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:
 mime-version:mime-version:content-type:content-type;
 bh=T1CcFgBtfSMTVLKTesbz14pEMNMMM3G1F+7akqu+jnw=;
 b=9U463RCYQGE9DxnMH3PXg73CwIyIVYeFQf6sUyAd0dEntyMMm2Dxy5aBxXhV5wG/EkUsft
 xaiSVOQmXX2dA1Dg==
Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest
 SHA256)
 (No client certificate requested)
 by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 4B39B133F3
 for <gcc-patches@gcc.gnu.org>; Fri,  7 Jun 2024 11:58:57 +0000 (UTC)
Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167])
 by imap1.dmz-prg2.suse.org with ESMTPSA id JLogEAH2YmacVAAAD6G6ig
 (envelope-from <rguenther@suse.de>)
 for <gcc-patches@gcc.gnu.org>; Fri, 07 Jun 2024 11:58:57 +0000
Date: Fri, 7 Jun 2024 13:58:56 +0200 (CEST)
From: Richard Biener <rguenther@suse.de>
To: gcc-patches@gcc.gnu.org
Subject: [PATCH] tree-optimization/114107 - avoid peeling for gaps in more
 cases
MIME-Version: 1.0
Message-Id: <20240607115857.4B39B133F3@imap1.dmz-prg2.suse.org>
X-Spam-Score: -4.30
X-Spam-Level: 
X-Spamd-Result: default: False [-4.30 / 50.00]; BAYES_HAM(-3.00)[100.00%];
 NEURAL_HAM_LONG(-1.00)[-1.000];
 NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com];
 DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519];
 RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[];
 FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[];
 RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[];
 MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[];
 RCVD_VIA_SMTP_AUTH(0.00)[];
 PREVIOUSLY_DELIVERED(0.00)[gcc-patches@gcc.gnu.org];
 DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo]
X-Spam-Status: No, score=-11.6 required=5.0 tests=BAYES_00, DKIM_SIGNED,
 DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE,
 SPF_PASS,
 TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
 server2.sourceware.org
X-BeenThere: gcc-patches@gcc.gnu.org
X-Mailman-Version: 2.1.30
Precedence: list
List-Id: Gcc-patches mailing list <gcc-patches.gcc.gnu.org>
List-Unsubscribe: <https://gcc.gnu.org/mailman/options/gcc-patches>,
 <mailto:gcc-patches-request@gcc.gnu.org?subject=unsubscribe>
List-Archive: <https://gcc.gnu.org/pipermail/gcc-patches/>
List-Post: <mailto:gcc-patches@gcc.gnu.org>
List-Help: <mailto:gcc-patches-request@gcc.gnu.org?subject=help>
List-Subscribe: <https://gcc.gnu.org/mailman/listinfo/gcc-patches>,
 <mailto:gcc-patches-request@gcc.gnu.org?subject=subscribe>
Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org

The following refactors the code to detect necessary peeling for
gaps, in particular the PR103116 case when there is no gap but
the group size is smaller than the vector size.  The testcase in
PR114107 shows we fail to SLP

  for (int i=0; i<n; i++)
    for (int k=0; k<4; k++)
      data[4*i+k] *= factor[i];

because peeling one scalar iteration isn't enough to cover a gap
of 3 elements of factor[i].  But the code detecting this is placed
after the logic that detects cases we handle properly already as
we'd code generate { factor[i], 0., 0., 0. } for V4DFmode vectorization
already.  In fact the check to detect when peeling a single iteration
isn't enough seems improperly guarded as it should apply to all cases.

I'm not sure we correctly handle VMAT_CONTIGUOUS_REVERSE but I
checked that VMAT_STRIDED_SLP and VMAT_ELEMENTWISE correctly avoid
touching excess elements within the same constraints as
VMAT_CONTIGUOUS but the PR103116 case only covered VMAT_CONTIGUOUS
which is too optimistic I think.

With this change we can use SLP for the above testcase and the
PR103116 testcases no longer require an epilogue on x86-64.  It
might be different on other targets so I made those testcases
runtime FAIL only instead of relying on dump scanning there's
currently no easy way to properly constrain.

Bootstrapped and tested on x86_64-unknown-linux-gnu.

	PR tree-optimization/114107
	PR tree-optimization/110445
	* tree-vect-stmts.cc (get_group_load_store_type): Refactor
	contiguous access case.  Make sure peeling for gap constraints
	are always tested and consistently relax when we know we can
	avoid touching excess elements during code generation.

	* gcc.dg/vect/pr114107.c: New testcase.
	* gcc.dg/vect/pr103116-1.c: Adjust.
	* gcc.dg/vect/pr103116-2.c: Likewise.
---
 gcc/testsuite/gcc.dg/vect/pr103116-1.c |   4 +-
 gcc/testsuite/gcc.dg/vect/pr103116-2.c |   3 +-
 gcc/testsuite/gcc.dg/vect/pr114107.c   |  31 +++++++
 gcc/tree-vect-stmts.cc                 | 110 ++++++++++++-------------
 4 files changed, 90 insertions(+), 58 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/vect/pr114107.c

diff --git a/gcc/testsuite/gcc.dg/vect/pr103116-1.c b/gcc/testsuite/gcc.dg/vect/pr103116-1.c
index d3639fc8cfd..280ec57eb4b 100644
--- a/gcc/testsuite/gcc.dg/vect/pr103116-1.c
+++ b/gcc/testsuite/gcc.dg/vect/pr103116-1.c
@@ -47,4 +47,6 @@ main (void)
   return 0;
 }
 
-/* { dg-final { scan-tree-dump "Data access with gaps requires scalar epilogue loop" "vect" { target { vect_perm && vect_int } } } } */
+/* When the target can compose a vector from its half we do not require
+   a scalar epilogue, but there's no effective target for this.  */
+/* { dg-final { scan-tree-dump "vectorized 1 loops" "vect" { target { vect_perm && vect_int } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/pr103116-2.c b/gcc/testsuite/gcc.dg/vect/pr103116-2.c
index aa9797a9407..ee5b82922f9 100644
--- a/gcc/testsuite/gcc.dg/vect/pr103116-2.c
+++ b/gcc/testsuite/gcc.dg/vect/pr103116-2.c
@@ -56,4 +56,5 @@ main (void)
   return 0;
 }
 
-/* { dg-final { scan-tree-dump "peeling for gaps insufficient for access" "vect" { target { vect_perm_short } } } } */
+/* Whether or not peeling for gaps is required depends on the ability of
+   the target to compose a vector from two-element bits.  */
diff --git a/gcc/testsuite/gcc.dg/vect/pr114107.c b/gcc/testsuite/gcc.dg/vect/pr114107.c
new file mode 100644
index 00000000000..65175d9a680
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/pr114107.c
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_double } */
+
+void rescale_x4 (double* __restrict data,
+		 const double* __restrict factor, int n)
+{
+  for (int i=0; i<n; i++)
+    {
+      data[4*i] *= factor[i];
+      data[4*i+1] *= factor[i];
+      data[4*i+2] *= factor[i];
+      data[4*i+3] *= factor[i];
+    }
+}
+
+void rescale_x4_s (double* __restrict data,
+		   const double* __restrict factor, int n, int s)
+{
+  for (int i=0; i<n; i++)
+    {
+      data[s*i] *= factor[s*i];
+      data[s*i+1] *= factor[s*i];
+      data[s*i+2] *= factor[s*i];
+      data[s*i+3] *= factor[s*i];
+    }
+}
+
+/* All targets should be able to compose a vector from scalar elements, but
+   depending on vector size different permutes might be necessary.  */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 2 "vect" { target vect_perm } } } */
+/* { dg-final { scan-tree-dump-not "Data access with gaps requires scalar epilogue loop" "vect" { target vect_perm } } } */
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 5098b7fab6a..69ba906a33f 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -2047,6 +2047,36 @@ get_group_load_store_type (vec_info *vinfo, stmt_vec_info stmt_info,
 	}
       else
 	{
+	  int cmp = compare_step_with_zero (vinfo, stmt_info);
+	  if (cmp < 0)
+	    {
+	      if (single_element_p)
+		/* ???  The VMAT_CONTIGUOUS_REVERSE code generation is
+		   only correct for single element "interleaving" SLP.  */
+		*memory_access_type = get_negative_load_store_type
+			     (vinfo, stmt_info, vectype, vls_type, 1, poffset);
+	      else
+		{
+		  /* Try to use consecutive accesses of DR_GROUP_SIZE elements,
+		     separated by the stride, until we have a complete vector.
+		     Fall back to scalar accesses if that isn't possible.  */
+		  if (multiple_p (nunits, group_size))
+		    *memory_access_type = VMAT_STRIDED_SLP;
+		  else
+		    *memory_access_type = VMAT_ELEMENTWISE;
+		}
+	    }
+	  else if (cmp == 0 && loop_vinfo)
+	    {
+	      gcc_assert (vls_type == VLS_LOAD);
+	      *memory_access_type = VMAT_INVARIANT;
+	      /* Invariant accesses perform only component accesses, alignment
+		 is irrelevant for them.  */
+	      *alignment_support_scheme = dr_unaligned_supported;
+	    }
+	  else
+	    *memory_access_type = VMAT_CONTIGUOUS;
+
 	  overrun_p = loop_vinfo && gap != 0;
 	  if (overrun_p && vls_type != VLS_LOAD)
 	    {
@@ -2065,6 +2095,20 @@ get_group_load_store_type (vec_info *vinfo, stmt_vec_info stmt_info,
 			/ vect_get_scalar_dr_size (first_dr_info)))
 	    overrun_p = false;
 
+	  /* When we have a contiguous access across loop iterations
+	     but the access in the loop doesn't cover the full vector
+	     we can end up with no gap recorded but still excess
+	     elements accessed, see PR103116.  Make sure we peel for
+	     gaps if necessary and sufficient and give up if not.
+
+	     If there is a combination of the access not covering the full
+	     vector and a gap recorded then we may need to peel twice.  */
+	  if (loop_vinfo
+	      && SLP_TREE_LOAD_PERMUTATION (slp_node).exists ()
+	      && !multiple_p (group_size * LOOP_VINFO_VECT_FACTOR (loop_vinfo),
+			      nunits))
+	    overrun_p = true;
+
 	  /* If the gap splits the vector in half and the target
 	     can do half-vector operations avoid the epilogue peeling
 	     by simply loading half of the vector only.  Usually
@@ -2097,68 +2141,22 @@ get_group_load_store_type (vec_info *vinfo, stmt_vec_info stmt_info,
 				 "Peeling for outer loop is not supported\n");
 	      return false;
 	    }
-	  int cmp = compare_step_with_zero (vinfo, stmt_info);
-	  if (cmp < 0)
-	    {
-	      if (single_element_p)
-		/* ???  The VMAT_CONTIGUOUS_REVERSE code generation is
-		   only correct for single element "interleaving" SLP.  */
-		*memory_access_type = get_negative_load_store_type
-			     (vinfo, stmt_info, vectype, vls_type, 1, poffset);
-	      else
-		{
-		  /* Try to use consecutive accesses of DR_GROUP_SIZE elements,
-		     separated by the stride, until we have a complete vector.
-		     Fall back to scalar accesses if that isn't possible.  */
-		  if (multiple_p (nunits, group_size))
-		    *memory_access_type = VMAT_STRIDED_SLP;
-		  else
-		    *memory_access_type = VMAT_ELEMENTWISE;
-		}
-	    }
-	  else if (cmp == 0 && loop_vinfo)
-	    {
-	      gcc_assert (vls_type == VLS_LOAD);
-	      *memory_access_type = VMAT_INVARIANT;
-	      /* Invariant accesses perform only component accesses, alignment
-		 is irrelevant for them.  */
-	      *alignment_support_scheme = dr_unaligned_supported;
-	    }
-	  else
-	    *memory_access_type = VMAT_CONTIGUOUS;
-
-	  /* When we have a contiguous access across loop iterations
-	     but the access in the loop doesn't cover the full vector
-	     we can end up with no gap recorded but still excess
-	     elements accessed, see PR103116.  Make sure we peel for
-	     gaps if necessary and sufficient and give up if not.
-
-	     If there is a combination of the access not covering the full
-	     vector and a gap recorded then we may need to peel twice.  */
-	  if (loop_vinfo
-	      && *memory_access_type == VMAT_CONTIGUOUS
-	      && SLP_TREE_LOAD_PERMUTATION (slp_node).exists ()
-	      && !multiple_p (group_size * LOOP_VINFO_VECT_FACTOR (loop_vinfo),
-			      nunits))
-	    {
-	      unsigned HOST_WIDE_INT cnunits, cvf;
-	      if (!can_overrun_p
-		  || !nunits.is_constant (&cnunits)
+	  unsigned HOST_WIDE_INT cnunits, cvf;
+	  if (overrun_p
+	      && (!nunits.is_constant (&cnunits)
 		  || !LOOP_VINFO_VECT_FACTOR (loop_vinfo).is_constant (&cvf)
 		  /* Peeling for gaps assumes that a single scalar iteration
 		     is enough to make sure the last vector iteration doesn't
 		     access excess elements.
 		     ???  Enhancements include peeling multiple iterations
 		     or using masked loads with a static mask.  */
-		  || (group_size * cvf) % cnunits + group_size - gap < cnunits)
-		{
-		  if (dump_enabled_p ())
-		    dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
-				     "peeling for gaps insufficient for "
-				     "access\n");
-		  return false;
-		}
-	      overrun_p = true;
+		  || (group_size * cvf) % cnunits + group_size - gap < cnunits))
+	    {
+	      if (dump_enabled_p ())
+		dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+				 "peeling for gaps insufficient for "
+				 "access\n");
+	      return false;
 	    }
 
 	  /* If this is single-element interleaving with an element