From patchwork Tue Apr 6 10:02:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 1462709 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=gcc-patches-bounces@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=tosxDf2H; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FF3472l1Lz9sWd for ; Tue, 6 Apr 2021 20:02:46 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0FDC5389246B; Tue, 6 Apr 2021 10:02:43 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 0FDC5389246B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1617703363; bh=PwqraR9av1CGmUAFAcvOxfM8lOxY9ByNQgqBLsjIRTw=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=tosxDf2HnBlKHmLI6B4Y0yQV2lPWCE/0z5zS0btGHoFaDVnY8qmKg6biw3VEU48Hc f/Ztrl8eaorlCd5N108BRE0wvd59hrosQA6F7MS/so2G3tY6TGJ5vL3JaLECjKla3S XcTenT91+yM5zKfX9JWQ6WtIeUdkBevhsd4cdBjo= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by sourceware.org (Postfix) with ESMTP id E5CB83858034 for ; Tue, 6 Apr 2021 10:02:39 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org E5CB83858034 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 802F931B for ; Tue, 6 Apr 2021 03:02:39 -0700 (PDT) Received: from localhost (e121540-lin.manchester.arm.com [10.32.98.126]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 26D7E3F73D for ; Tue, 6 Apr 2021 03:02:39 -0700 (PDT) To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@arm.com Subject: vect: Don't split store groups if we have IFN_STORE_LANES [PR99873] Date: Tue, 06 Apr 2021 11:02:38 +0100 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-Spam-Status: No, score=-12.6 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Sandiford via Gcc-patches From: Richard Sandiford Reply-To: Richard Sandiford Errors-To: gcc-patches-bounces@gcc.gnu.org Sender: "Gcc-patches" As noted in the PR, we were no longer using ST3 for the testcase and instead stored each lane individually. This is because we'd split the store group during SLP and couldn't recover when SLP failed. However, we seem to get better code with ST3 and ST4 even if SLP would have succeeded, such as for vect-complex-5.c. I think the best thing for GCC 11 would therefore be to skip the split entirely if we could use IFN_STORE_LANES. A downside of this is that SLP can handle smaller iteration counts than IFN_STORE_LANES can, but we don't have the infrastructure to choose reliably based on that. Tested on aarch64-linux-gnu (with and without SVE), arm-linux-gnueabihf, armeb-eabi and x86_64-linux-gnu. OK to install? Richard gcc/ PR tree-optimization/99873 * tree-vect-slp.c (vect_could_use_store_lanes_p): New function. (vect_build_slp_instance): Don't split store groups that could use IFN_STORE_LANES. gcc/testsuite/ * gcc.dg/vect/slp-21.c: Only expect 2 of the loops to use SLP if IFN_STORE_LANES is available. * vect/vect-complex-5.c: Expect no loops to use SLP if IFN_STORE_LANES is available. * gcc.target/aarch64/pr99873.c: New test. * gcc.target/aarch64/sve/pr99873.c: Likewise. --- gcc/testsuite/gcc.dg/vect/slp-21.c | 4 ++-- gcc/testsuite/gcc.dg/vect/vect-complex-5.c | 3 ++- gcc/testsuite/gcc.target/aarch64/pr99873.c | 17 +++++++++++++++++ gcc/testsuite/gcc.target/aarch64/sve/pr99873.c | 15 +++++++++++++++ gcc/tree-vect-slp.c | 15 ++++++++++++++- 5 files changed, 50 insertions(+), 4 deletions(-) create mode 100644 gcc/testsuite/gcc.target/aarch64/pr99873.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/pr99873.c diff --git a/gcc/tree-vect-slp.c b/gcc/tree-vect-slp.c index ceec7f5c410..b0c03da3aeb 100644 --- a/gcc/tree-vect-slp.c +++ b/gcc/tree-vect-slp.c @@ -2458,6 +2458,18 @@ vect_match_slp_patterns (slp_instance instance, vec_info *vinfo, return vect_match_slp_patterns_2 (ref_node, vinfo, perm_cache, visited); } +/* Return true if the store group in STMT_INFO could use IFN_STORE_LANES. + GROUP_SIZE is the number of elements in the group. */ + +static bool +vect_could_use_store_lanes_p (vec_info *vinfo, stmt_vec_info stmt_info, + unsigned int group_size) +{ + tree scalar_type = TREE_TYPE (DR_REF (STMT_VINFO_DATA_REF (stmt_info))); + tree vectype = get_vectype_for_scalar_type (vinfo, scalar_type, group_size); + return vectype && vect_store_lanes_supported (vectype, group_size, false); +} + /* Analyze an SLP instance starting from a group of grouped stores. Call vect_build_slp_tree to build a tree of packed stmts if possible. Return FALSE if it's impossible to SLP any stmt in the loop. */ @@ -2693,7 +2705,8 @@ vect_build_slp_instance (vec_info *vinfo, /* For loop vectorization split into arbitrary pieces of size > 1. */ if (is_a (vinfo) - && (i > 1 && i < group_size)) + && (i > 1 && i < group_size) + && !vect_could_use_store_lanes_p (vinfo, stmt_info, group_size)) { unsigned group1_size = i; diff --git a/gcc/testsuite/gcc.dg/vect/slp-21.c b/gcc/testsuite/gcc.dg/vect/slp-21.c index bf8f434dd50..85393975b45 100644 --- a/gcc/testsuite/gcc.dg/vect/slp-21.c +++ b/gcc/testsuite/gcc.dg/vect/slp-21.c @@ -210,7 +210,7 @@ int main (void) Not all vect_perm targets support that, and it's a bit too specific to have its own effective-target selector, so we just test targets directly. */ -/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect" { target { aarch64*-*-* arm*-*-* powerpc64*-*-* } } } } */ -/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target { vect_strided4 && { ! { aarch64*-*-* arm*-*-* powerpc64*-*-* } } } } } } */ +/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect" { target powerpc64*-*-* } } } */ +/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target { vect_strided4 && { ! powerpc64*-*-* } } } } } */ /* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 0 "vect" { target { ! { vect_strided4 } } } } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-complex-5.c b/gcc/testsuite/gcc.dg/vect/vect-complex-5.c index 81fdb67ce81..addcf60438c 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-complex-5.c +++ b/gcc/testsuite/gcc.dg/vect/vect-complex-5.c @@ -40,4 +40,5 @@ main (void) return 0; } -/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { xfail { ! vect_hw_misalign } } } } */ +/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 0 "vect" { target vect_load_lanes } } } */ +/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target { ! vect_load_lanes } xfail { ! vect_hw_misalign } } } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/pr99873.c b/gcc/testsuite/gcc.target/aarch64/pr99873.c new file mode 100644 index 00000000000..bc4d81e3ae5 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/pr99873.c @@ -0,0 +1,17 @@ +/* { dg-options "-O3" } */ + +#pragma GCC target "+nosve" + +void +f (int *restrict x, int *restrict y, int *restrict z, int n) +{ + for (int i = 0; i < n; i += 3) + { + x[i] = y[i] + z[i]; + x[i + 1] = y[i + 1] - z[i + 1]; + x[i + 2] = y[i + 2] | z[i + 2]; + } +} + +/* { dg-final { scan-assembler-times {\tld3\t} 2 } } */ +/* { dg-final { scan-assembler-times {\tst3\t} 1 } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/pr99873.c b/gcc/testsuite/gcc.target/aarch64/sve/pr99873.c new file mode 100644 index 00000000000..f4b95da2afa --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/pr99873.c @@ -0,0 +1,15 @@ +/* { dg-options "-O3" } */ + +void +f (int *restrict x, int *restrict y, int *restrict z, int n) +{ + for (int i = 0; i < n; i += 3) + { + x[i] = y[i] + z[i]; + x[i + 1] = y[i + 1] - z[i + 1]; + x[i + 2] = y[i + 2] | z[i + 2]; + } +} + +/* { dg-final { scan-assembler-times {\tld3w\t} 2 } } */ +/* { dg-final { scan-assembler-times {\tst3w\t} 1 } } */