From patchwork Wed Oct 30 13:08:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Joel Hutton X-Patchwork-Id: 1186689 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=gcc-patches-return-512050-incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b="HsGVVtHK"; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=armh.onmicrosoft.com header.i=@armh.onmicrosoft.com header.b="2IF1fiTt"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=armh.onmicrosoft.com header.i=@armh.onmicrosoft.com header.b="2IF1fiTt"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47380F2P3mz9sPj for ; Thu, 31 Oct 2019 00:08:26 +1100 (AEDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:references:in-reply-to :content-type:mime-version; q=dns; s=default; b=mYNlnBJEpAAeiTzk Ywn1xm2rJiseRe+j/iJ5Nrzr1jv8IXETYeG8HInOak8nn0XaYw4EscgiwMHaKksx L6UW8AQ8KYDti3ge+L6ZsHuDL3dVGPc8YCMefYYWUDZWpkDP5SnH5rVhvlP0b6VN oV4wgnaT2G7EUuTauxiFNXP4844= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:references:in-reply-to :content-type:mime-version; s=default; bh=Ob2ScEvdeNktxrhoFPLemx R9B9A=; b=HsGVVtHKbuGi5J0yHLNiQ+3uCQYCuQXTV2gTMyENDdCd0ew0+RVuJc bywzu9C0sXLviLDLDV+CxWTglXg33rvlv/ats9j8w/R7p0d5OqeRTDDv75CCaZiS QJJHstEIoiLlmZsoMu49i6bZDnT0lU+vdQVwEkPeXy3WRRRMUmbZk= Received: (qmail 62793 invoked by alias); 30 Oct 2019 13:08:18 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 62784 invoked by uid 89); 30 Oct 2019 13:08:17 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-23.7 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_LOTSOFHASH, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_PASS autolearn=ham version=3.3.1 spammy= X-HELO: EUR04-VI1-obe.outbound.protection.outlook.com Received: from mail-eopbgr80047.outbound.protection.outlook.com (HELO EUR04-VI1-obe.outbound.protection.outlook.com) (40.107.8.47) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 30 Oct 2019 13:08:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aP+wONorU7qfGQ0qBX2qCBq8bMWi6YIOuWQCraAWzRs=; b=2IF1fiTtTYqFKzhC21B69FSjIdgqGXBP7GjMxZl6vu9uqy1zlay8Gy3mDwL77xrBGTBQqHwbQscFJ2C2JGZwD7VQhtN2tt6Cwod0cZi93XRwpSzc8RSlBEto2o+qJ+wzSRLoIjf45lD5CmxQKLSyTb9/JTdku3kY4cgB4uM3aFM= Received: from VI1PR0802CA0025.eurprd08.prod.outlook.com (2603:10a6:800:a9::11) by VI1PR0802MB2254.eurprd08.prod.outlook.com (2603:10a6:800:9c::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2387.25; Wed, 30 Oct 2019 13:08:09 +0000 Received: from DB5EUR03FT046.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e0a::205) by VI1PR0802CA0025.outlook.office365.com (2603:10a6:800:a9::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2408.20 via Frontend Transport; Wed, 30 Oct 2019 13:08:09 +0000 Authentication-Results: spf=fail (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; gcc.gnu.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; gcc.gnu.org; dmarc=none action=none header.from=arm.com; Received-SPF: Fail (protection.outlook.com: domain of arm.com does not designate 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT046.mail.protection.outlook.com (10.152.21.230) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2387.20 via Frontend Transport; Wed, 30 Oct 2019 13:08:08 +0000 Received: ("Tessian outbound 081de437afc7:v33"); Wed, 30 Oct 2019 13:08:08 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: d461904ed1158139 X-CR-MTA-TID: 64aa7808 Received: from f06b012ed8c9.1 (cr-mta-lb-1.cr-mta-net [104.47.14.56]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id 30EBC495-997C-465B-A010-A92E1E586FBD.1; Wed, 30 Oct 2019 13:08:03 +0000 Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-vi1eur04lp2056.outbound.protection.outlook.com [104.47.14.56]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id f06b012ed8c9.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 30 Oct 2019 13:08:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TrB6XlJIeijD4VeYbZ6uSWHx/QgwjEZrEUG7/Kd7ir2sahV7pEFhF3V5WLDsZKXyp7cQ2Xrxahm11Q+gC+woqApHDYllbiDeU+4H4EIecBTRL4KvTaM81tyJnbxLqU+QpZDdLiX9lh2ck7dg+F9a0DMcXl3cY3A5TCw5hZSKti0Fk9/u/bFMYE40R7Bva6xjxhfCVzMzjDkZGu0rkbc0pGvmG0q8bxbhGVMTLtb59Fi53EkLXAROGXAlt1gWu1eGoVqlwXzKAXioT5V/Mg7+iThtWXvJmEidD/OScwijzE6xc7PW81FwLsRP55PJ/liv/J8aA/CbvC5kOT2LShdqaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aP+wONorU7qfGQ0qBX2qCBq8bMWi6YIOuWQCraAWzRs=; b=IdyRrM6sQXFBqyD+FhXxNzoS1N2nOZo9ONmfm5aOxlDbj55S3uTFlSCVwXkMaW4GeT+6/aMcnNXtfzN/m9b/PQNH8hWN/lD1XlOaHqWPeI0HYdHzMILXW3IGsGJVrzM4wxQExLWbIwSR4wp2vB1gteXrF8ozvthOxWT6CFQ7T6PD63zydY78ErR6HoLy2mC5ReTvPvt7FrMowtJoYDFwXF/ZzsyfIkenCCxmVLQtyPwJc/QNy7wRImjjp7EMpm5fL32qbtPgxPiu/n7+t3pk4YUUoNgdHlM+s1PmH9Jj5vvHa3GDolOnISS2QiNiGr6m66DJaQuHLMo0ZhIXVHFkFw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aP+wONorU7qfGQ0qBX2qCBq8bMWi6YIOuWQCraAWzRs=; b=2IF1fiTtTYqFKzhC21B69FSjIdgqGXBP7GjMxZl6vu9uqy1zlay8Gy3mDwL77xrBGTBQqHwbQscFJ2C2JGZwD7VQhtN2tt6Cwod0cZi93XRwpSzc8RSlBEto2o+qJ+wzSRLoIjf45lD5CmxQKLSyTb9/JTdku3kY4cgB4uM3aFM= Received: from HE1PR0801MB2059.eurprd08.prod.outlook.com (10.168.96.136) by HE1PR0801MB1867.eurprd08.prod.outlook.com (10.168.96.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2408.17; Wed, 30 Oct 2019 13:08:01 +0000 Received: from HE1PR0801MB2059.eurprd08.prod.outlook.com ([fe80::98fd:57d8:4367:a2af]) by HE1PR0801MB2059.eurprd08.prod.outlook.com ([fe80::98fd:57d8:4367:a2af%11]) with mapi id 15.20.2408.016; Wed, 30 Oct 2019 13:08:01 +0000 From: Joel Hutton To: Richard Biener CC: GCC Patches , nd Subject: [SLP] SLP vectorization: vectorize vector constructors Date: Wed, 30 Oct 2019 13:08:01 +0000 Message-ID: <6b66b279-eba2-80be-58f0-f72dc2316fcb@arm.com> References: <5edb0b00-4ae2-41c0-80ec-76de15d0b110@arm.com> In-Reply-To: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Joel.Hutton@arm.com; x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:9508;OLM:9508; X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(979002)(4636009)(346002)(396003)(136003)(366004)(376002)(39860400002)(189003)(199004)(54534003)(11346002)(8676002)(3846002)(52116002)(6116002)(6486002)(81166006)(76176011)(81156014)(102836004)(26005)(186003)(99286004)(53546011)(6512007)(4326008)(6436002)(6916009)(2906002)(25786009)(386003)(486006)(476003)(2616005)(6506007)(14454004)(8936002)(31686004)(4001150100001)(446003)(71190400001)(478600001)(71200400001)(58126008)(36756003)(316002)(305945005)(99936001)(31696002)(256004)(65806001)(54906003)(66066001)(86362001)(65956001)(7736002)(66446008)(64756008)(66946007)(66556008)(66476007)(66616009)(5660300002)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0801MB1867; H:HE1PR0801MB2059.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: mIKDvmBvjka9tUrTBPJaeCeIoMBd7ud8mVYBNHNdq8NMrHYtM4QpezVSuDjubLPEDmDEqct6Low50MpHvP4MzxI1305obSNZg5esS02cm2eljPiktmM4aOhzafwnfhTYmjRiWs2uAYIy9GXbOvEKGfzDa6fr+nvNrff3pp1npUO4jsTazlcW+/vdJX++vzdjT/s76yMK3jGgci/RUldR1xCEC2zAoE743EtpVBe4vqs+N+Qvlm1h+ElmlHo0kJXTDwR9JC/756sqI3TAyzrGnKUwe6LQYN9N8tyLk8/6+rr5idWNyf7nBMMqtCHHGWd3pmK7uXm07TrYMM2N7BclmD8GCLJMTpF8Jl90eOBSObsv6Q3bMzbeNehbtjj+f2pNXM+l2stxOdfnyCm+ji3DWvBUF/kPokFmFRUKzk8EQjQE1jgXp80qQK0P7U2ASjaD x-ms-exchange-transport-forked: True MIME-Version: 1.0 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Joel.Hutton@arm.com; X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT046.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 04eea878-b59a-478e-9a61-08d75d3a30f6 X-IsSubscribed: yes On 15/10/2019 13:11, Richard Biener wrote: >>  > You miss to check that CONSTRUCTOR_NELTS == TYPE_VECTOR_SUBPARTS >>  > (we can have omitted trailing zeros). > > ^^^ > > I don't see this being handled?  You give up on non-SSA names > but not on the omitted trailing zeros. I had thought checking the number of vectors produced would work. I've added that check. I'm slightly confused about what should be done for non-SSA names. There's no scalar stmt to gather for a constant in a vector constructor. > > You build a CONSTRUCTOR of vectors, thus > > _orig_ssa_1 = { vect_part1_2, vect_part2_3, ... }; I've added code to do this, and a testcase which triggers it. > > + > +         if (constructor) > +           { > +             SLP_INSTANCE_ROOT_STMT (new_instance) = stmt_info->stmt; > +           } > +         else > +           SLP_INSTANCE_ROOT_STMT (new_instance) = NULL; > + > > too much vertical space, no {} around single-stmt if clauses Fixed. > > > @@ -2725,6 +2760,10 @@ vect_bb_slp_scalar_cost (basic_block bb, >                 stmt_vec_info use_stmt_info = vinfo->lookup_stmt > (use_stmt); >                 if (!use_stmt_info || !PURE_SLP_STMT (use_stmt_info)) >                   { > +                   /* Check this is not a constructor that will be > vectorized > +                      away.  */ > +                   if (BB_VINFO_GROUPED_STORES (vinfo).contains > (use_stmt_info)) > +                       continue; > > hmm, so why not set the slp type on SLP_INSTANCE_ROOT_STMT instead? > In theory the stmt should be part of the SLP tree itself but that's > probably too awkward to be made work at the moment ;) I did try this, but it was indeed very awkward to be made to work. > > vect_ssa_use_outside_bb and vect_slp_check_for_constructors are new > functions which need comments. Fixed. I've also taken the 'vectorize the root' out into a separate function. > > +  /* For vector constructors, the same SSA name must be used to maintain > data > +     flow into other basic blocks.  */ > +  if (instance->root == node && SLP_INSTANCE_ROOT_STMT (instance) > +      && SLP_TREE_NUMBER_OF_VEC_STMTS (node) == 1 > +      && SLP_TREE_VEC_STMTS (node).exists ()) > +    { > > it should read > >   /* Vectorize the instance root.  */ > > and be in vect_schedule_slp after the vect_schedule_slp_instance. > As said above you need to handle SLP_TREE_NUMBER_OF_VEC_STMTS > 1, > you also cannot simply do "nothing" here, "failing" vectorization > (well, you probably can - DCE will remove the vectorized code - but > at least if there were calls in the SLP tree they will be mangled > by vectorization so the scalar code is wrecked).  SO it should be > >  if (SLP_INSTANCE_ROOT_STMT (instance)) >    .. you may not fail to replace the scalar root stmt here .. > So what should be done in the case that CONSTRUCTOR_NELTS < TYPE_VECTOR_SUBPARTS? > +         if (CONSTRUCTOR_NELTS (rhs) == 0) > +           vectorizable = false; > + > > if you use continue; you can elide the 'vectorizable' variable. Done. > > +         if (!vect_ssa_use_outside_bb (gimple_assign_lhs (stmt))) > +           vectorizable = false; > + > > why's that?  no comments that clarify ;)  The vector may be > used as argument to a call or as source of a store.  So I'd simply > remove this check (and the function). Done. The thinking was that if the vector was used as a source of a store the SLP tree would be built from the grouped store instead. Will it not cause problems if both the grouped store and the vector constructor are used to build SLP trees? 2019-10-10 Joel Hutton * expr.c (store_constructor): Add case for constructor of vectors. * tree-vect-slp.c (vect_analyze_slp_instance): Add case for vector constructors. (vect_bb_slp_scalar_cost): Likewise. (vect_slp_check_for_constructors): New function. (vect_slp_analyze_bb_1): Add check for vector constructors. (vect_schedule_slp_instance): Add case to fixup vector constructor stmt. (vectorize_slp_instance_root_stmt): New function * tree-vectorizer.h (SLP_INSTANCE_ROOT_STMT): New field. gcc/testsuite/ChangeLog: 2019-10-10 Joel Hutton * gcc.dg/vect/bb-slp-40.c: New test. * gcc.dg/vect/bb-slp-41.c: New test. From 902510bd498acfc9e30636f8267b57027bc63254 Mon Sep 17 00:00:00 2001 From: Joel Hutton Date: Tue, 22 Oct 2019 10:05:19 +0100 Subject: [PATCH] SLP Vectorization: Vectorize vector constructors --- gcc/expr.c | 5 +- gcc/testsuite/gcc.dg/vect/bb-slp-40.c | 34 +++++ gcc/testsuite/gcc.dg/vect/bb-slp-41.c | 61 +++++++++ gcc/tree-vect-slp.c | 171 +++++++++++++++++++++++++- gcc/tree-vectorizer.h | 5 + 5 files changed, 273 insertions(+), 3 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-40.c create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-41.c diff --git a/gcc/expr.c b/gcc/expr.c index 476c6865f20828fc68f455e70d4874eaabd9d08d..ea4705e47c7e8424cca5e19319ef21db415f496a 100644 --- a/gcc/expr.c +++ b/gcc/expr.c @@ -6825,7 +6825,10 @@ store_constructor (tree exp, rtx target, int cleared, poly_int64 size, { unsigned int n = const_n_elts; - if (emode != eltmode) + if (emode != eltmode + || (TREE_CODE (TREE_TYPE (exp)) == VECTOR_TYPE + && TREE_CODE (TREE_TYPE (CONSTRUCTOR_ELT(exp, 0)->value)) + == VECTOR_TYPE)) { n = CONSTRUCTOR_NELTS (exp); vec_vec_init_p = true; diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-40.c b/gcc/testsuite/gcc.dg/vect/bb-slp-40.c new file mode 100644 index 0000000000000000000000000000000000000000..a1dd372184623f34f8f2825aa5da50dc70c98084 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-40.c @@ -0,0 +1,34 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -fdump-tree-slp-all" } */ +/* { dg-require-effective-target vect_int } */ + +char g_d[1024], g_s1[1024], g_s2[1024]; +void foo(void) +{ + char *d = g_d, *s1 = g_s1, *s2 = g_s2; + + for ( int y = 0; y < 128; y++ ) + { + d[0 ] = s1[0 ] + s2[0 ]; + d[1 ] = s1[1 ] + s2[1 ]; + d[2 ] = s1[2 ] + s2[2 ]; + d[3 ] = s1[3 ] + s2[3 ]; + d[4 ] = s1[4 ] + s2[4 ]; + d[5 ] = s1[5 ] + s2[5 ]; + d[6 ] = s1[6 ] + s2[6 ]; + d[7 ] = s1[7 ] + s2[7 ]; + d[8 ] = s1[8 ] + s2[8 ]; + d[9 ] = s1[9 ] + s2[9 ]; + d[10] = s1[10] + s2[10]; + d[11] = s1[11] + s2[11]; + d[12] = s1[12] + s2[12]; + d[13] = s1[13] + s2[13]; + d[14] = s1[14] + s2[14]; + d[15] = s1[15] + s2[15]; + d += 16; + } +} + +/* See that we vectorize an SLP instance. */ +/* { dg-final { scan-tree-dump-times "Found vectorizable constructor" 1 "slp1" } } */ +/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "slp1" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-41.c b/gcc/testsuite/gcc.dg/vect/bb-slp-41.c new file mode 100644 index 0000000000000000000000000000000000000000..618d1d0ef3f6da93b53df62296620d589ad2e21a --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/bb-slp-41.c @@ -0,0 +1,61 @@ +/* { dg-do run } */ +/* { dg-options "-O3 -fdump-tree-slp-all -fno-vect-cost-model" } */ +/* { dg-require-effective-target vect_int } */ + +#define ARR_SIZE 1000 + +void foo (int *a, int *b) +{ + int i; + for (i = 0; i < (ARR_SIZE - 2); ++i) + a[i] = b[0] + b[1] + b[i+1] + b[i+2]; +} + +void bar (int *a, int *b) +{ + int i; + for (i = 0; i < (ARR_SIZE - 2); ++i) + { + a[i] = b[0]; + } + for (i = 0; i < (ARR_SIZE - 2); ++i) + { + a[i] = a[i] + b[1]; + } + for (i = 0; i < (ARR_SIZE - 2); ++i) + { + a[i] = a[i] + b[i+1]; + } + for (i = 0; i < (ARR_SIZE - 2); ++i) + { + a[i] = a[i] + b[i+2]; + } +} + +int main () +{ + int a1[ARR_SIZE]; + int a2[ARR_SIZE]; + int b[ARR_SIZE]; + int i; + + for (i = 0; i < ARR_SIZE; i++) + { + a1[i] = 0; + a2[i] = 0; + b[i] = i; + } + + foo (a1, b); + bar (a2, b); + + for (i = 0; i < ARR_SIZE; i++) + if (a1[i] != a2[i]) + return 1; + + return 0; + +} +/* See that we vectorize an SLP instance. */ +/* { dg-final { scan-tree-dump-times "Found vectorizable constructor" 4 "slp1" } } */ +/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "slp1" } } */ diff --git a/gcc/tree-vect-slp.c b/gcc/tree-vect-slp.c index e1061ede061751f5fd6c55edb56671a854e7456e..be7bb27899da06d22ddc4833b3e90a2f8c000177 100644 --- a/gcc/tree-vect-slp.c +++ b/gcc/tree-vect-slp.c @@ -1983,6 +1983,7 @@ vect_analyze_slp_instance (vec_info *vinfo, unsigned int i; struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info); vec scalar_stmts; + bool constructor = false; if (STMT_VINFO_GROUPED_ACCESS (stmt_info)) { @@ -1996,6 +1997,13 @@ vect_analyze_slp_instance (vec_info *vinfo, vectype = STMT_VINFO_VECTYPE (stmt_info); group_size = REDUC_GROUP_SIZE (stmt_info); } + else if (is_gimple_assign (stmt_info->stmt) + && gimple_assign_rhs_code (stmt_info->stmt) == CONSTRUCTOR) + { + vectype = TREE_TYPE (gimple_assign_rhs1 (stmt_info->stmt)); + group_size = CONSTRUCTOR_NELTS (gimple_assign_rhs1 (stmt_info->stmt)); + constructor = true; + } else { gcc_assert (is_a (vinfo)); @@ -2042,6 +2050,25 @@ vect_analyze_slp_instance (vec_info *vinfo, STMT_VINFO_REDUC_DEF (vect_orig_stmt (stmt_info)) = STMT_VINFO_REDUC_DEF (vect_orig_stmt (scalar_stmts.last ())); } + else if (constructor) + { + tree rhs = gimple_assign_rhs1 (stmt_info->stmt); + tree val; + FOR_EACH_CONSTRUCTOR_VALUE (CONSTRUCTOR_ELTS (rhs), i, val) + { + if (TREE_CODE (val) == SSA_NAME) + { + gimple* def = SSA_NAME_DEF_STMT (val); + stmt_vec_info def_info = vinfo->lookup_stmt (def); + /* Value is defined in another basic block. */ + if (!def_info) + return false; + scalar_stmts.safe_push (def_info); + } + else + return false; + } + } else { /* Collect reduction statements. */ @@ -2099,6 +2126,8 @@ vect_analyze_slp_instance (vec_info *vinfo, SLP_INSTANCE_GROUP_SIZE (new_instance) = group_size; SLP_INSTANCE_UNROLLING_FACTOR (new_instance) = unrolling_factor; SLP_INSTANCE_LOADS (new_instance) = vNULL; + SLP_INSTANCE_ROOT_STMT (new_instance) = constructor ? stmt_info->stmt\ + : NULL; vect_gather_slp_loads (new_instance, node); if (dump_enabled_p ()) dump_printf_loc (MSG_NOTE, vect_location, @@ -2801,6 +2830,10 @@ vect_bb_slp_scalar_cost (basic_block bb, stmt_vec_info use_stmt_info = vinfo->lookup_stmt (use_stmt); if (!use_stmt_info || !PURE_SLP_STMT (use_stmt_info)) { + /* Check this is not a constructor that will be vectorized + away. */ + if (BB_VINFO_GROUPED_STORES (vinfo).contains (use_stmt_info)) + continue; (*life)[i] = true; BREAK_FROM_IMM_USE_STMT (use_iter); } @@ -2912,6 +2945,38 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo) return true; } +/* Find any vectorizable constructors, and add them to the grouped_store + array. */ + +static void +vect_slp_check_for_constructors (bb_vec_info bb_vinfo) +{ + gimple_stmt_iterator gsi; + + for (gsi = bb_vinfo->region_begin; + gsi_stmt (gsi) != gsi_stmt (bb_vinfo->region_end); gsi_next (&gsi)) + { + gimple *stmt = gsi_stmt (gsi); + + if (is_gimple_assign (stmt) + && gimple_assign_rhs_code (stmt) == CONSTRUCTOR + && TREE_CODE (gimple_assign_lhs (stmt)) == SSA_NAME + && TREE_CODE (TREE_TYPE (gimple_assign_lhs (stmt))) == VECTOR_TYPE) + { + tree rhs = gimple_assign_rhs1 (stmt); + + if (CONSTRUCTOR_NELTS (rhs) == 0) + continue; + + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "Found vectorizable constructor: %G\n", stmt); + stmt_vec_info stmt_info = bb_vinfo->lookup_stmt (stmt); + BB_VINFO_GROUPED_STORES (bb_vinfo).safe_push (stmt_info); + } + } +} + /* Check if the region described by BB_VINFO can be vectorized, returning true if so. When returning false, set FATAL to true if the same failure would prevent vectorization at other vector sizes, false if it is still @@ -2959,6 +3024,8 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int n_stmts, bool &fatal) return false; } + vect_slp_check_for_constructors (bb_vinfo); + /* If there are no grouped stores in the region there is no need to continue with pattern recog as vect_analyze_slp will fail anyway. */ @@ -4022,6 +4089,68 @@ vect_remove_slp_scalar_calls (slp_tree node) vect_remove_slp_scalar_calls (node, visited); } +/* Vectorize the instance root. Return success or failure. */ + +bool +vectorize_slp_instance_root_stmt (slp_tree node, slp_instance instance) +{ + + if (SLP_TREE_NUMBER_OF_VEC_STMTS (node) == 1) + { + stmt_vec_info child_stmt_info; + int j; + tree constructor; + constructor = gimple_assign_rhs1 (SLP_INSTANCE_ROOT_STMT (instance)); + + FOR_EACH_VEC_ELT (SLP_TREE_VEC_STMTS (node), j, child_stmt_info) + { + tree vect_lhs = gimple_get_lhs (child_stmt_info->stmt); + gassign *rstmt + = gimple_build_assign (gimple_get_lhs (instance->root_stmt), + gimple_get_lhs (child_stmt_info->stmt)); + gimple_stmt_iterator rgsi = gsi_for_stmt (instance->root_stmt); + if (TYPE_VECTOR_SUBPARTS (TREE_TYPE (vect_lhs)).is_constant () + && (CONSTRUCTOR_NELTS (constructor) + == TYPE_VECTOR_SUBPARTS (TREE_TYPE (vect_lhs)).to_constant ())) + { + gsi_replace (&rgsi, rstmt, true); + SLP_INSTANCE_ROOT_STMT (instance) = NULL; + } + break; + } + } + else if (SLP_TREE_NUMBER_OF_VEC_STMTS (node) > 1) + { + int nelts = SLP_TREE_NUMBER_OF_VEC_STMTS (node); + stmt_vec_info child_stmt_info; + int j; + vec *v; + vec_alloc (v, nelts); + FOR_EACH_VEC_ELT (SLP_TREE_VEC_STMTS (node), j, child_stmt_info) + { + CONSTRUCTOR_APPEND_ELT (v, + NULL_TREE, + gimple_get_lhs (child_stmt_info->stmt)); + } + tree type = TREE_TYPE (gimple_assign_rhs1 (instance->root_stmt)); + tree r_constructor = build_constructor (type, v); + gassign *rstmt + = gimple_build_assign (gimple_get_lhs (instance->root_stmt), + r_constructor); + gimple_stmt_iterator rgsi = gsi_for_stmt (instance->root_stmt); + gsi_replace (&rgsi, rstmt, true); + SLP_INSTANCE_ROOT_STMT (instance) = NULL; + } + else + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "vectorization of constructor failed.\n"); + return false; + } + return true; +} + /* Generate vector code for all SLP instances in the loop/basic block. */ void @@ -4036,9 +4165,20 @@ vect_schedule_slp (vec_info *vinfo) slp_instances = vinfo->slp_instances; FOR_EACH_VEC_ELT (slp_instances, i, instance) { + slp_tree node = SLP_INSTANCE_TREE (instance); /* Schedule the tree of INSTANCE. */ - vect_schedule_slp_instance (SLP_INSTANCE_TREE (instance), + vect_schedule_slp_instance (node, instance, bst_map); + + /* Vectorize the instance root. */ + if (instance->root == node && SLP_INSTANCE_ROOT_STMT (instance) + && SLP_TREE_VEC_STMTS (node).exists ()) + if (!vectorize_slp_instance_root_stmt (node, instance)) + { + delete bst_map; + return; + } + if (dump_enabled_p ()) dump_printf_loc (MSG_NOTE, vect_location, "vectorizing stmts using SLP.\n"); @@ -4061,15 +4201,42 @@ vect_schedule_slp (vec_info *vinfo) if (is_a (vinfo)) vect_remove_slp_scalar_calls (root); + auto_vec removed; + for (j = 0; SLP_TREE_SCALAR_STMTS (root).iterate (j, &store_info) && j < SLP_INSTANCE_GROUP_SIZE (instance); j++) { if (!STMT_VINFO_DATA_REF (store_info)) break; + if (removed.contains (store_info)) + continue; + store_info = vect_orig_stmt (store_info); + tree lhs = gimple_get_lhs (store_info->stmt); + bool used = false; + /* We must ensure we don't remove def stmts for ssa names used + elsewhere. */ + if (TREE_CODE (lhs) == SSA_NAME) + { + gimple *use_stmt; + imm_use_iterator use_iter; + + FOR_EACH_IMM_USE_STMT (use_stmt, use_iter, lhs) + { + if (use_stmt != store_info->stmt) + { + used = true; + BREAK_FROM_IMM_USE_STMT (use_iter); + } + } + } + /* Free the attached stmt_vec_info and remove the stmt. */ - vinfo->remove_stmt (store_info); + + if (!used) + vinfo->remove_stmt (store_info); + removed.safe_push (store_info); } } } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 56be28b0cc5a77412f996e70636b08d5b615813e..9f8419e4208b7d438ace41892022f93ebcadd019 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -151,6 +151,10 @@ public: /* The root of SLP tree. */ slp_tree root; + /* For vector constructors, the constructor stmt that the SLP tree is built + from, NULL otherwise. */ + gimple *root_stmt; + /* Size of groups of scalar stmts that will be replaced by SIMD stmt/s. */ unsigned int group_size; @@ -170,6 +174,7 @@ public: #define SLP_INSTANCE_GROUP_SIZE(S) (S)->group_size #define SLP_INSTANCE_UNROLLING_FACTOR(S) (S)->unrolling_factor #define SLP_INSTANCE_LOADS(S) (S)->loads +#define SLP_INSTANCE_ROOT_STMT(S) (S)->root_stmt #define SLP_TREE_CHILDREN(S) (S)->children #define SLP_TREE_SCALAR_STMTS(S) (S)->stmts -- 2.17.1