From patchwork Mon Nov 6 07:38:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tamar Christina X-Patchwork-Id: 1859825 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=armh.onmicrosoft.com header.i=@armh.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-armh-onmicrosoft-com header.b=6SmjiiCI; dkim=pass (1024-bit key) header.d=armh.onmicrosoft.com header.i=@armh.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-armh-onmicrosoft-com header.b=6SmjiiCI; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SP3BL1djMz1yQ9 for ; Mon, 6 Nov 2023 18:38:50 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A8550385842A for ; Mon, 6 Nov 2023 07:38:47 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2068.outbound.protection.outlook.com [40.107.22.68]) by sourceware.org (Postfix) with ESMTPS id D9EA83858C31 for ; Mon, 6 Nov 2023 07:38:28 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D9EA83858C31 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org D9EA83858C31 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=40.107.22.68 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1699256313; cv=pass; b=rDlTjYkX4uvGnczWxl/iY1oLZDf2/cGSbSGlKy5WTS3p9XnzFDTAWPQl2M/2OIEGp0qY3OL5rXOJYiHaVvmLiGadwMHtypbQ5yZMOMYAlaPvWuF2Gc7TD3ZmyRReoVnuUkyS9qEbzIFPJbCMFWgtwyvXtlZ3j4u76qj5hMo1xD4= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1699256313; c=relaxed/simple; bh=rH/sNApI2nZ7QWn6Fs1iT0MwXrKAq7O5/ObvGTyTBlI=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=J4uH01DggDRN6gMfpos+A8EaH+Mx9zmXDfYR4KlOpO94+x21Ncff6476D0LRHL3ibSQCim/qprAiSemc3ygLoQ5MI0LjCW4zyDHtySZwsNnwmdt5Upyeuuph+l1U384VvS6eMLSan9NmSKidt1WWcKzYNU9LvEZyuKwMX35qHSI= ARC-Authentication-Results: i=3; server2.sourceware.org ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=aiZXO5xSEHM4l6dJUPfbKKqJRTO2jK0DG9bih/BDrA7MOe1O0DbRz0P6JgWRUFI3iy7qn+/MrowkodPYeYLFutFvNJukOkQ9FWAQE8rW9PeIQ9pAU7IT2no23Icnh2HAwLxkJEGU9COAZnPENcqCr953zUpzgRKUs374sPntflCXEPftaSNXmcC+xWqqKMixeTA3L1mH3U6eWnMB5+YHFWa7CxXG0wRx7p8D4CXVDKTus1vnv60PQ2OHbVqBdXfzclHBhfEMysFUIt4zMTg+AZGHmsRrjOrG0ADDV5BzC63CEJnEKo1NUjMzjSYP9PBoWE1laLErdCG43bUcg5goHQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Dt7Z3l0QA4Gx+nXeK2Lp/Ss74JFPkLGxbZihaSrrMI8=; b=fMkrh+sW83S6l0MMqGePBrr1KCFNbjxgJRbiSdKQ8zZMU7CbzoCznjXiS2hDp5aeH94XFcCUJbhKkajkH6tyMYZVUrPGyG8X7zhy1tH4C7uWfQgCFkP+fbX5Fgw6UQZoaRieewQpaxmILFL4lWMmjqlMoJV1M0uBIzZL4q2jiu9dGPg1Ff4Oe2oME5l62ZIhpXY1C02gedqwNHBX/snnWpPsXt8Bqp6woBZ0nQHy5fVH6Da/vywz6xYlaHg21a+feQ2O9YG2RIwfmBol9KwOqmvpJQ3qcbdywEycrHXqbUm4q5PP67q2XORj40Yfw7QQr5vgQYg3tazzx3YtKzR49w== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Dt7Z3l0QA4Gx+nXeK2Lp/Ss74JFPkLGxbZihaSrrMI8=; b=6SmjiiCI3wWLdHwbtBj2e8Pkf9icu7bymiHSex/Cs7uENYF+Ze7/1hpLoDg7sqBZsJanm5Jo5VGcY2+3rA+l3oDqvF0J51/D10dYC8IzS+7Fixv2j6fsFEzU2oS25splpRHGV11OZwjyx5Qbwx4Ka1OGtJXmgaLhCpdNWbXJHG0= Received: from DB9PR05CA0027.eurprd05.prod.outlook.com (2603:10a6:10:1da::32) by PAXPR08MB6526.eurprd08.prod.outlook.com (2603:10a6:102:df::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28; Mon, 6 Nov 2023 07:38:25 +0000 Received: from DU2PEPF0001E9C5.eurprd03.prod.outlook.com (2603:10a6:10:1da:cafe::98) by DB9PR05CA0027.outlook.office365.com (2603:10a6:10:1da::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28 via Frontend Transport; Mon, 6 Nov 2023 07:38:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DU2PEPF0001E9C5.mail.protection.outlook.com (10.167.8.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6977.16 via Frontend Transport; Mon, 6 Nov 2023 07:38:25 +0000 Received: ("Tessian outbound 385ad2f98d71:v228"); Mon, 06 Nov 2023 07:38:25 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 79e3a54454ebbccc X-CR-MTA-TID: 64aa7808 Received: from 0e93ee853faf.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 59E4390F-3862-4C6A-A00E-4C196BFB3E9C.1; Mon, 06 Nov 2023 07:38:18 +0000 Received: from EUR03-AM7-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 0e93ee853faf.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 06 Nov 2023 07:38:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TmBpvEf8up5TALGu5bJ4xeqr+rrk87wn4N1peQIUiAalMr3Gw+xUAzi7PrNt5JdG3yw4/0QhqPwLIRFvf0QnJ2lJJy/Q5S8U7B0ZENIf/c7IYzF5Knv5ICcXMsHvMgscl1+Yu1fUcCU0782cJ88LC/YnaBCaqlJ6N9iq8ghRGFKrpGV/LrR09ANv8/kE2iIR6s/Cc9raYYAG44zVo0OWDU/6U0hSR9NehxxAgRmjw853m0FQZasz9zewLjzcrCK/6v0xM+uYCE5HUuhhxKZjec3P3IGIezZ8MD/S1MXuXYL4BFytVtF5xX2JcN5F7PTj3YUAkublKxtifFQLv+2mqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Dt7Z3l0QA4Gx+nXeK2Lp/Ss74JFPkLGxbZihaSrrMI8=; b=cVvAzcfnBjvQgHOsbWoLT5a9JNb8yLhAb5CChbB4IMoocX9w0aLPhwFh0Fzg0c12J0FjkQjQz/UYSO4DwH892eKOWDT3K2tGJ0fa20oFkOuSAtM4yPNRbb0tGaf8arM/kKuxVwp6ACkcyZZo/YhnE6uhs2XoZR5htAJ3+VI1gPOqF30ZJfAkI3OyQe7NY2gfGg5q0DoTWd5IT+T6bj5KdFRireep2MP0dp2TL4elbfps66xFvI7n6Bu0JyvNps6QKxI0183vzvqjEsPdXRJmIN3eqmzrXKUSv0O9+N8WWT6Ozfg2sJXB3b8reufZS06R+2tLkv052Ie16jdr4p14EA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Dt7Z3l0QA4Gx+nXeK2Lp/Ss74JFPkLGxbZihaSrrMI8=; b=6SmjiiCI3wWLdHwbtBj2e8Pkf9icu7bymiHSex/Cs7uENYF+Ze7/1hpLoDg7sqBZsJanm5Jo5VGcY2+3rA+l3oDqvF0J51/D10dYC8IzS+7Fixv2j6fsFEzU2oS25splpRHGV11OZwjyx5Qbwx4Ka1OGtJXmgaLhCpdNWbXJHG0= Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; Received: from VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) by DU0PR08MB9581.eurprd08.prod.outlook.com (2603:10a6:10:44b::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28; Mon, 6 Nov 2023 07:38:15 +0000 Received: from VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::26aa:efdd:a74a:27d0]) by VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::26aa:efdd:a74a:27d0%5]) with mapi id 15.20.6954.028; Mon, 6 Nov 2023 07:38:15 +0000 Date: Mon, 6 Nov 2023 07:38:12 +0000 From: Tamar Christina To: gcc-patches@gcc.gnu.org Cc: nd@arm.com, rguenther@suse.de, jlaw@ventanamicro.com Subject: [PATCH 4/21]middle-end: update loop peeling code to maintain LCSSA form for early breaks Message-ID: Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO2P265CA0364.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:a3::16) To VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) MIME-Version: 1.0 X-MS-TrafficTypeDiagnostic: VI1PR08MB5325:EE_|DU0PR08MB9581:EE_|DU2PEPF0001E9C5:EE_|PAXPR08MB6526:EE_ X-MS-Office365-Filtering-Correlation-Id: 600aff09-49aa-4eff-dc41-08dbde9b5c89 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 0oMec9wXXPjIWj4lSJ+wrSXbTqz1VlpNjDjZAV3gTx8YwhpdcNegvwVwmbwGGaazmFh9qgWV/oSs2HTpMnopvohUleaaOETN/qUoAJ7bNr6DFAPw88RRvcwddc7Bo6WqU4ebRNXd7Phw5eXLn/2L4zIODZiHAbENSv5r++fIZtz4QtDjQWXZEjascrxDf1hGnY8vyKU+6tZG8dhT87H8sHsXRaE/QUTW95rc8XH0L7fXWvcgvF6t7qHwqVWyXSk4L3Rn0BTE52aMQkIPFzMVBa1ZEFSAcSfEtqMMGRYKNJ5akito719r51EqpUnUQFn/2/v4Na/70IljV5PYrU53h70oo92RnxWOsNc+p9doBqJJCFlNMSsEBsGW0l2KQwGaIE5ytLyBTIOTKyLRmy2QAR9gEuDsjk1twyoZreuqGWig6NhWkdC2iBg1rFMzdyDOf+F8mWwy008xp5YuxLk3yK5j/A8JlLq7x7187RPMVnj/4NifXR1BOUI9IyU5x4B6yGvPqri8cOTAgw3ywfCef66zOJ4i79YQMAUds8E7FvcQKA8dNDsoMKpcciFCwyklHmVGPBgrEe1H7bYNPmPWRX5PVkrLL5Ofo8445c8dsaPWIvKIV9hoTblqu1DpqJAV X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR08MB5325.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(396003)(376002)(136003)(346002)(39860400002)(366004)(230922051799003)(64100799003)(451199024)(186009)(1800799009)(6506007)(478600001)(2616005)(44144004)(6512007)(6666004)(6486002)(33964004)(36756003)(38100700002)(86362001)(66946007)(5660300002)(235185007)(44832011)(316002)(66476007)(66556008)(41300700001)(4743002)(83380400001)(2906002)(15650500001)(30864003)(26005)(8676002)(4326008)(6916009)(8936002)(2700100001); DIR:OUT; SFP:1101; X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB9581 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DU2PEPF0001E9C5.eurprd03.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 22db92b6-866c-4b8d-d125-08dbde9b565a X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: l/g052/2hF3PowVjuR+RpPEPJkgXvwFJXIqtuFT8jnOQqpDbRrhmk5jIidq7UyojVH8CW44TOB9MQOkYaL78mwi7nZVPkqiSc0iuDfg8MOqwJz06aAqgwBRQJNXInzbuigW2IDud/9X5NDwzeaVJna6Gk60kWJgSgRLH/E5qucQrtNWS9gJwAEGrU/3GZpwotviQXXBK98dxnBf5pPe5A7tAruITKY31VEtZFiz6zo99Nc2qrVHc4zPqlG7HHT8x+u3sybsNs+roQkPLJvvkfvlOhAjMqknVN3xPyffwE4/X5hACZJuynp4LEH/uJMXyI6K/fFvhnV4q6FDgb52bVEcpcIqX9r3RjPWCYUOfNZInY7V6BxlrmA91qN7vP5CEto7/mFOK4iK1Oz736LTKQh8F3oS71TZRHA32kDE62zDIubN6GJo2vhrUNEFNliR1utqfWPgm26WiUXFae5Np5Exui1Z4vl5Q9Na248FsLVZqVEcFXiYt2CpUINWtFTjn9cDStdNhyYNlSZoqsPEHfsjAbuQyMRGhW7YOd4UcUe8OCjnQuBYDWSnq3h5XrksW3i65LepitD6k2Viv4ojZ/paik76SHkTVyL0NQMho4/59TqCtNZKJ9x1nrWo6ORE/itqIjHkiygJFvMVpI6REG7pFPO1bqrvJbf1klQFiIasIdm66W/t31H5HX+joCaffvJwyQBNjJZURpm8jh1ckG+zKy0IlRWUWqxyKzD6ajXM05MqeaLSUs8rq1k3LP4SNBpJFfyO1exEoTZHvBtCS+Q== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230031)(4636009)(39860400002)(136003)(376002)(396003)(346002)(230922051799003)(1800799009)(82310400011)(64100799003)(186009)(451199024)(36840700001)(46966006)(40470700004)(82740400003)(81166007)(356005)(40480700001)(40460700003)(6486002)(6506007)(6512007)(6666004)(44144004)(33964004)(478600001)(70586007)(70206006)(36756003)(86362001)(26005)(107886003)(336012)(2616005)(4743002)(47076005)(83380400001)(8676002)(2906002)(235185007)(30864003)(6916009)(316002)(15650500001)(5660300002)(41300700001)(4326008)(44832011)(36860700001)(8936002)(2700100001); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Nov 2023 07:38:25.6892 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 600aff09-49aa-4eff-dc41-08dbde9b5c89 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DU2PEPF0001E9C5.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAXPR08MB6526 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, GIT_PATCH_0, KAM_DMARC_NONE, KAM_LOTSOFHASH, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Hi All, This splits the part of the function that does peeling for loops at exits to a different function. In this new function we also peel for early breaks. Peeling for early breaks works by redirecting all early break exits to a single "early break" block and combine them and the normal exit edge together later in a different block which then goes into the epilog preheader. This allows us to re-use all the existing code for IV updates, Additionally this also enables correct linking for multiple vector epilogues. flush_pending_stmts cannot be used in this scenario since it updates the PHI nodes in the order that they are in the exit destination blocks. This means they are in CFG visit order. With a single exit this doesn't matter but with multiple exits with different live values through the different exits the order usually does not line up. Additionally the vectorizer helper functions expect to be able to iterate over the nodes in the order that they occur in the loop header blocks. This is an invariant we must maintain. To do this we just inline the work flush_pending_stmts but maintain the order by using the header blocks to guide the work. Bootstrapped Regtested on aarch64-none-linux-gnu and no issues. Ok for master? Thanks, Tamar gcc/ChangeLog: * tree-vect-loop-manip.cc (vect_is_loop_exit_latch_pred): New. (slpeel_tree_duplicate_loop_for_vectorization): New. (slpeel_tree_duplicate_loop_to_edge_cfg): use it. * tree-vectorizer.h (is_loop_header_bb_p): Drop assert. (slpeel_tree_duplicate_loop_to_edge_cfg): Update signature. (vect_is_loop_exit_latch_pred): New. --- inline copy of patch -- diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index 43ca985c53ce58aa83fb9689a9ea9b20b207e0a8..6fbb5b80986fd657814b48eb009b52b094f331e6 100644 --- diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index 43ca985c53ce58aa83fb9689a9ea9b20b207e0a8..6fbb5b80986fd657814b48eb009b52b094f331e6 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -1444,6 +1444,151 @@ slpeel_duplicate_current_defs_from_edges (edge from, edge to) get_current_def (PHI_ARG_DEF_FROM_EDGE (from_phi, from))); } +/* Determine if the exit choosen by the loop vectorizer differs from the + natural loop exit. i.e. if the exit leads to the loop patch or not. + When this happens we need to flip the understanding of main and other + exits by peeling and IV updates. */ + +bool +vect_is_loop_exit_latch_pred (edge loop_exit, class loop *loop) +{ + return single_pred (loop->latch) == loop_exit->src; +} + +/* Perform peeling for when the peeled loop is placed after the original loop. + This maintains LCSSA and creates the appropriate blocks for multiple exit + vectorization. */ + +void static +slpeel_tree_duplicate_loop_for_vectorization (class loop *loop, edge loop_exit, + vec &loop_exits, edge e, + class loop *new_loop, + bool flow_loops, + basic_block new_preheader) +{ + bool multiple_exits_p = loop_exits.length () > 1; + basic_block main_loop_exit_block = new_preheader; + if (multiple_exits_p) + { + edge loop_entry = single_succ_edge (new_preheader); + new_preheader = split_edge (loop_entry); + } + + /* First create the empty phi nodes so that when we flush the + statements they can be filled in. However because there is no order + between the PHI nodes in the exits and the loop headers we need to + order them base on the order of the two headers. First record the new + phi nodes. Then redirect the edges and flush the changes. This writes out the new + SSA names. */ + for (auto exit : loop_exits) + { + basic_block dest + = exit == loop_exit ? main_loop_exit_block : new_preheader; + redirect_edge_and_branch (exit, dest); + } + + /* Copy the current loop LC PHI nodes between the original loop exit + block and the new loop header. This allows us to later split the + preheader block and still find the right LC nodes. */ + edge loop_entry = single_succ_edge (new_preheader); + hash_set lcssa_vars; + if (flow_loops) + for (auto gsi_from = gsi_start_phis (loop->header), + gsi_to = gsi_start_phis (new_loop->header); + !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); + gsi_next (&gsi_from), gsi_next (&gsi_to)) + { + gimple *from_phi = gsi_stmt (gsi_from); + gimple *to_phi = gsi_stmt (gsi_to); + tree new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, loop_latch_edge (loop)); + + /* In all cases, even in early break situations we're only + interested in the number of fully executed loop iters. As such + we discard any partially done iteration. So we simply propagate + the phi nodes from the latch to the merge block. */ + tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); + gphi *lcssa_phi = create_phi_node (new_res, main_loop_exit_block); + + /* Check if we haven't picked a different loop exit. If we have we + need to flip the LCSSA vars to prevent incorrect linking. */ + tree alt_arg = gimple_phi_result (from_phi); + if (!vect_is_loop_exit_latch_pred (loop_exit, loop)) + std::swap (new_arg, alt_arg); + + lcssa_vars.add (new_arg); + + /* Main loop exit should use the final iter value. */ + add_phi_arg (lcssa_phi, new_arg, loop_exit, UNKNOWN_LOCATION); + + /* All other exits use the previous iters. */ + if (multiple_exits_p) + { + tree alt_res = copy_ssa_name (alt_arg); + gphi *alt_lcssa_phi = create_phi_node (alt_res, new_preheader); + edge main_e = single_succ_edge (main_loop_exit_block); + for (edge e : loop_exits) + if (e != loop_exit) + { + add_phi_arg (alt_lcssa_phi, alt_arg, e, UNKNOWN_LOCATION); + SET_PHI_ARG_DEF (alt_lcssa_phi, main_e->dest_idx, new_res); + } + new_res = alt_res; /* Push it down to the new_loop header. */ + } + + adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); + } + + /* Copy over any live SSA vars that may not have been materialized in the + loops themselves but would be in the exit block. However when the live + value is not used inside the loop then we don't need to do this, if we do + then when we split the guard block the branch edge can end up containing the + wrong reference, particularly if it shares an edge with something that has + bypassed the loop. This is not something peeling can check so we need to + anticipate the usage of the live variable here. */ + auto exit_map = redirect_edge_var_map_vector (loop_exit); + if (exit_map) + for (auto vm : exit_map) + { + if (lcssa_vars.contains (vm.def) + || TREE_CODE (vm.def) != SSA_NAME) + continue; + + imm_use_iterator imm_iter; + use_operand_p use_p; + bool use_in_loop = false; + + FOR_EACH_IMM_USE_FAST (use_p, imm_iter, vm.def) + { + basic_block bb = gimple_bb (USE_STMT (use_p)); + if (flow_bb_inside_loop_p (loop, bb) + && !gimple_vuse (USE_STMT (use_p))) + { + use_in_loop = true; + break; + } + } + + if (!use_in_loop && SSA_VAR_P (vm.def)) + { + /* Do a final check to see if it's perhaps defined in the loop. + This mirrors the relevancy analysis's used_outside_scope. */ + if (virtual_operand_p (vm.def) + && (SSA_NAME_IS_DEFAULT_DEF (vm.def) + || !flow_bb_inside_loop_p (loop, + gimple_bb (SSA_NAME_DEF_STMT (vm.def))))) + continue; + } + + tree new_res = copy_ssa_name (vm.result); + gphi *lcssa_phi = create_phi_node (new_res, e->dest); + add_phi_arg (lcssa_phi, vm.def, loop_exit, vm.locus); + } + + /* Now clear all the redirect maps. */ + for (auto exit : loop_exits) + redirect_edge_var_map_clear (exit); +} + /* Given LOOP this function generates a new copy of it and puts it on E which is either the entry or exit of LOOP. If SCALAR_LOOP is non-NULL, assume LOOP and SCALAR_LOOP are equivalent and copy the @@ -1455,13 +1600,16 @@ slpeel_duplicate_current_defs_from_edges (edge from, edge to) copies remains the same. If UPDATED_DOMS is not NULL it is update with the list of basic blocks whoms - dominators were updated during the peeling. */ + dominators were updated during the peeling. When doing early break vectorization + then LOOP_VINFO needs to be provided and is used to keep track of any newly created + memory references that need to be updated should we decide to vectorize. */ class loop * slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, class loop *scalar_loop, edge scalar_exit, edge e, edge *new_e, - bool flow_loops) + bool flow_loops, + vec *updated_doms) { class loop *new_loop; basic_block *new_bbs, *bbs, *pbbs; @@ -1593,7 +1741,9 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, } auto loop_exits = get_loop_exit_edges (loop); + bool multiple_exits_p = loop_exits.length () > 1; auto_vec doms; + class loop *update_loop = NULL; if (at_exit) /* Add the loop copy at exit. */ { @@ -1603,103 +1753,9 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, flush_pending_stmts (new_exit); } - auto_vec new_phis; - hash_map new_phi_args; - /* First create the empty phi nodes so that when we flush the - statements they can be filled in. However because there is no order - between the PHI nodes in the exits and the loop headers we need to - order them base on the order of the two headers. First record the new - phi nodes. */ - for (auto gsi_from = gsi_start_phis (scalar_exit->dest); - !gsi_end_p (gsi_from); gsi_next (&gsi_from)) - { - gimple *from_phi = gsi_stmt (gsi_from); - tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); - gphi *res = create_phi_node (new_res, new_preheader); - new_phis.safe_push (res); - } - - /* Then redirect the edges and flush the changes. This writes out the new - SSA names. */ - for (edge exit : loop_exits) - { - edge temp_e = redirect_edge_and_branch (exit, new_preheader); - flush_pending_stmts (temp_e); - } - /* Record the new SSA names in the cache so that we can skip materializing - them again when we fill in the rest of the LCSSA variables. */ - for (auto phi : new_phis) - { - tree new_arg = gimple_phi_arg (phi, 0)->def; - - if (!SSA_VAR_P (new_arg)) - continue; - /* If the PHI MEM node dominates the loop then we shouldn't create - a new LC-SSSA PHI for it in the intermediate block. */ - /* A MEM phi that consitutes a new DEF for the vUSE chain can either - be a .VDEF or a PHI that operates on MEM. And said definition - must not be inside the main loop. Or we must be a parameter. - In the last two cases we may remove a non-MEM PHI node, but since - they dominate both loops the removal is unlikely to cause trouble - as the exits must already be using them. */ - if (virtual_operand_p (new_arg) - && (SSA_NAME_IS_DEFAULT_DEF (new_arg) - || !flow_bb_inside_loop_p (loop, - gimple_bb (SSA_NAME_DEF_STMT (new_arg))))) - { - auto gsi = gsi_for_stmt (phi); - remove_phi_node (&gsi, true); - continue; - } - new_phi_args.put (new_arg, gimple_phi_result (phi)); - - if (TREE_CODE (new_arg) != SSA_NAME) - continue; - /* If the PHI node dominates the loop then we shouldn't create - a new LC-SSSA PHI for it in the intermediate block. Unless the - the loop has been versioned. If it has then we need the PHI - node such that later when the loop guard is added the original - dominating PHI can be found. */ - basic_block def_bb = gimple_bb (SSA_NAME_DEF_STMT (new_arg)); - if (loop == scalar_loop - && (!def_bb || !flow_bb_inside_loop_p (loop, def_bb))) - { - auto gsi = gsi_for_stmt (phi); - remove_phi_node (&gsi, true); - } - } - - /* Copy the current loop LC PHI nodes between the original loop exit - block and the new loop header. This allows us to later split the - preheader block and still find the right LC nodes. */ - edge loop_entry = single_succ_edge (new_preheader); - if (flow_loops) - for (auto gsi_from = gsi_start_phis (loop->header), - gsi_to = gsi_start_phis (new_loop->header); - !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); - gsi_next (&gsi_from), gsi_next (&gsi_to)) - { - gimple *from_phi = gsi_stmt (gsi_from); - gimple *to_phi = gsi_stmt (gsi_to); - tree new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, - loop_latch_edge (loop)); - - /* Check if we've already created a new phi node during edge - redirection. If we have, only propagate the value downwards. */ - if (tree *res = new_phi_args.get (new_arg)) - { - adjust_phi_and_debug_stmts (to_phi, loop_entry, *res); - continue; - } - - tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); - gphi *lcssa_phi = create_phi_node (new_res, new_preheader); - - /* Main loop exit should use the final iter value. */ - add_phi_arg (lcssa_phi, new_arg, loop_exit, UNKNOWN_LOCATION); - - adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); - } + slpeel_tree_duplicate_loop_for_vectorization (loop, loop_exit, loop_exits, + e, new_loop, flow_loops, + new_preheader); set_immediate_dominator (CDI_DOMINATORS, new_preheader, e->src); @@ -1713,6 +1769,21 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, delete_basic_block (preheader); set_immediate_dominator (CDI_DOMINATORS, scalar_loop->header, loop_preheader_edge (scalar_loop)->src); + + /* Finally after wiring the new epilogue we need to update its main exit + to the original function exit we recorded. Other exits are already + correct. */ + if (multiple_exits_p) + { + update_loop = new_loop; + for (edge e : get_loop_exit_edges (loop)) + doms.safe_push (e->dest); + doms.safe_push (exit_dest); + + /* Likely a fall-through edge, so update if needed. */ + if (single_succ_p (exit_dest)) + doms.safe_push (single_succ (exit_dest)); + } } else /* Add the copy at entry. */ { @@ -1760,6 +1831,34 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, delete_basic_block (new_preheader); set_immediate_dominator (CDI_DOMINATORS, new_loop->header, loop_preheader_edge (new_loop)->src); + + if (multiple_exits_p) + update_loop = loop; + } + + if (multiple_exits_p) + { + for (edge e : get_loop_exit_edges (update_loop)) + { + edge ex; + edge_iterator ei; + FOR_EACH_EDGE (ex, ei, e->dest->succs) + { + /* Find the first non-fallthrough block as fall-throughs can't + dominate other blocks. */ + if (single_succ_p (ex->dest)) + { + doms.safe_push (ex->dest); + ex = single_succ_edge (ex->dest); + } + doms.safe_push (ex->dest); + } + doms.safe_push (e->dest); + } + + iterate_fix_dominators (CDI_DOMINATORS, doms, false); + if (updated_doms) + updated_doms->safe_splice (doms); } free (new_bbs); diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 1418913d2c308b0cf78352e29dc9958746fb9c94..d8b532c4b8ca92a856368a686598859fab9d40e9 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -1821,7 +1821,7 @@ is_loop_header_bb_p (basic_block bb) { if (bb == (bb->loop_father)->header) return true; - gcc_checking_assert (EDGE_COUNT (bb->preds) == 1); + return false; } @@ -2212,7 +2212,8 @@ extern bool slpeel_can_duplicate_loop_p (const class loop *, const_edge, const_edge); class loop *slpeel_tree_duplicate_loop_to_edge_cfg (class loop *, edge, class loop *, edge, - edge, edge *, bool = true); + edge, edge *, bool = true, + vec * = NULL); class loop *vect_loop_versioning (loop_vec_info, gimple *); extern class loop *vect_do_peeling (loop_vec_info, tree, tree, tree *, tree *, tree *, int, bool, bool, @@ -2223,6 +2224,7 @@ extern dump_user_location_t find_loop_location (class loop *); extern bool vect_can_advance_ivs_p (loop_vec_info); extern void vect_update_inits_of_drs (loop_vec_info, tree, tree_code); extern edge vec_init_loop_exit_info (class loop *); +extern bool vect_is_loop_exit_latch_pred (edge, class loop *); /* In tree-vect-stmts.cc. */ extern tree get_related_vectype_for_scalar_type (machine_mode, tree, --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -1444,6 +1444,151 @@ slpeel_duplicate_current_defs_from_edges (edge from, edge to) get_current_def (PHI_ARG_DEF_FROM_EDGE (from_phi, from))); } +/* Determine if the exit choosen by the loop vectorizer differs from the + natural loop exit. i.e. if the exit leads to the loop patch or not. + When this happens we need to flip the understanding of main and other + exits by peeling and IV updates. */ + +bool +vect_is_loop_exit_latch_pred (edge loop_exit, class loop *loop) +{ + return single_pred (loop->latch) == loop_exit->src; +} + +/* Perform peeling for when the peeled loop is placed after the original loop. + This maintains LCSSA and creates the appropriate blocks for multiple exit + vectorization. */ + +void static +slpeel_tree_duplicate_loop_for_vectorization (class loop *loop, edge loop_exit, + vec &loop_exits, edge e, + class loop *new_loop, + bool flow_loops, + basic_block new_preheader) +{ + bool multiple_exits_p = loop_exits.length () > 1; + basic_block main_loop_exit_block = new_preheader; + if (multiple_exits_p) + { + edge loop_entry = single_succ_edge (new_preheader); + new_preheader = split_edge (loop_entry); + } + + /* First create the empty phi nodes so that when we flush the + statements they can be filled in. However because there is no order + between the PHI nodes in the exits and the loop headers we need to + order them base on the order of the two headers. First record the new + phi nodes. Then redirect the edges and flush the changes. This writes out the new + SSA names. */ + for (auto exit : loop_exits) + { + basic_block dest + = exit == loop_exit ? main_loop_exit_block : new_preheader; + redirect_edge_and_branch (exit, dest); + } + + /* Copy the current loop LC PHI nodes between the original loop exit + block and the new loop header. This allows us to later split the + preheader block and still find the right LC nodes. */ + edge loop_entry = single_succ_edge (new_preheader); + hash_set lcssa_vars; + if (flow_loops) + for (auto gsi_from = gsi_start_phis (loop->header), + gsi_to = gsi_start_phis (new_loop->header); + !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); + gsi_next (&gsi_from), gsi_next (&gsi_to)) + { + gimple *from_phi = gsi_stmt (gsi_from); + gimple *to_phi = gsi_stmt (gsi_to); + tree new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, loop_latch_edge (loop)); + + /* In all cases, even in early break situations we're only + interested in the number of fully executed loop iters. As such + we discard any partially done iteration. So we simply propagate + the phi nodes from the latch to the merge block. */ + tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); + gphi *lcssa_phi = create_phi_node (new_res, main_loop_exit_block); + + /* Check if we haven't picked a different loop exit. If we have we + need to flip the LCSSA vars to prevent incorrect linking. */ + tree alt_arg = gimple_phi_result (from_phi); + if (!vect_is_loop_exit_latch_pred (loop_exit, loop)) + std::swap (new_arg, alt_arg); + + lcssa_vars.add (new_arg); + + /* Main loop exit should use the final iter value. */ + add_phi_arg (lcssa_phi, new_arg, loop_exit, UNKNOWN_LOCATION); + + /* All other exits use the previous iters. */ + if (multiple_exits_p) + { + tree alt_res = copy_ssa_name (alt_arg); + gphi *alt_lcssa_phi = create_phi_node (alt_res, new_preheader); + edge main_e = single_succ_edge (main_loop_exit_block); + for (edge e : loop_exits) + if (e != loop_exit) + { + add_phi_arg (alt_lcssa_phi, alt_arg, e, UNKNOWN_LOCATION); + SET_PHI_ARG_DEF (alt_lcssa_phi, main_e->dest_idx, new_res); + } + new_res = alt_res; /* Push it down to the new_loop header. */ + } + + adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); + } + + /* Copy over any live SSA vars that may not have been materialized in the + loops themselves but would be in the exit block. However when the live + value is not used inside the loop then we don't need to do this, if we do + then when we split the guard block the branch edge can end up containing the + wrong reference, particularly if it shares an edge with something that has + bypassed the loop. This is not something peeling can check so we need to + anticipate the usage of the live variable here. */ + auto exit_map = redirect_edge_var_map_vector (loop_exit); + if (exit_map) + for (auto vm : exit_map) + { + if (lcssa_vars.contains (vm.def) + || TREE_CODE (vm.def) != SSA_NAME) + continue; + + imm_use_iterator imm_iter; + use_operand_p use_p; + bool use_in_loop = false; + + FOR_EACH_IMM_USE_FAST (use_p, imm_iter, vm.def) + { + basic_block bb = gimple_bb (USE_STMT (use_p)); + if (flow_bb_inside_loop_p (loop, bb) + && !gimple_vuse (USE_STMT (use_p))) + { + use_in_loop = true; + break; + } + } + + if (!use_in_loop && SSA_VAR_P (vm.def)) + { + /* Do a final check to see if it's perhaps defined in the loop. + This mirrors the relevancy analysis's used_outside_scope. */ + if (virtual_operand_p (vm.def) + && (SSA_NAME_IS_DEFAULT_DEF (vm.def) + || !flow_bb_inside_loop_p (loop, + gimple_bb (SSA_NAME_DEF_STMT (vm.def))))) + continue; + } + + tree new_res = copy_ssa_name (vm.result); + gphi *lcssa_phi = create_phi_node (new_res, e->dest); + add_phi_arg (lcssa_phi, vm.def, loop_exit, vm.locus); + } + + /* Now clear all the redirect maps. */ + for (auto exit : loop_exits) + redirect_edge_var_map_clear (exit); +} + /* Given LOOP this function generates a new copy of it and puts it on E which is either the entry or exit of LOOP. If SCALAR_LOOP is non-NULL, assume LOOP and SCALAR_LOOP are equivalent and copy the @@ -1455,13 +1600,16 @@ slpeel_duplicate_current_defs_from_edges (edge from, edge to) copies remains the same. If UPDATED_DOMS is not NULL it is update with the list of basic blocks whoms - dominators were updated during the peeling. */ + dominators were updated during the peeling. When doing early break vectorization + then LOOP_VINFO needs to be provided and is used to keep track of any newly created + memory references that need to be updated should we decide to vectorize. */ class loop * slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, class loop *scalar_loop, edge scalar_exit, edge e, edge *new_e, - bool flow_loops) + bool flow_loops, + vec *updated_doms) { class loop *new_loop; basic_block *new_bbs, *bbs, *pbbs; @@ -1593,7 +1741,9 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, } auto loop_exits = get_loop_exit_edges (loop); + bool multiple_exits_p = loop_exits.length () > 1; auto_vec doms; + class loop *update_loop = NULL; if (at_exit) /* Add the loop copy at exit. */ { @@ -1603,103 +1753,9 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, flush_pending_stmts (new_exit); } - auto_vec new_phis; - hash_map new_phi_args; - /* First create the empty phi nodes so that when we flush the - statements they can be filled in. However because there is no order - between the PHI nodes in the exits and the loop headers we need to - order them base on the order of the two headers. First record the new - phi nodes. */ - for (auto gsi_from = gsi_start_phis (scalar_exit->dest); - !gsi_end_p (gsi_from); gsi_next (&gsi_from)) - { - gimple *from_phi = gsi_stmt (gsi_from); - tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); - gphi *res = create_phi_node (new_res, new_preheader); - new_phis.safe_push (res); - } - - /* Then redirect the edges and flush the changes. This writes out the new - SSA names. */ - for (edge exit : loop_exits) - { - edge temp_e = redirect_edge_and_branch (exit, new_preheader); - flush_pending_stmts (temp_e); - } - /* Record the new SSA names in the cache so that we can skip materializing - them again when we fill in the rest of the LCSSA variables. */ - for (auto phi : new_phis) - { - tree new_arg = gimple_phi_arg (phi, 0)->def; - - if (!SSA_VAR_P (new_arg)) - continue; - /* If the PHI MEM node dominates the loop then we shouldn't create - a new LC-SSSA PHI for it in the intermediate block. */ - /* A MEM phi that consitutes a new DEF for the vUSE chain can either - be a .VDEF or a PHI that operates on MEM. And said definition - must not be inside the main loop. Or we must be a parameter. - In the last two cases we may remove a non-MEM PHI node, but since - they dominate both loops the removal is unlikely to cause trouble - as the exits must already be using them. */ - if (virtual_operand_p (new_arg) - && (SSA_NAME_IS_DEFAULT_DEF (new_arg) - || !flow_bb_inside_loop_p (loop, - gimple_bb (SSA_NAME_DEF_STMT (new_arg))))) - { - auto gsi = gsi_for_stmt (phi); - remove_phi_node (&gsi, true); - continue; - } - new_phi_args.put (new_arg, gimple_phi_result (phi)); - - if (TREE_CODE (new_arg) != SSA_NAME) - continue; - /* If the PHI node dominates the loop then we shouldn't create - a new LC-SSSA PHI for it in the intermediate block. Unless the - the loop has been versioned. If it has then we need the PHI - node such that later when the loop guard is added the original - dominating PHI can be found. */ - basic_block def_bb = gimple_bb (SSA_NAME_DEF_STMT (new_arg)); - if (loop == scalar_loop - && (!def_bb || !flow_bb_inside_loop_p (loop, def_bb))) - { - auto gsi = gsi_for_stmt (phi); - remove_phi_node (&gsi, true); - } - } - - /* Copy the current loop LC PHI nodes between the original loop exit - block and the new loop header. This allows us to later split the - preheader block and still find the right LC nodes. */ - edge loop_entry = single_succ_edge (new_preheader); - if (flow_loops) - for (auto gsi_from = gsi_start_phis (loop->header), - gsi_to = gsi_start_phis (new_loop->header); - !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); - gsi_next (&gsi_from), gsi_next (&gsi_to)) - { - gimple *from_phi = gsi_stmt (gsi_from); - gimple *to_phi = gsi_stmt (gsi_to); - tree new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, - loop_latch_edge (loop)); - - /* Check if we've already created a new phi node during edge - redirection. If we have, only propagate the value downwards. */ - if (tree *res = new_phi_args.get (new_arg)) - { - adjust_phi_and_debug_stmts (to_phi, loop_entry, *res); - continue; - } - - tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); - gphi *lcssa_phi = create_phi_node (new_res, new_preheader); - - /* Main loop exit should use the final iter value. */ - add_phi_arg (lcssa_phi, new_arg, loop_exit, UNKNOWN_LOCATION); - - adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); - } + slpeel_tree_duplicate_loop_for_vectorization (loop, loop_exit, loop_exits, + e, new_loop, flow_loops, + new_preheader); set_immediate_dominator (CDI_DOMINATORS, new_preheader, e->src); @@ -1713,6 +1769,21 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, delete_basic_block (preheader); set_immediate_dominator (CDI_DOMINATORS, scalar_loop->header, loop_preheader_edge (scalar_loop)->src); + + /* Finally after wiring the new epilogue we need to update its main exit + to the original function exit we recorded. Other exits are already + correct. */ + if (multiple_exits_p) + { + update_loop = new_loop; + for (edge e : get_loop_exit_edges (loop)) + doms.safe_push (e->dest); + doms.safe_push (exit_dest); + + /* Likely a fall-through edge, so update if needed. */ + if (single_succ_p (exit_dest)) + doms.safe_push (single_succ (exit_dest)); + } } else /* Add the copy at entry. */ { @@ -1760,6 +1831,34 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, delete_basic_block (new_preheader); set_immediate_dominator (CDI_DOMINATORS, new_loop->header, loop_preheader_edge (new_loop)->src); + + if (multiple_exits_p) + update_loop = loop; + } + + if (multiple_exits_p) + { + for (edge e : get_loop_exit_edges (update_loop)) + { + edge ex; + edge_iterator ei; + FOR_EACH_EDGE (ex, ei, e->dest->succs) + { + /* Find the first non-fallthrough block as fall-throughs can't + dominate other blocks. */ + if (single_succ_p (ex->dest)) + { + doms.safe_push (ex->dest); + ex = single_succ_edge (ex->dest); + } + doms.safe_push (ex->dest); + } + doms.safe_push (e->dest); + } + + iterate_fix_dominators (CDI_DOMINATORS, doms, false); + if (updated_doms) + updated_doms->safe_splice (doms); } free (new_bbs); diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 1418913d2c308b0cf78352e29dc9958746fb9c94..d8b532c4b8ca92a856368a686598859fab9d40e9 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -1821,7 +1821,7 @@ is_loop_header_bb_p (basic_block bb) { if (bb == (bb->loop_father)->header) return true; - gcc_checking_assert (EDGE_COUNT (bb->preds) == 1); + return false; } @@ -2212,7 +2212,8 @@ extern bool slpeel_can_duplicate_loop_p (const class loop *, const_edge, const_edge); class loop *slpeel_tree_duplicate_loop_to_edge_cfg (class loop *, edge, class loop *, edge, - edge, edge *, bool = true); + edge, edge *, bool = true, + vec * = NULL); class loop *vect_loop_versioning (loop_vec_info, gimple *); extern class loop *vect_do_peeling (loop_vec_info, tree, tree, tree *, tree *, tree *, int, bool, bool, @@ -2223,6 +2224,7 @@ extern dump_user_location_t find_loop_location (class loop *); extern bool vect_can_advance_ivs_p (loop_vec_info); extern void vect_update_inits_of_drs (loop_vec_info, tree, tree_code); extern edge vec_init_loop_exit_info (class loop *); +extern bool vect_is_loop_exit_latch_pred (edge, class loop *); /* In tree-vect-stmts.cc. */ extern tree get_related_vectype_for_scalar_type (machine_mode, tree,