From patchwork Mon Nov 6 07:40:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tamar Christina X-Patchwork-Id: 1859832 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=armh.onmicrosoft.com header.i=@armh.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-armh-onmicrosoft-com header.b=zGLECUtI; dkim=pass (1024-bit key) header.d=armh.onmicrosoft.com header.i=@armh.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-armh-onmicrosoft-com header.b=zGLECUtI; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SP3Dd4Xsjz1yQY for ; Mon, 6 Nov 2023 18:40:49 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8D5DD386193F for ; Mon, 6 Nov 2023 07:40:44 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04on2040.outbound.protection.outlook.com [40.107.7.40]) by sourceware.org (Postfix) with ESMTPS id ADA993857717 for ; Mon, 6 Nov 2023 07:40:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org ADA993857717 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org ADA993857717 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=40.107.7.40 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1699256432; cv=pass; b=tL5vyPrI+hx7Owmeuz3VZ0cYsiWonWfbjTnHkHWbF4Gx2PrMxhzrIbq+w93TMtDSyd9xg99OOA8ziCEBdbV9l0dvzr/+ppQwmKUpbp26xyToU6AG5jGjztbsalJdgcXxb1OkXZrhiavb4xonI9QQC/DQ1x62kuhJAn39Rw43gBQ= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1699256432; c=relaxed/simple; bh=6tbshpWQ9g1hOw1XfdAGQwTZBfMbIWx65UsNDKXH00c=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=Isp8ajyepjqtbJm/NQerlO27XBneSH5vzLc6tirPEQO5VO4tCzqckDKKKAUJkOn3vL9H3uWJt8VGz04RSJQU4JpoqbuxVmU/3hHrlT1ATb2SK3CnbMmM5NNe5hAAUJqAOQi48tT9RxrhNu7zJTIdSdhifq7JOOITHfXtvVLz2Ig= ARC-Authentication-Results: i=3; server2.sourceware.org ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=lAb7oMURnMdrhWwQKHJIoti/fxbPXYFXTV0460FhjkSDjSz8r3UgllWBDqQAppNR+d2qvAez1v0GlWQTDt17NXtmIFF5RrKxJHJPfBdHzLw+dz/2VLb5GEi7oJHsK6OYBfnQmAHcoNKMwk1E5ZlQYHq6B6YGjvQyf7fV+etqHbcGgJe1BUbn+zEBOJpqGSxkAxBYnZ3h6/hWMywwhqXhEhkxRTfksQGeM106xZIJtmFmzNufPlbRYftX7ZqTrX3Fh2Q7Ra2U5KBRXE8/KkDiA//Bebu/R7klEtBFlYhQbJ0GU8fe0DQHo4y5whtRV0dodSqp26+AMYhLNHGLlLx/mQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Qxl83AR98eelgy8HBXksJKU0rKBbNx2ShIdYqtDbLzg=; b=b/KvuCA7z0bGhNU230+Vn8o3aVtfmZWUwRWFDZOm8QB+WnYuLk+YMA8zReJeTgDSe3fareQowFUI5DmilPRaZyl3se5va+gGBbwOxHGxB9BXggqfGqTkEjAVtiQdH7MaEcWmFjosstDXHNLEt6abhZeFw/Mg0h5x9RKDHabIhzO1jR5cDFSFSDZ5zcPeiNhTHt8ddVSzq9ULMdrQYncWA9DrQIJzMWkP0Ap62w1fju/MKfR7saeQnZccO5e+AONdXamfjh+LbR6OU4oFm15JtVC9epjybXPb96/VLl8sY2agUUpld/1uTc/E6QUOv5qsJ3FyMD9N4ZkfrPwnKRunPA== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Qxl83AR98eelgy8HBXksJKU0rKBbNx2ShIdYqtDbLzg=; b=zGLECUtIty2hs+mEevNFwiW+kmaOvBnUSFrlQQHNiH1lmFFOaSST0tY63J4v1CH3eyywN6jhB5Vw4IJaHsiGTEAyUyS45WGq49I65I3hXlw45lbws6f8kVUzYjVAnss+dtjWjNtYqL9hRZQlDEvJTONtCzmyACpwTbIxcW5DFFo= Received: from DUZPR01CA0065.eurprd01.prod.exchangelabs.com (2603:10a6:10:3c2::8) by AS4PR08MB7735.eurprd08.prod.outlook.com (2603:10a6:20b:512::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.27; Mon, 6 Nov 2023 07:40:26 +0000 Received: from DU6PEPF0000B621.eurprd02.prod.outlook.com (2603:10a6:10:3c2:cafe::d3) by DUZPR01CA0065.outlook.office365.com (2603:10a6:10:3c2::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28 via Frontend Transport; Mon, 6 Nov 2023 07:40:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DU6PEPF0000B621.mail.protection.outlook.com (10.167.8.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6977.16 via Frontend Transport; Mon, 6 Nov 2023 07:40:25 +0000 Received: ("Tessian outbound 20615a7e7970:v228"); Mon, 06 Nov 2023 07:40:25 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 7b30dd7d4c61451e X-CR-MTA-TID: 64aa7808 Received: from 214080e359b5.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 857409DD-D098-4D01-91D8-82547B30D564.1; Mon, 06 Nov 2023 07:40:19 +0000 Received: from EUR04-VI1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 214080e359b5.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 06 Nov 2023 07:40:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E8D+LeIIp4N1UzZEdfRjLAjni58avbuDE/KgmvHTJH37V6TxMVXSrGa5TgBBDoL7M1iybCi3JMc3wvg4wCCoq4rSlv9Uub8SNtNAqEIrLo6yXy+m7IQ5l9vhmqTNUGQC7woeYwcqlgoSMkYl0zOr8J5fVm/vGwv+JP2JkDiUfOyctpRhBRSTAYMe0J702ElWt29BV2E+/1PnnPBgaJUxawUSO7wcEi/uOj6XWWZ3UpopS2vPb2x8Whlrp5Wtl7x04Q1yqU70EQ6W5M4zdS3EMQbWuwFYIWlziptYUrylSwFypVDao7/JygFx5u4b2ZppH8MSn8VqzAk3QgqzVfsMcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Qxl83AR98eelgy8HBXksJKU0rKBbNx2ShIdYqtDbLzg=; b=ihlIhQjh3vscpm+cvYHcwboywvNK890zfMumiuJLo7HjTODnsMmAaxGfLbkvOqGaAZbsAW/7JRpcyztVeLoXlRwGF2IwMAhAaAsI3IdfThLhIJEVARf8KbdV3Belmmyw5Vp7gcV8fPLJGQOOq/O8+jBdM2MM+TZKremjEDnA4ogFvvk525E7RcxMW7a4g2Mu2kYo7c/QT02kka39AHBoCeCBTBCMI3jh3fB9yHeGxkx8+0Ib4oRkc3KA96JeZuz0L+iXGZ5gSdXZW38wbUp6jGNE8P2iDYc6iZC8qZXbJUUxDGgzX5E22i0T1wd5y2BiaxkYLYRD6U21qFnIHBoZww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Qxl83AR98eelgy8HBXksJKU0rKBbNx2ShIdYqtDbLzg=; b=zGLECUtIty2hs+mEevNFwiW+kmaOvBnUSFrlQQHNiH1lmFFOaSST0tY63J4v1CH3eyywN6jhB5Vw4IJaHsiGTEAyUyS45WGq49I65I3hXlw45lbws6f8kVUzYjVAnss+dtjWjNtYqL9hRZQlDEvJTONtCzmyACpwTbIxcW5DFFo= Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; Received: from VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) by DBBPR08MB10676.eurprd08.prod.outlook.com (2603:10a6:10:530::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28; Mon, 6 Nov 2023 07:40:16 +0000 Received: from VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::26aa:efdd:a74a:27d0]) by VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::26aa:efdd:a74a:27d0%5]) with mapi id 15.20.6954.028; Mon, 6 Nov 2023 07:40:16 +0000 Date: Mon, 6 Nov 2023 07:40:13 +0000 From: Tamar Christina To: gcc-patches@gcc.gnu.org Cc: nd@arm.com, rguenther@suse.de, jlaw@ventanamicro.com Subject: [PATCH 11/21]middle-end: wire through peeling changes and dominator updates after guard edge split Message-ID: Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO3P123CA0015.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:ba::20) To VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) MIME-Version: 1.0 X-MS-TrafficTypeDiagnostic: VI1PR08MB5325:EE_|DBBPR08MB10676:EE_|DU6PEPF0000B621:EE_|AS4PR08MB7735:EE_ X-MS-Office365-Filtering-Correlation-Id: 77c12a0d-4881-47d1-c871-08dbde9ba42d x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: U65UqWvmjk5wVZs9O3EwjpfBjxEqzBBXMKyZylBUJbm4xmtGl3Yt1XtkOdrsP6hchChJj/dcnf2QzXaqeuD21BCppIRDhjtxL0ouuvZpdr4NHGCxsV7mPn7R1+XsJ7Yy+CpnjtPOB2V705RtoEHgG7aX35nLTJQNwlcRWpEfFeKyB7LeulV3tOosd90kGCFy9qM8RJ+nNW6piMRnL0Z4tLbK00uuDFiLanSlzUx4cwe4jZgDHx3brFvRlx5/Y0x5b8eDGDL1crlsPTld7kCaZ/m7I2IffSMnX1E8qm6LqMCuRNpm3Ebfk3Z0dwGWJh9ya0AB/ANSPmMGJzqUllCx5sxixQy3RcjC74K4SryAtEtQs91l6p1pNMYC2O2tTgZi6NhVasYoPofKW/1jd3B9blokdvA8trUz5mNV4tfKsMgY7zLXLkT1sSEOSz2+Mlxxj4qrnysWyS+EPn7169P1RLhNNbjGCAgokRHT+dlfwcf0Uf8G5+RG1dUPHvWfMK8MD5JDIwZw/6vF7Mn+OU5Ykz7Eu4l0KS2tVrBAM5e/aup+l5Pv65bthYNAJjslfcitCylgiV/GI/BKHMGDSqbROyBfETO5VcUb3Dlrj3pwTTQuWXPEvN7XtB86zF7nWJbY X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR08MB5325.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(396003)(376002)(366004)(346002)(39860400002)(230922051799003)(186009)(64100799003)(1800799009)(451199024)(6916009)(4326008)(8676002)(8936002)(41300700001)(36756003)(2906002)(235185007)(38100700002)(5660300002)(6486002)(66556008)(6512007)(316002)(66476007)(86362001)(44832011)(15650500001)(4743002)(66946007)(83380400001)(26005)(2616005)(6666004)(6506007)(478600001)(33964004)(44144004)(2700100001); DIR:OUT; SFP:1101; X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB10676 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DU6PEPF0000B621.eurprd02.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: eaffacca-f5fc-4df7-ada2-08dbde9b9eaf X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fdvc69u07+r2Wy+H2kXc3yCa1rDvGjw4rmlXZ8Z9Vg7f7fZM5/PEKsxHg5fCq1An5vntxDp43FLeTZjx4QIHi536BIU23TdEB2TlcQQHm6SrawPaeouRbhaaSfdwF4sV5NAzsVCrLmCNFeEXXPw0c/3VJKHhwqYCSGaM5WvlaHT+qQY0wuSXt7SAOIYP9x8mhf3hcY60ZTZIQxJl1EN6tqamJfyo/h92Ns22Y+7oCG3QCNCgMu/gipmSl5J6z1fTvIqihShPNMUsLVIjsgCdPkQH1o7jaM/GQotxV3RT9a1kVFExnc3jlEpYiyhQQS+j2Xnub79P0PmDkprHc+z4lSI5foUX7OAb9BAAcdVjP6sQcApljNVrQfJRnKaF1EmqL7s2gBt4O1m/R6rlxYO5r9/nQMy7CqFk/nJSKaF2OY7/t3HxkpLNxGzeX0cePyhCDVdlFf32s+7B1URMKFVmy4naMsRj+oaPJb8JcTw4Gpe3Fnwg5v85U+RxQpXxV+kiNm/vVrkFTcAisWgms5vbLkGLfCkyEMgR/ksiy1IZfqh4lykQ0gZFQhDBcIQLjwT3hAUFX/ztZEWP8Cmf8EiXm3PyOQWQ9+LQte9OT+4kRqerOMYQGi2l1t+OQCBZocaR7blOBGSUV8zbUQQNV+Z1h9/REXxHS0wGUbAUJ980IMOfkemzpWRWMRAQZtv+1y9SNyti42p1XFOqpPOO3vk7ztQuauxLGwYIjlLnr2EC/b8/4l64hKqmHEkzZXHyJsV/wabQ2PeCdW19057JOSzU2g== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230031)(4636009)(136003)(376002)(39860400002)(346002)(396003)(230922051799003)(82310400011)(186009)(1800799009)(451199024)(64100799003)(40470700004)(36840700001)(46966006)(33964004)(107886003)(2616005)(6666004)(4743002)(336012)(44144004)(6506007)(6512007)(41300700001)(26005)(47076005)(83380400001)(8936002)(4326008)(8676002)(15650500001)(2906002)(36860700001)(44832011)(5660300002)(235185007)(6486002)(6916009)(478600001)(316002)(70586007)(70206006)(356005)(81166007)(82740400003)(36756003)(86362001)(40460700003)(40480700001)(2700100001); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Nov 2023 07:40:25.8818 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 77c12a0d-4881-47d1-c871-08dbde9ba42d X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DU6PEPF0000B621.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR08MB7735 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, GIT_PATCH_0, KAM_DMARC_NONE, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Hi All, This wires through the final bits to support adding the guard block between the loop and epilog. For an "inverted loop", i.e. one where an early exit was chosen as the main exit then we can never skip the scalar loop since we know we have side effects to still perform. For those cases we always restart the scalar loop regardless of which exit is taken. When we add the guard we also need to update the dominators calculated by the peeling code as the bypass edge changes the dominators. Bootstrapped Regtested on aarch64-none-linux-gnu and no issues. Ok for master? Thanks, Tamar gcc/ChangeLog: * tree-vect-loop-manip.cc (vect_do_peeling): --- inline copy of patch -- diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index abd905b78f3661f80168c3866d7c3e68a9c15521..eef2bb50c1505f5cf802d5d80300affc2cbe69f6 100644 --- diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index abd905b78f3661f80168c3866d7c3e68a9c15521..eef2bb50c1505f5cf802d5d80300affc2cbe69f6 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -3512,11 +3512,14 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, epilog = vect_epilogues ? get_loop_copy (loop) : scalar_loop; edge epilog_e = vect_epilogues ? e : scalar_e; edge new_epilog_e = NULL; - epilog = slpeel_tree_duplicate_loop_to_edge_cfg (loop, e, epilog, - epilog_e, e, - &new_epilog_e); + auto_vec doms; + epilog + = slpeel_tree_duplicate_loop_to_edge_cfg (loop, e, epilog, epilog_e, e, + &new_epilog_e, true, &doms); + LOOP_VINFO_EPILOGUE_IV_EXIT (loop_vinfo) = new_epilog_e; gcc_assert (epilog); + gcc_assert (new_epilog_e); epilog->force_vectorize = false; bb_before_epilog = loop_preheader_edge (epilog)->src; @@ -3610,10 +3613,16 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, } - if (skip_epilog) + if (skip_epilog || LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) { - guard_cond = fold_build2 (EQ_EXPR, boolean_type_node, + /* For the case where a different exit was chosen we must execute + the scalar loop with the remaining iterations. */ + if (inversed_iv) + guard_cond = boolean_false_node; + else + guard_cond = fold_build2 (EQ_EXPR, boolean_type_node, niters, niters_vector_mult_vf); + guard_bb = LOOP_VINFO_IV_EXIT (loop_vinfo)->dest; edge epilog_e = LOOP_VINFO_EPILOGUE_IV_EXIT (loop_vinfo); guard_to = split_edge (epilog_e); @@ -3621,11 +3630,13 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, skip_vector ? anchor : guard_bb, prob_epilog.invert (), irred_flag); + doms.safe_push (guard_to); if (vect_epilogues) epilogue_vinfo->skip_this_loop_edge = guard_e; edge main_iv = LOOP_VINFO_IV_EXIT (loop_vinfo); slpeel_update_phi_nodes_for_guard2 (loop, epilog, main_iv, guard_e, epilog_e); + /* Only need to handle basic block before epilog loop if it's not the guard_bb, which is the case when skip_vector is true. */ if (guard_bb != bb_before_epilog) @@ -3637,6 +3648,10 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, scale_loop_profile (epilog, prob_epilog, -1); } + /* Recalculate the dominators after adding the guard edge. */ + if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) + iterate_fix_dominators (CDI_DOMINATORS, doms, false); + unsigned HOST_WIDE_INT bound; if (bound_scalar.is_constant (&bound)) { --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -3512,11 +3512,14 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, epilog = vect_epilogues ? get_loop_copy (loop) : scalar_loop; edge epilog_e = vect_epilogues ? e : scalar_e; edge new_epilog_e = NULL; - epilog = slpeel_tree_duplicate_loop_to_edge_cfg (loop, e, epilog, - epilog_e, e, - &new_epilog_e); + auto_vec doms; + epilog + = slpeel_tree_duplicate_loop_to_edge_cfg (loop, e, epilog, epilog_e, e, + &new_epilog_e, true, &doms); + LOOP_VINFO_EPILOGUE_IV_EXIT (loop_vinfo) = new_epilog_e; gcc_assert (epilog); + gcc_assert (new_epilog_e); epilog->force_vectorize = false; bb_before_epilog = loop_preheader_edge (epilog)->src; @@ -3610,10 +3613,16 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, } - if (skip_epilog) + if (skip_epilog || LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) { - guard_cond = fold_build2 (EQ_EXPR, boolean_type_node, + /* For the case where a different exit was chosen we must execute + the scalar loop with the remaining iterations. */ + if (inversed_iv) + guard_cond = boolean_false_node; + else + guard_cond = fold_build2 (EQ_EXPR, boolean_type_node, niters, niters_vector_mult_vf); + guard_bb = LOOP_VINFO_IV_EXIT (loop_vinfo)->dest; edge epilog_e = LOOP_VINFO_EPILOGUE_IV_EXIT (loop_vinfo); guard_to = split_edge (epilog_e); @@ -3621,11 +3630,13 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, skip_vector ? anchor : guard_bb, prob_epilog.invert (), irred_flag); + doms.safe_push (guard_to); if (vect_epilogues) epilogue_vinfo->skip_this_loop_edge = guard_e; edge main_iv = LOOP_VINFO_IV_EXIT (loop_vinfo); slpeel_update_phi_nodes_for_guard2 (loop, epilog, main_iv, guard_e, epilog_e); + /* Only need to handle basic block before epilog loop if it's not the guard_bb, which is the case when skip_vector is true. */ if (guard_bb != bb_before_epilog) @@ -3637,6 +3648,10 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, scale_loop_profile (epilog, prob_epilog, -1); } + /* Recalculate the dominators after adding the guard edge. */ + if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) + iterate_fix_dominators (CDI_DOMINATORS, doms, false); + unsigned HOST_WIDE_INT bound; if (bound_scalar.is_constant (&bound)) {