From patchwork Tue Sep 19 18:16:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eugene Rozenfeld X-Patchwork-Id: 1836834 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=microsoft.com header.i=@microsoft.com header.a=rsa-sha256 header.s=selector2 header.b=RGMjfyNH; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Rqqcj3LtLz1ync for ; Wed, 20 Sep 2023 04:16:53 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 64DCD3857006 for ; Tue, 19 Sep 2023 18:16:51 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DM5PR00CU002.outbound.protection.outlook.com (mail-centralusazon11021019.outbound.protection.outlook.com [52.101.62.19]) by sourceware.org (Postfix) with ESMTPS id 75FB53858D39 for ; Tue, 19 Sep 2023 18:16:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 75FB53858D39 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=microsoft.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=microsoft.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mT9thp0/ZaMs3kaDobOgQQBRF/5r6lXwj0xJAq8FguNOTUQLDmi/7EfFTfah6K3zX72F8Ak6cy/6shYoiDbzjkRNXfEA71iz/xiIBTEggtAokOWlmToqG8JALTP/zSbjQ/vLQ61Y0WVUUCd5BxaXqll+NfVCfKriAUy4jDGloLufQSnvsjp0r+GEq+OhCMqBNurU4AkVywCY7g4hHqtsr7HNEP1A7pA1Dqkp3gcNM5uplKrr3tCqofcFX5FsL30wy4CuzCb6rCqlAr/UEH8kfGuHUCiSihb9GBYLyTA+Qs15iukyrdK0KRNypKBK8TNXV1AueeXMCKjDKyugUOrHLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4WCPOrYjbE2Jam1h/Pxq3f8GTma3zxpAwr9Nsr1/E/8=; b=QRpZRpaAGRINKK/JEUy+iQuMGHKYoIWNLuA5QNqX+Y9PyRieF1H0CNUuX1bZhagNzftHxMhMwVVCYVmb8uSNkDhuE4A8s16fd0rgY8XrBC4CIfT5wH6Mql0HkzV1b/eLSOcGo1a9takrzSngDpvi5+MYkGtVn1B52rNIxyGpNRwbFDaSNBDkrQuc1tqcG9ien9SPuVpMrap32axHxbVzKlHAoKE918Gn8RxhZPE1I3DDwW/g7O6NLprH0fKq+7ehuTkq+w1njOSilDQclMVSHNsmeZZE2krkVTMsVWh58udg439z5wag5IYvjLdSSYf6C9aqxQjERNPpRg9wB3+wSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4WCPOrYjbE2Jam1h/Pxq3f8GTma3zxpAwr9Nsr1/E/8=; b=RGMjfyNHEKN9Wr1+09vQvG3D3/ufN8RRRj1H9RacZRA0HLsK/3yz/7luNnq5xRDkKjROBriLORne3zKrUndIek5XuqZkj6M95dXSiEB8E/ivS/Vm8Km5iq3zSUiT0xNbpUg9XOz2/DjznqiF7hVxi+7V/aty2g4aPP9tejbBpmc= Received: from DS7PR21MB3479.namprd21.prod.outlook.com (2603:10b6:8:90::11) by MW4PR21MB2041.namprd21.prod.outlook.com (2603:10b6:303:11c::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.6; Tue, 19 Sep 2023 18:16:22 +0000 Received: from DS7PR21MB3479.namprd21.prod.outlook.com ([fe80::1e33:1eb9:c91b:a]) by DS7PR21MB3479.namprd21.prod.outlook.com ([fe80::1e33:1eb9:c91b:a%4]) with mapi id 15.20.6838.003; Tue, 19 Sep 2023 18:16:22 +0000 From: Eugene Rozenfeld To: "gcc-patches@gcc.gnu.org" Subject: [PATCH] Fixes for profile count/probability maintenance Thread-Topic: [PATCH] Fixes for profile count/probability maintenance Thread-Index: AdnoP65wGpvO1rgdTXS34mry3Ww3dg== Date: Tue, 19 Sep 2023 18:16:21 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=03445a67-1243-411d-bf45-c94889974f7e; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-09-16T01:44:31Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DS7PR21MB3479:EE_|MW4PR21MB2041:EE_ x-ms-office365-filtering-correlation-id: b09c0272-fc27-46f6-499f-08dbb93c8724 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: UYVrbFu/xl1BTsrHCcOaVXGG2oRnhUdj48PMzDjk82VzH0TT1/IQGlI2nhgx3Z2dzgkN5zZNZ2SBSE0w/ZH7PGy4b8yfDmXesI63HyqQcFW6SB+Gq/oWBSMrgL0B98snjYDADVD1pjDPGgDoRqJ2f5TBTpIIdgy8EDg/D40xN6I97LXWfnWMVHChVhQw6M+LhaXT8HL5MfWuNshPSKMFWvi2VM7MOh4INoLfldHhSSmNYSZuXfV0wyTpSSyBsn3jRc9peS1x+uZBXx9EbqHuGj3TwV9LMVe6cglDvgvVtZmOEcOqvYwTwCcRCIJc2AavNfJwXLDzFFGA1HX6ToN0uocjbYI4p+Nje32pCg+prHa2ChPfxL7CeyyLFAd6j8qqiE+8hwQCYWhrKpaM2VOxHFNz/PnOq1H2vsNHxtAUdjavud4HaIcIJRmA9nCN5OzGJU2uCrjnwxWNWqiTMxfkiguY+GWotJamPCSAW7ZCie9MpDgqJ3RYz8Kme4+Q0Ck7Pb//SHCdqLLYaojb0D/YOo3bQA+en3X/tg0L8S0ysfOq3kfMkaEo5DeIUHt/YR6g+fWFbVR8xCZP6HgaFkl5NPfbUAJnqWS+DeOnTvM6HCsqbB16iUoDveprBRSZ7Y/9eLV6mM7rgx3Anemrpe5nHWXWoUeNpCmCptZyqiB1lYY= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS7PR21MB3479.namprd21.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(366004)(376002)(346002)(136003)(396003)(451199024)(1800799009)(186009)(9686003)(71200400001)(6506007)(7696005)(83380400001)(82950400001)(82960400001)(122000001)(86362001)(33656002)(38070700005)(38100700002)(8990500004)(55016003)(6916009)(316002)(66446008)(64756008)(76116006)(66946007)(66556008)(66476007)(41300700001)(2906002)(52536014)(5660300002)(8936002)(8676002)(10290500003)(478600001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: A4sDljw1rAdjUBxOmjiyhUc+TVSbA1R/6qAkxKwxwCCAhBpNRZMkRvYvLd/Vvmb0iLHaUlNCyHsMRHkA17fymYCcLmkkYYyY91RlBHDGnR2ILv4Q+VRRS60mViUviDdA6xVTWhDcAXjtF0wO8pRCxnhBk5zeZTK/IrdvVqsQrucow9M/2sDXrNXUbYLnBDoB2clS3O0VcdU3BSYFiVY/S8RrIroWD2VVQV96SP1+qQXz8+iLhKCLYqWKWw4ZUY4xrbM7Qj4qZVCMeUR74paXJQ81ek6V05VotrmxtBT5Fw+9MXULQk4E1OweQLXKQ2d0bfykprB88kW4bBNm6zFFgGov7KycFgLgaFG779AomHg8zyGHkVuA1PrTofOXZ9KwFoio87DeGOsjT4anKAqfpoFYLyU/0A28IMzv4Xhkisolo1CoWYWO6bwlBblCSnRxqHaDNkWn2rRsQOaWa3n1WGuRUiCq9EzbdyUm2OH0qu3MXqsuz3Uy9XmXOFTp+N77ARyjRDROWWgK0uQO62NUvkNRf9hn785iTq9SbDilgGbXYqBUoJtnITMArk6COAdB6tWzdFlhkxzWSUI5YIK1cp4VF8jawaEEBEI46WuD9dofR6LrOIVnQM+FevLU7TI9dqWSiIOalFCap9NXX05X7F/yuX5R2ihGZojK+wvtC9/d2cRhGxU8yn/byiVMHZ+7vVkS43nmBKcbo8f8MpNhwtlofIEbU2H4jDEY4mQmXOnl4upNPUpTeQ43r/OEB+8vkT6xwVNu3HEsRUXAiM/ZYXtLhc41PA1DGctCWx/lbDYazv3MMybWqYAyLD7EmF0dJg9gZfWIPWrjhOVq9dm38MZo9h3Olsv0zt27VPIhl7rBiIoWoX6z2YfB+pgOncxEHyQQ2WsYScXAvw7bT+jeEVVETmfgX5mMCmcnmNiGeSS/e/1wwIfivG8tmV5rnHtJ9NDEJ8uVK7wtxDQjFbBEgp5yKSdFfsNGxhVwFzsqHpLFJ1du8P9EdIj5Ra7itZ8En+WdFCNi3LXgrDBNLN3wKCF3W9UKAyeQbeLlrlLvJfgXMnRMTsev6dY10EE2or4eSl9TgF2LcBNjJDuCx4VI5PZtzyrNUrV/AdeYOdEPk7UmqjJU7qTi7Pf8vypNETEFRnItBWH6gT8XD4O76RfDn9APTeRHdZTbGv5D4L+Ncc+hoF8Hyum/TxVz8hpQMeWXt7OKUNGABH9TJUzpHtg7SZoHXSHkjSmycJDu60gN4vxHpY6s6mevamedhpYKKfrYotg2dmjzi+nT9lV/YOQ8gI4TEe9+DT9UIsPRT1tYiejWxmm6SJMaRJ5f52WI8238c+B3tWYstRhuuQ+83rNwakUaegXKq1/dPJsyaPf1oyEgNwy0xSPRSF/MUP6Xih8F7PdwcYKi0PNIUPoXNaSiQVW5f7GyuvjKfF9s9TCbl54wrtRFJVvbAkWYob1JyIldj38yct+LTOqeDMwogUQm8sifdRY2rQG0ZGcnN5AyNf7+ax8ZHx6x+jA0osEaiFKh6kqNtZYnwzfpiTyZCoxBZfUyQWxgpS9k6lNzWpku0XYYO5O90n5CNzh8/MwqP/d73OsagZkOWKIU7CjP/H9oXnBkhK1izkChx1/F6dPn15iBFH1NbyhyVpuSvlyEa6f2 MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DS7PR21MB3479.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: b09c0272-fc27-46f6-499f-08dbb93c8724 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Sep 2023 18:16:21.9138 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: R4I8kUuV0ZGdaeryfQpkKlVLTfvOONLidmn2phcEtpXoHmkGqkqETJN3eJkKf6/SjOikw6shNd2D69sQuagkYg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR21MB2041 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Verifier checks have recently been strengthened to check that all counts and probabilities are initialized. The checks fired during autoprofiledbootstrap build and this patch fixes it. gcc/ChangeLog: * auto-profile.cc (afdo_calculate_branch_prob): Fix count comparisons * ipa-utils.cc (ipa_merge_profiles): Guard against zero count when computing probabilities * tree-vect-loop-manip.cc (vect_do_peeling): Guard against zero count when scaling loop profile Tested on x86_64-pc-linux-gnu. --- gcc/auto-profile.cc | 4 ++-- gcc/ipa-utils.cc | 16 +++++++++------- gcc/tree-vect-loop-manip.cc | 2 +- 3 files changed, 12 insertions(+), 10 deletions(-) -- 2.25.1 diff --git a/gcc/auto-profile.cc b/gcc/auto-profile.cc index ff3b763945c..3e61f36c29b 100644 --- a/gcc/auto-profile.cc +++ b/gcc/auto-profile.cc @@ -1434,7 +1434,7 @@ afdo_calculate_branch_prob (bb_set *annotated_bb) else total_count += AFDO_EINFO (e)->get_count (); } - if (num_unknown_succ == 0 && total_count > profile_count::zero ()) + if (num_unknown_succ == 0 && total_count > profile_count::zero ().afdo ()) { FOR_EACH_EDGE (e, ei, bb->succs) e->probability @@ -1571,7 +1571,7 @@ afdo_annotate_cfg (const stmt_set &promoted_stmts) DECL_SOURCE_LOCATION (current_function_decl)); afdo_source_profile->mark_annotated (cfun->function_start_locus); afdo_source_profile->mark_annotated (cfun->function_end_locus); - if (max_count > profile_count::zero ()) + if (max_count > profile_count::zero ().afdo ()) { /* Calculate, propagate count and probability information on CFG. */ afdo_calculate_branch_prob (&annotated_bb); diff --git a/gcc/ipa-utils.cc b/gcc/ipa-utils.cc index 956c6294fd7..3aaf7e595df 100644 --- a/gcc/ipa-utils.cc +++ b/gcc/ipa-utils.cc @@ -651,13 +651,15 @@ ipa_merge_profiles (struct cgraph_node *dst, { edge srce = EDGE_SUCC (srcbb, i); edge dste = EDGE_SUCC (dstbb, i); - dste->probability = - dste->probability * dstbb->count.ipa ().probability_in - (dstbb->count.ipa () - + srccount.ipa ()) - + srce->probability * srcbb->count.ipa ().probability_in - (dstbb->count.ipa () - + srccount.ipa ()); + profile_count total = dstbb->count.ipa () + srccount.ipa (); + if (total.nonzero_p ()) + { + dste->probability = + dste->probability * dstbb->count.ipa ().probability_in + (total) + + srce->probability * srcbb->count.ipa ().probability_in + (total); + } } dstbb->count = dstbb->count.ipa () + srccount.ipa (); } diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index 09641901ff1..2608c286e5d 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -3335,7 +3335,7 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, free (bbs); free (original_bbs); } - else + else if (old_count.nonzero_p ()) scale_loop_profile (epilog, guard_to->count.probability_in (old_count), -1); /* Only need to handle basic block before epilog loop if it's not