From patchwork Wed Jun 8 16:32:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wilco Dijkstra X-Patchwork-Id: 1640807 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=Y4HDFhHS; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LJCSd14q3z9sFk for ; Thu, 9 Jun 2022 02:32:49 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 488D03850214 for ; Wed, 8 Jun 2022 16:32:47 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 488D03850214 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1654705967; bh=WTsLS9TNLKQjxrgY1hwy3nzew6zPqv8XOgzUIe3G1PI=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=Y4HDFhHScp00TTO2UyYpsIJu5jJNdlXq4hiK/83PeB06RZPQy70GR1FVu+N3INsZj iyUCOVEpizOOhIOBn+nv3Kby9VnEJ94/OctGRP5tRhozFN1Y8XFE7PXafnq+uluFxL NKgKO4Ao8cktI3odvUNjqL1kPoIGNJ343gJy1sZg= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2047.outbound.protection.outlook.com [40.107.21.47]) by sourceware.org (Postfix) with ESMTPS id 397643850846 for ; Wed, 8 Jun 2022 16:32:31 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 397643850846 ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=kXtWZr4YAknT5niCVluY2857h+5HG7MIxmj0kJjp5EzKdMJr7JHanCTIArkSvYRUg2lKcdBfWmDiWgpA3e9wNjW8qkZEn2a5ogSRTZ+3+F/AGt+mwLXsvgX+SH9AO38yOthrXGkrHZWBKIkmTkHOvvRE0rTS6bkrxY44rrxGLuV3y2VcJb39ZGYkRWJCzNpAmBMArSBuq/sZ5Gcx8JUZag4jqa12M1qwuym3ctJoJjYCrrh0X3u9NwsZiSxTmQh9dzuiIBVZbOR/KXffU4trN03FB8OpJK6e82TXvDeLhXZ3auD0ehyHys7a68wDZ8JLMe9OMyeqzS2PSYJvQeS0ig== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WTsLS9TNLKQjxrgY1hwy3nzew6zPqv8XOgzUIe3G1PI=; b=X3RNkcXJLDLn2pmIhofAs372V7jhoednfLqQxDnZ03r0i6DMORzheP0mORI77bpsmCYvnTnGaDlcOxQc1TpupDKl4HsQ8WMthrJMorBs17lV9yeIsfe6V9ernxu3CmjKj46jbrn7N9U9FCqEfh6ZEagi9+9Cl70Z4K3YXuwYOuSWdx4wc0E0gyZjkiemJz6xItQ/Wc4eoKW5yK6j93SV/60PHXP46+kOgoHSrOdlGhmFRDKuPzi2VHl4CGiFhh4DFjzHvF5plPVWQ5Q1SOnEvGP+8MVY2PudaGvZS/jWKPW0gzGmHiXXudTGGHYrFpvEw9wmqzHReY3fqq0WLrKnjQ== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=sourceware.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com]) Received: from AM6P191CA0049.EURP191.PROD.OUTLOOK.COM (2603:10a6:209:7f::26) by AM9PR08MB6737.eurprd08.prod.outlook.com (2603:10a6:20b:304::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.11; Wed, 8 Jun 2022 16:32:29 +0000 Received: from AM5EUR03FT003.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:7f:cafe::26) by AM6P191CA0049.outlook.office365.com (2603:10a6:209:7f::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.12 via Frontend Transport; Wed, 8 Jun 2022 16:32:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT003.mail.protection.outlook.com (10.152.16.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.12 via Frontend Transport; Wed, 8 Jun 2022 16:32:28 +0000 Received: ("Tessian outbound 4ab5a053767b:v120"); Wed, 08 Jun 2022 16:32:28 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 3721d5f6a0bd870b X-CR-MTA-TID: 64aa7808 Received: from c1ab9317cd02.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id F8AEBAD0-72F9-4ABF-936D-63F163DC7CD6.1; Wed, 08 Jun 2022 16:32:22 +0000 Received: from EUR04-VI1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id c1ab9317cd02.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 08 Jun 2022 16:32:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VeHmfrqsfqRcI0orU939w6u6HEJkxa6oB+ULOpITOMTV54oxWn0UdkbxVgF5Jm3b+/gbnBvJoVtXtIeVEfintm00FoNSOz2wc9YDWNaCGHZdmttMtc9PTbROWrEemRAIq2kB66EUqfrpPu8MuypNsFPcBMag2ItMPBKDBhDOeUpO/iMFWH4ZBJcMQXxC4VbyJgtCpk4HvtrhHH3jbPvpyD5zaFXs9T8XYiwwTL0u7mW4N1LImv6jOnLDTyKX+JHdQx1ZoQOZvFhc4SM6jiAYNopUg8ZzxVe4bcYPqCEluS3d85mOqaOl2Wy1FeTtweXvSd5h0MRsGFC3dHw8RiktzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WTsLS9TNLKQjxrgY1hwy3nzew6zPqv8XOgzUIe3G1PI=; b=YtNAL2cP7LXnVgCIyCQ25fLauCMJIi4bljZhRVW2u+JqqLpI5xUBXBC0By+CgdZHoml5Z+AXcTBGGUh+1XsOaLeE8ebkJ8Wpy33tVONro9w4waT48DvIML500n7N6RAc7mNkCSDdaVYm4lF/CKM7PA0eDVrYJ/wfrkWUjZ/Zxi//ueV+LUTOiB+9Taq/XB2GXcZPnaBOf6BsX+kUZHswRdAV/aKmC3ZgbaNzqFMZNHVPOyP1YJUfreOG1Lt7n1vT8ZboE6H0Z1FcSboBKtiK/zvNq7yGuv5Cf+TAic4oNXHv1RRbE0nOrZF6vGn2lPrMAwGZywhQGjO4UV13JKPSJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Received: from AM5PR0801MB1668.eurprd08.prod.outlook.com (2603:10a6:203:3c::14) by HE1PR0801MB1900.eurprd08.prod.outlook.com (2603:10a6:3:4d::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Wed, 8 Jun 2022 16:32:20 +0000 Received: from AM5PR0801MB1668.eurprd08.prod.outlook.com ([fe80::a114:fc7:45f8:421]) by AM5PR0801MB1668.eurprd08.prod.outlook.com ([fe80::a114:fc7:45f8:421%5]) with mapi id 15.20.5314.019; Wed, 8 Jun 2022 16:32:19 +0000 To: 'GNU C Library' Subject: [PATCH] Improve performance of IO locks Thread-Topic: [PATCH] Improve performance of IO locks Thread-Index: AQHYe1UOi6DbewpxBkWYPihtt69DaQ== Date: Wed, 8 Jun 2022 16:32:19 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-MS-Office365-Filtering-Correlation-Id: 66c85610-2a63-439d-c317-08da496c7a66 x-ms-traffictypediagnostic: HE1PR0801MB1900:EE_|AM5EUR03FT003:EE_|AM9PR08MB6737:EE_ X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 69mv3Zf/jjtAlZUPmJz7Ll4yKLpCMeobsvtttf/LD7Uk0fXhp3EDK1B3rxGkSnlUB7EvYj+GL1uY2Vi4Ajl3Hrz7pJrVwHj7Xe6LuJxCtT4ILPt8ZcV7bOdUbdYPAey/SEYs0olK2clSx/wBcu0s4FwTJwPLaRScLbu6qN7SXIMM58U570olEfcc4VGag6e9EK9RznBsXhgtA9M0oJ4CY7WkMIT3JfNwo2us/3dh7Wt00az5o2EEszGPOoMi8cCxqMlP7aF1KjVEMc7mYkrmX9/x2+qIxCX7h8YFWBunwA49qHKVMWws4Cx8g51HTFMZufVfO0Jj94oEbCc8xAxH8oExPZzqSQFHkQejvAO7wQFGKo/V71u5tnJ+kMzmTU34uxFUzNL+nIItyRxDJjCBzSwe27YqjlV0dN0jGn5emCAPtg4qwPO3rYaKRxSaNQQa5Gi556KDS+AP49XG6NczXw8hF8GrZfzpMt4uPybK2uJ15g1V+vZ9cEEZ2S3rJBvz8SgR6b/vCt2epg2QWG0YTpg75Zk2q22XFiBHJffB8dv1NcsEj5CWkyC4vAoYvsr+a+30cUbA9VPt4VfXDAklqvbrU+LJATC//JrJg5XJMEWpj30voPvRlEO2ZgUXDUNZfP/JXwDkK8oTcjTXu2hxxx+88Ukf6sLDlXeAvd+mGvJwGnjWDKBf4zcAJwu2JI1DzI4WqA94GX9YKnP5mBCSLg== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM5PR0801MB1668.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(316002)(8936002)(38070700005)(66556008)(186003)(71200400001)(2906002)(508600001)(33656002)(83380400001)(38100700002)(66946007)(122000001)(64756008)(55016003)(66476007)(66446008)(6916009)(7696005)(4326008)(76116006)(8676002)(26005)(91956017)(6506007)(86362001)(5660300002)(52536014)(9686003); DIR:OUT; SFP:1101; MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0801MB1900 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT003.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 1af91c71-d853-4d2c-546a-08da496c754a X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: o8qdIxbhUy6X8Cjvht0V/rzqZXfh1lZiXTLKYFs7z1JnngANU0CmLGk4cAidmNAWnSKRfpkR3UZgBiX5dlCESSMrGVsNSDqjb/Krzixu/VJvAdDNYmcV3KHtBjkXYGvfJyU9Rk5k8juUn3XErdLNMzLvKN4jNNAN0mI5rfZNghbBSSuu7SLSGYFBaixmnBBt6pN5djz2fCYmmaOFw9drcDL9GnuIeOK5O4loiy7IFq+XuhNK02qAxHt3yZT+ZATdTcLCnDlbD6eC7/33bUFl5urXLXJdBIK9S14I2kx58161nMP99PgQqGFwH05weGTT+p44BvPZf3p7JWzgmQ5/jE2Y/aLPWHvzNH4yeq0wlvpnv2c7meMnMlRCdixmkDDy3msQFQZhWnkegT5niog34v0H2uRKdMwG6nGl7thFaXKuGxTk5cN9ety1+yoUkmCWY/iPKqnBdNnjR1q1w5YA0HU3s9LYljenxYo6/i6jrcQ2FXZzK7+91BQm5MQ1522rzU/nuHWk+4+kY0vdT6xq17lBKYq7w20jPB4s8XyRJh2IYKBWZ83y+9BxqSxYwCCntVu6kNyUWkM6ke1kKPMkVR7F8zJDJZ66fVaz4hHbV0R/hItZ2dOGneKlHPWpEu+tpkX+MvFb49Yvf8iBdOTv9GUFsuOSChgHYKtK8gL4kWw= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(336012)(70586007)(52536014)(26005)(316002)(47076005)(8676002)(33656002)(107886003)(40460700003)(70206006)(9686003)(186003)(8936002)(81166007)(83380400001)(86362001)(36860700001)(82310400005)(4326008)(2906002)(356005)(55016003)(5660300002)(508600001)(7696005)(6506007)(6916009); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jun 2022 16:32:28.4643 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 66c85610-2a63-439d-c317-08da496c7a66 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT003.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR08MB6737 X-Spam-Status: No, score=-11.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, GIT_PATCH_0, KAM_DMARC_NONE, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Wilco Dijkstra via Libc-alpha From: Wilco Dijkstra Reply-To: Wilco Dijkstra Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" Improve performance of recursive IO locks by adding a fast path for the single-threaded case. To reduce the number of memory accesses for locking/unlocking, only increment the recursion counter if the lock is already taken. On Neoverse V1, a microbenchmark with many small freads improved by 2.9 times. Multithreaded performance improved by 2%. Passes GLIBC testsuite, OK for commit? diff --git a/sysdeps/nptl/stdio-lock.h b/sysdeps/nptl/stdio-lock.h index afa0b779c81d7dd915f8edb6c0974e4f231d4e0a..45823cd1629d3e3efecc64a7d07706a6e6de9af1 100644 --- a/sysdeps/nptl/stdio-lock.h +++ b/sysdeps/nptl/stdio-lock.h @@ -37,12 +37,18 @@ typedef struct { int lock; int cnt; void *owner; } _IO_lock_t; #define _IO_lock_lock(_name) \ do { \ void *__self = THREAD_SELF; \ - if ((_name).owner != __self) \ + if (SINGLE_THREAD_P && (_name).owner == NULL) \ + { \ + (_name).lock = LLL_LOCK_INITIALIZER_LOCKED; \ + (_name).owner = __self; \ + } \ + else if ((_name).owner != __self) \ { \ lll_lock ((_name).lock, LLL_PRIVATE); \ - (_name).owner = __self; \ + (_name).owner = __self; \ } \ - ++(_name).cnt; \ + else \ + ++(_name).cnt; \ } while (0) #define _IO_lock_trylock(_name) \ @@ -52,10 +58,7 @@ typedef struct { int lock; int cnt; void *owner; } _IO_lock_t; if ((_name).owner != __self) \ { \ if (lll_trylock ((_name).lock) == 0) \ - { \ - (_name).owner = __self; \ - (_name).cnt = 1; \ - } \ + (_name).owner = __self; \ else \ __result = EBUSY; \ } \ @@ -66,11 +69,18 @@ typedef struct { int lock; int cnt; void *owner; } _IO_lock_t; #define _IO_lock_unlock(_name) \ do { \ - if (--(_name).cnt == 0) \ + if (SINGLE_THREAD_P && (_name).cnt == 0) \ + { \ + (_name).owner = NULL; \ + (_name).lock = 0; \ + } \ + else if ((_name).cnt == 0) \ { \ - (_name).owner = NULL; \ + (_name).owner = NULL; \ lll_unlock ((_name).lock, LLL_PRIVATE); \ } \ + else \ + --(_name).cnt; \ } while (0)