From patchwork Tue Feb 23 10:17:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 1443455 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces@sourceware.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=ujpt5VUh; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DlFMP0z1Dz9sVV for ; Tue, 23 Feb 2021 21:16:33 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3430F3894405; Tue, 23 Feb 2021 10:16:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3430F3894405 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1614075391; bh=rs0k8EVsyTiwV6D5gtpLiLT1zN9l8IaQRV08USSMQ0A=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=ujpt5VUhMv1UbhQTxAGDi0zw5koWPoDUJenWCssuwWPf+LN1VF0OocXjPBkA2TlDA 9vULP2e43avxrsHM9/uK7O+CPpSUODPwOF1uFSXPTbEpm6odpm0Il/A3vXKnnCw5vM 8lrqBtPTqu42KUerHz2+Kc9rQr2obANpr1/yaCqg= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by sourceware.org (Postfix) with ESMTP id DF7C5388CC1E for ; Tue, 23 Feb 2021 10:16:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org DF7C5388CC1E Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-165-ea9CNTcJPPmuX9tCvos77Q-1; Tue, 23 Feb 2021 05:16:20 -0500 X-MC-Unique: ea9CNTcJPPmuX9tCvos77Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 864E085EE61 for ; Tue, 23 Feb 2021 10:16:19 +0000 (UTC) Received: from oldenburg.str.redhat.com (ovpn-113-131.ams2.redhat.com [10.36.113.131]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 152E05F706 for ; Tue, 23 Feb 2021 10:16:17 +0000 (UTC) To: libc-alpha@sourceware.org Subject: [PATCH v2] nptl: Move elision implementations into libc Date: Tue, 23 Feb 2021 11:17:01 +0100 Message-ID: <87o8gbytpe.fsf@oldenburg.str.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.4 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Florian Weimer via Libc-alpha From: Florian Weimer Reply-To: Florian Weimer Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" The elision interfaces are closely aligned between the targets that implement them, so declare them in the generic file. Empty .c stubs are provided, so that fewer makefile updates under sysdeps are needed. Also simplify initialization via __libc_early_init. The symbols __lll_clocklock_elision, __lll_lock_elision, __lll_trylock_elision, __lll_unlock_elision, __pthread_force_elision move into libc. For the time being, non-hidden references are used from libpthread to access them, but once that part of libpthread is moved into libc, hidden symbols will be used again. (Hidden references seem desirable to reduce the likelihood of transactions aborts.) --- v2: Renamed HAVE_ELISION to ENABLE_ELISION_SUPPORT. Replaced incorrect #ifdef in pthread_mutex_lock (but apparently harmless) with #if. Retested on aarch64-linux-gnu (no elision support), i686-linux-gnu, x86_64-linux-gnu. Built with build-many-glibcs.py. elf/libc_early_init.c | 6 ++ nptl/Makefile | 5 + nptl/Versions | 5 + .../pthread_mutex_lock.c => nptl/elision-conf.c | 9 +- .../pthread_mutex_lock.c => nptl/elision-lock.c | 9 +- .../elision-timed.c | 9 +- .../pthread_mutex_lock.c => nptl/elision-trylock.c | 9 +- nptl/elision-unlock.c | 17 ++++ nptl/lowlevellock.c | 5 + nptl/pthread_mutex_lock.c | 15 +-- nptl/pthread_mutex_timedlock.c | 13 --- nptl/pthread_mutex_trylock.c | 8 -- nptl/pthread_mutex_unlock.c | 4 - sysdeps/nptl/lowlevellock.h | 102 +++++++++++++++++++++ .../elision-conf.h} | 17 +++- sysdeps/unix/sysv/linux/powerpc/Makefile | 2 - sysdeps/unix/sysv/linux/powerpc/elision-conf.c | 26 +----- sysdeps/unix/sysv/linux/powerpc/elision-conf.h | 8 +- sysdeps/unix/sysv/linux/powerpc/elision-lock.c | 7 +- sysdeps/unix/sysv/linux/powerpc/elision-trylock.c | 1 + sysdeps/unix/sysv/linux/powerpc/elision-unlock.c | 1 + sysdeps/unix/sysv/linux/powerpc/force-elision.h | 62 ------------- sysdeps/unix/sysv/linux/powerpc/lowlevellock.h | 49 ---------- .../sysv/linux/powerpc/pthread_mutex_cond_lock.c | 22 ----- sysdeps/unix/sysv/linux/s390/Makefile | 3 - sysdeps/unix/sysv/linux/s390/elision-conf.c | 27 +----- sysdeps/unix/sysv/linux/s390/elision-conf.h | 8 +- sysdeps/unix/sysv/linux/s390/elision-lock.c | 7 +- sysdeps/unix/sysv/linux/s390/elision-trylock.c | 1 + sysdeps/unix/sysv/linux/s390/elision-unlock.c | 1 + sysdeps/unix/sysv/linux/s390/force-elision.h | 62 ------------- sysdeps/unix/sysv/linux/s390/lowlevellock.h | 49 ---------- .../unix/sysv/linux/s390/pthread_mutex_cond_lock.c | 22 ----- .../unix/sysv/linux/s390/pthread_mutex_timedlock.c | 22 ----- .../unix/sysv/linux/s390/pthread_mutex_trylock.c | 22 ----- sysdeps/unix/sysv/linux/x86/Makefile | 2 - sysdeps/unix/sysv/linux/x86/elision-conf.c | 25 +---- sysdeps/unix/sysv/linux/x86/elision-conf.h | 8 +- sysdeps/unix/sysv/linux/x86/elision-lock.c | 7 +- sysdeps/unix/sysv/linux/x86/elision-trylock.c | 1 + sysdeps/unix/sysv/linux/x86/elision-unlock.c | 1 + sysdeps/unix/sysv/linux/x86/force-elision.h | 62 ------------- sysdeps/unix/sysv/linux/x86/lowlevellock.h | 24 ----- .../unix/sysv/linux/x86/pthread_mutex_cond_lock.c | 22 ----- .../unix/sysv/linux/x86/pthread_mutex_timedlock.c | 22 ----- .../unix/sysv/linux/x86/pthread_mutex_trylock.c | 22 ----- 46 files changed, 185 insertions(+), 646 deletions(-) diff --git a/elf/libc_early_init.c b/elf/libc_early_init.c index 4755554f12..9b1901a706 100644 --- a/elf/libc_early_init.c +++ b/elf/libc_early_init.c @@ -17,8 +17,10 @@ . */ #include +#include #include #include +#include #include #ifdef SHARED @@ -37,4 +39,8 @@ __libc_early_init (_Bool initial) #ifdef SHARED __libc_initial = initial; #endif + +#if ENABLE_ELISION_SUPPORT + __lll_elision_init (); +#endif } diff --git a/nptl/Makefile b/nptl/Makefile index 4dd9c6a6a8..5f85dd7854 100644 --- a/nptl/Makefile +++ b/nptl/Makefile @@ -30,6 +30,11 @@ extra-libs-others := $(extra-libs) routines = \ alloca_cutoff \ + elision-conf \ + elision-lock \ + elision-timed \ + elision-trylock \ + elision-unlock \ forward \ futex-internal \ libc-cancellation \ diff --git a/nptl/Versions b/nptl/Versions index 6cca579a0a..186befebfd 100644 --- a/nptl/Versions +++ b/nptl/Versions @@ -79,8 +79,12 @@ libc { __libc_current_sigrtmin_private; __libc_dl_error_tsd; __libc_pthread_init; + __lll_clocklock_elision; + __lll_lock_elision; __lll_lock_wait; __lll_lock_wait_private; + __lll_trylock_elision; + __lll_unlock_elision; __pthread_attr_copy; __pthread_attr_destroy; __pthread_attr_init; @@ -88,6 +92,7 @@ libc { __pthread_attr_setsigmask_internal; __pthread_cond_destroy; # Used by the C11 threads. __pthread_cond_init; # Used by the C11 threads. + __pthread_force_elision; __pthread_getattr_default_np; } } diff --git a/sysdeps/unix/sysv/linux/s390/pthread_mutex_lock.c b/nptl/elision-conf.c similarity index 79% rename from sysdeps/unix/sysv/linux/s390/pthread_mutex_lock.c rename to nptl/elision-conf.c index b193b5b2b0..d71c30a01b 100644 --- a/sysdeps/unix/sysv/linux/s390/pthread_mutex_lock.c +++ b/nptl/elision-conf.c @@ -1,5 +1,5 @@ -/* Elided version of pthread_mutex_lock. - Copyright (C) 2014-2021 Free Software Foundation, Inc. +/* elision-conf.c: Lock elision tunable parameters. Stub version. + Copyright (C) 2021 Free Software Foundation, Inc. This file is part of the GNU C Library. The GNU C Library is free software; you can redistribute it and/or @@ -15,8 +15,3 @@ You should have received a copy of the GNU Lesser General Public License along with the GNU C Library; if not, see . */ - -#include -#include - -#include diff --git a/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_lock.c b/nptl/elision-lock.c similarity index 79% rename from sysdeps/unix/sysv/linux/powerpc/pthread_mutex_lock.c rename to nptl/elision-lock.c index 110a6bf47b..a8235cbbea 100644 --- a/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_lock.c +++ b/nptl/elision-lock.c @@ -1,5 +1,5 @@ -/* Elided version of pthread_mutex_lock. - Copyright (C) 2015-2021 Free Software Foundation, Inc. +/* elision-lock.c: Lock elision locking. Stub version. + Copyright (C) 2021 Free Software Foundation, Inc. This file is part of the GNU C Library. The GNU C Library is free software; you can redistribute it and/or @@ -15,8 +15,3 @@ You should have received a copy of the GNU Lesser General Public License along with the GNU C Library; if not, see . */ - -#include -#include - -#include diff --git a/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_trylock.c b/nptl/elision-timed.c similarity index 78% rename from sysdeps/unix/sysv/linux/powerpc/pthread_mutex_trylock.c rename to nptl/elision-timed.c index ee9fe1d186..4723197363 100644 --- a/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_trylock.c +++ b/nptl/elision-timed.c @@ -1,5 +1,5 @@ -/* Elided version of pthread_mutex_trylock. - Copyright (C) 2015-2021 Free Software Foundation, Inc. +/* elision-lock.c: Lock elision timed locking. Stub version. + Copyright (C) 2021 Free Software Foundation, Inc. This file is part of the GNU C Library. The GNU C Library is free software; you can redistribute it and/or @@ -15,8 +15,3 @@ You should have received a copy of the GNU Lesser General Public License along with the GNU C Library; if not, see . */ - -#include -#include - -#include diff --git a/sysdeps/unix/sysv/linux/x86/pthread_mutex_lock.c b/nptl/elision-trylock.c similarity index 79% rename from sysdeps/unix/sysv/linux/x86/pthread_mutex_lock.c rename to nptl/elision-trylock.c index 6e5f1f9964..286a7417fa 100644 --- a/sysdeps/unix/sysv/linux/x86/pthread_mutex_lock.c +++ b/nptl/elision-trylock.c @@ -1,5 +1,5 @@ -/* Elided version of pthread_mutex_lock. - Copyright (C) 2011-2021 Free Software Foundation, Inc. +/* elision-lock.c: Lock elision locking attempts. Stub version. + Copyright (C) 2021 Free Software Foundation, Inc. This file is part of the GNU C Library. The GNU C Library is free software; you can redistribute it and/or @@ -15,8 +15,3 @@ You should have received a copy of the GNU Lesser General Public License along with the GNU C Library; if not, see . */ - -#include -#include "force-elision.h" - -#include "nptl/pthread_mutex_lock.c" diff --git a/nptl/elision-unlock.c b/nptl/elision-unlock.c new file mode 100644 index 0000000000..552a90ade2 --- /dev/null +++ b/nptl/elision-unlock.c @@ -0,0 +1,17 @@ +/* elision-lock.c: Lock elision unlocking support. Stub version. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ diff --git a/nptl/lowlevellock.c b/nptl/lowlevellock.c index 7976c6d8f5..2d077d8694 100644 --- a/nptl/lowlevellock.c +++ b/nptl/lowlevellock.c @@ -51,3 +51,8 @@ __lll_lock_wait (int *futex, int private) } } libc_hidden_def (__lll_lock_wait) + +#if ENABLE_ELISION_SUPPORT +int __pthread_force_elision __attribute__ ((nocommon)); +libc_hidden_data_def (__pthread_force_elision) +#endif diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index 319658adc4..f0de7b7fd6 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -27,15 +27,6 @@ #include #include -#ifndef lll_lock_elision -#define lll_lock_elision(lock, try_lock, private) ({ \ - lll_lock (lock, private); 0; }) -#endif - -#ifndef lll_trylock_elision -#define lll_trylock_elision(a,t) lll_trylock(a) -#endif - /* Some of the following definitions differ when pthread_mutex_cond_lock.c includes this file. */ #ifndef LLL_MUTEX_LOCK @@ -52,10 +43,6 @@ PTHREAD_MUTEX_PSHARED (mutex)) #endif -#ifndef FORCE_ELISION -#define FORCE_ELISION(m, s) -#endif - static int __pthread_mutex_lock_full (pthread_mutex_t *mutex) __attribute_noinline__; @@ -80,7 +67,7 @@ __pthread_mutex_lock (pthread_mutex_t *mutex) LLL_MUTEX_LOCK (mutex); assert (mutex->__data.__owner == 0); } -#ifdef HAVE_ELISION +#if ENABLE_ELISION_SUPPORT else if (__glibc_likely (type == PTHREAD_MUTEX_TIMED_ELISION_NP)) { elision: __attribute__((unused)) diff --git a/nptl/pthread_mutex_timedlock.c b/nptl/pthread_mutex_timedlock.c index 8428ebca0f..5e52a4d856 100644 --- a/nptl/pthread_mutex_timedlock.c +++ b/nptl/pthread_mutex_timedlock.c @@ -29,19 +29,6 @@ #include -#ifndef lll_clocklock_elision -#define lll_clocklock_elision(futex, adapt_count, clockid, abstime, private) \ - __futex_clocklock64 (&(futex), clockid, abstime, private) -#endif - -#ifndef lll_trylock_elision -#define lll_trylock_elision(a,t) lll_trylock(a) -#endif - -#ifndef FORCE_ELISION -#define FORCE_ELISION(m, s) -#endif - int __pthread_mutex_clocklock_common (pthread_mutex_t *mutex, clockid_t clockid, diff --git a/nptl/pthread_mutex_trylock.c b/nptl/pthread_mutex_trylock.c index 519727580d..e52dddafbd 100644 --- a/nptl/pthread_mutex_trylock.c +++ b/nptl/pthread_mutex_trylock.c @@ -23,14 +23,6 @@ #include #include -#ifndef lll_trylock_elision -#define lll_trylock_elision(a,t) lll_trylock(a) -#endif - -#ifndef FORCE_ELISION -#define FORCE_ELISION(m, s) -#endif - int __pthread_mutex_trylock (pthread_mutex_t *mutex) { diff --git a/nptl/pthread_mutex_unlock.c b/nptl/pthread_mutex_unlock.c index e29bb7fa53..3b5ccdacf9 100644 --- a/nptl/pthread_mutex_unlock.c +++ b/nptl/pthread_mutex_unlock.c @@ -24,10 +24,6 @@ #include #include -#ifndef lll_unlock_elision -#define lll_unlock_elision(a,b,c) ({ lll_unlock (a,c); 0; }) -#endif - static int __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) __attribute_noinline__; diff --git a/sysdeps/nptl/lowlevellock.h b/sysdeps/nptl/lowlevellock.h index 176ba96251..4b011e9cc5 100644 --- a/sysdeps/nptl/lowlevellock.h +++ b/sysdeps/nptl/lowlevellock.h @@ -20,6 +20,7 @@ #define _LOWLEVELLOCK_H 1 #include +#include #include #include @@ -160,4 +161,105 @@ libc_hidden_proto (__lll_lock_wait) #define LLL_LOCK_INITIALIZER (0) #define LLL_LOCK_INITIALIZER_LOCKED (1) +/* Elision support. */ + +/* Force elision for all new locks. This is used to decide whether + existing DEFAULT locks should be automatically upgraded to elision + in pthread_mutex_lock. Disabled for suid programs. Only used when + elision is available. */ +extern int __pthread_force_elision; +libc_hidden_proto (__pthread_force_elision) + +extern void __lll_elision_init (void) attribute_hidden; +extern int __lll_clocklock_elision (int *futex, short *adapt_count, + clockid_t clockid, + const struct __timespec64 *timeout, + int private); +libc_hidden_proto (__lll_clocklock_elision) + +extern int __lll_lock_elision (int *futex, short *adapt_count, int private); +libc_hidden_proto (__lll_lock_elision) + +#if ELISION_UNLOCK_NEEDS_ADAPT_COUNT +extern int __lll_unlock_elision (int *lock, short *adapt_count, int private); +#else +extern int __lll_unlock_elision (int *lock, int private); +#endif +libc_hidden_proto (__lll_unlock_elision) + +extern int __lll_trylock_elision (int *lock, short *adapt_count); +libc_hidden_proto (__lll_trylock_elision) + +#if ENABLE_ELISION_SUPPORT +# define lll_clocklock_elision(futex, adapt_count, clockid, timeout, private) \ + __lll_clocklock_elision (&(futex), &(adapt_count), clockid, timeout, private) +# define lll_lock_elision(futex, adapt_count, private) \ + __lll_lock_elision (&(futex), &(adapt_count), private) +# define lll_trylock_elision(futex, adapt_count) \ + __lll_trylock_elision (&(futex), &(adapt_count)) +# if ELISION_UNLOCK_NEEDS_ADAPT_COUNT +# define lll_unlock_elision(futex, adapt_count, private) \ + __lll_unlock_elision (&(futex), &(adapt_count), private) +#else +# define lll_unlock_elision(futex, adapt_count, private) \ + __lll_unlock_elision (&(futex), private) +#endif + +/* Automatically enable elision for existing user lock kinds. */ +# define FORCE_ELISION(m, s) \ + if (__pthread_force_elision) \ + { \ + /* See concurrency notes regarding __kind in \ + struct __pthread_mutex_s in \ + sysdeps/nptl/bits/thread-shared-types.h. \ + \ + There are the following cases for the kind of a mutex \ + (The mask PTHREAD_MUTEX_ELISION_FLAGS_NP covers the flags \ + PTHREAD_MUTEX_ELISION_NP and PTHREAD_MUTEX_NO_ELISION_NP where \ + only one of both flags can be set): \ + - both flags are not set: \ + This is the first lock operation for this mutex. Enable \ + elision as it is not enabled so far. \ + Note: It can happen that multiple threads are calling e.g. \ + pthread_mutex_lock at the same time as the first lock \ + operation for this mutex. Then elision is enabled for this \ + mutex by multiple threads. Storing with relaxed MO is enough \ + as all threads will store the same new value for the kind of \ + the mutex. But we have to ensure that we always use the \ + elision path regardless if this thread has enabled elision or \ + another one. \ + \ + - PTHREAD_MUTEX_ELISION_NP flag is set: \ + Elision was already enabled for this mutex by a previous lock \ + operation. See case above. Just use the elision path. \ + \ + - PTHREAD_MUTEX_NO_ELISION_NP flag is set: \ + Elision was explicitly disabled by pthread_mutexattr_settype. \ + Do not use the elision path. \ + Note: The flag PTHREAD_MUTEX_NO_ELISION_NP will never be \ + changed after mutex initialization. */ \ + int mutex_kind = atomic_load_relaxed (&((m)->__data.__kind)); \ + if ((mutex_kind & PTHREAD_MUTEX_ELISION_FLAGS_NP) == 0) \ + { \ + mutex_kind |= PTHREAD_MUTEX_ELISION_NP; \ + atomic_store_relaxed (&((m)->__data.__kind), mutex_kind); \ + } \ + if ((mutex_kind & PTHREAD_MUTEX_ELISION_NP) != 0) \ + { \ + s; \ + } \ + } + +#else /* !ENABLE_ELISION_SUPPORT */ + +#define lll_clocklock_elision(futex, adapt_count, clockid, abstime, private) \ + __futex_clocklock64 (&(futex), clockid, abstime, private) +# define lll_lock_elision(lock, try_lock, private) \ + ({ lll_lock (lock, private); 0; }) +# define lll_trylock_elision(a,t) lll_trylock(a) +# define lll_unlock_elision(a,b,c) ({ lll_unlock (a,c); 0; }) +# define FORCE_ELISION(m, s) + +#endif /* !ENABLE_ELISION_SUPPORT */ + #endif /* lowlevellock.h */ diff --git a/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_timedlock.c b/sysdeps/pthread/elision-conf.h similarity index 63% rename from sysdeps/unix/sysv/linux/powerpc/pthread_mutex_timedlock.c rename to sysdeps/pthread/elision-conf.h index 6f8b06d459..946f2c43b1 100644 --- a/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_timedlock.c +++ b/sysdeps/pthread/elision-conf.h @@ -1,5 +1,5 @@ -/* Elided version of pthread_mutex_timedlock. - Copyright (C) 2015-2021 Free Software Foundation, Inc. +/* elision-conf.h: Lock elision configuration. Stub version. + Copyright (C) 2021 Free Software Foundation, Inc. This file is part of the GNU C Library. The GNU C Library is free software; you can redistribute it and/or @@ -16,7 +16,14 @@ License along with the GNU C Library; if not, see . */ -#include -#include +#ifndef _ELISION_CONF_H +#define _ELISION_CONF_H 1 -#include +/* No elision support by default. */ +#define ENABLE_ELISION_SUPPORT 0 + +/* Whether __lll_unlock_elision expects a pointer argument to the + adaptive counter. Here, an unused arbitrary value. */ +#define ELISION_UNLOCK_NEEDS_ADAPT_COUNT 0 + +#endif diff --git a/sysdeps/unix/sysv/linux/powerpc/Makefile b/sysdeps/unix/sysv/linux/powerpc/Makefile index cc2f804d86..a093cda68b 100644 --- a/sysdeps/unix/sysv/linux/powerpc/Makefile +++ b/sysdeps/unix/sysv/linux/powerpc/Makefile @@ -32,7 +32,5 @@ endif ifeq ($(subdir),nptl) libpthread-routines += sysdep -libpthread-sysdep_routines += elision-lock elision-unlock elision-timed \ - elision-trylock libpthread-shared-only-routines += sysdep endif diff --git a/sysdeps/unix/sysv/linux/powerpc/elision-conf.c b/sysdeps/unix/sysv/linux/powerpc/elision-conf.c index 003bc82343..1ecc2a7b8f 100644 --- a/sysdeps/unix/sysv/linux/powerpc/elision-conf.c +++ b/sysdeps/unix/sysv/linux/powerpc/elision-conf.c @@ -51,12 +51,6 @@ struct elision_config __elision_aconf = .skip_trylock_internal_abort = 3, }; -/* Force elision for all new locks. This is used to decide whether existing - DEFAULT locks should be automatically use elision in pthread_mutex_lock(). - Disabled for suid programs. Only used when elision is available. */ - -int __pthread_force_elision attribute_hidden = 0; - #if HAVE_TUNABLES static inline void __always_inline @@ -104,10 +98,8 @@ TUNABLE_CALLBACK_FNDECL (skip_trylock_internal_abort, int32_t); /* Initialize elision. */ -static void -elision_init (int argc __attribute__ ((unused)), - char **argv __attribute__ ((unused)), - char **environ) +void +__lll_elision_init (void) { #if HAVE_TUNABLES /* Elision depends on tunables and must be explicitly turned on by setting @@ -150,17 +142,3 @@ elision_init (int argc __attribute__ ((unused)), if (!__pthread_force_elision) __elision_aconf.try_tbegin = 0; /* Disable elision on rwlocks. */ } - -#ifdef SHARED -# define INIT_SECTION ".init_array" -# define MAYBE_CONST -#else -# define INIT_SECTION ".preinit_array" -# define MAYBE_CONST const -#endif - -void (*MAYBE_CONST __pthread_init_array []) (int, char **, char **) - __attribute__ ((section (INIT_SECTION), aligned (sizeof (void *)))) = -{ - &elision_init -}; diff --git a/sysdeps/unix/sysv/linux/powerpc/elision-conf.h b/sysdeps/unix/sysv/linux/powerpc/elision-conf.h index 8c444d8695..86c677fb7a 100644 --- a/sysdeps/unix/sysv/linux/powerpc/elision-conf.h +++ b/sysdeps/unix/sysv/linux/powerpc/elision-conf.h @@ -22,6 +22,9 @@ #include #include +#define ENABLE_ELISION_SUPPORT 1 +#define ELISION_UNLOCK_NEEDS_ADAPT_COUNT 1 + /* Should make sure there is no false sharing on this. */ struct elision_config { @@ -34,9 +37,4 @@ struct elision_config extern struct elision_config __elision_aconf attribute_hidden; -extern int __pthread_force_elision attribute_hidden; - -/* Tell the test suite to test elision for this architecture. */ -#define HAVE_ELISION 1 - #endif diff --git a/sysdeps/unix/sysv/linux/powerpc/elision-lock.c b/sysdeps/unix/sysv/linux/powerpc/elision-lock.c index 767d439f88..f5fb46df25 100644 --- a/sysdeps/unix/sysv/linux/powerpc/elision-lock.c +++ b/sysdeps/unix/sysv/linux/powerpc/elision-lock.c @@ -23,12 +23,6 @@ #include #include "htm.h" -#if !defined(LLL_LOCK) && !defined(EXTRAARG) -/* Make sure the configuration code is always linked in for static - libraries. */ -#include "elision-conf.c" -#endif - #ifndef EXTRAARG # define EXTRAARG #endif @@ -84,3 +78,4 @@ __lll_lock_elision (int *lock, short *adapt_count, EXTRAARG int pshared) use_lock: return LLL_LOCK ((*lock), pshared); } +libc_hidden_def (__lll_lock_elision) diff --git a/sysdeps/unix/sysv/linux/powerpc/elision-trylock.c b/sysdeps/unix/sysv/linux/powerpc/elision-trylock.c index ab92f4d72f..5ebbced427 100644 --- a/sysdeps/unix/sysv/linux/powerpc/elision-trylock.c +++ b/sysdeps/unix/sysv/linux/powerpc/elision-trylock.c @@ -67,3 +67,4 @@ __lll_trylock_elision (int *futex, short *adapt_count) use_lock: return lll_trylock (*futex); } +libc_hidden_def (__lll_trylock_elision) diff --git a/sysdeps/unix/sysv/linux/powerpc/elision-unlock.c b/sysdeps/unix/sysv/linux/powerpc/elision-unlock.c index 35044af40b..5aa87521c5 100644 --- a/sysdeps/unix/sysv/linux/powerpc/elision-unlock.c +++ b/sysdeps/unix/sysv/linux/powerpc/elision-unlock.c @@ -41,3 +41,4 @@ __lll_unlock_elision (int *lock, short *adapt_count, int pshared) } return 0; } +libc_hidden_def (__lll_unlock_elision) diff --git a/sysdeps/unix/sysv/linux/powerpc/force-elision.h b/sysdeps/unix/sysv/linux/powerpc/force-elision.h deleted file mode 100644 index d1fa611c52..0000000000 --- a/sysdeps/unix/sysv/linux/powerpc/force-elision.h +++ /dev/null @@ -1,62 +0,0 @@ -/* force-elision.h: Automatic enabling of elision for mutexes - Copyright (C) 2015-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -/* Automatically enable elision for existing user lock kinds. */ -#define FORCE_ELISION(m, s) \ - if (__pthread_force_elision) \ - { \ - /* See concurrency notes regarding __kind in \ - struct __pthread_mutex_s in \ - sysdeps/nptl/bits/thread-shared-types.h. \ - \ - There are the following cases for the kind of a mutex \ - (The mask PTHREAD_MUTEX_ELISION_FLAGS_NP covers the flags \ - PTHREAD_MUTEX_ELISION_NP and PTHREAD_MUTEX_NO_ELISION_NP where \ - only one of both flags can be set): \ - - both flags are not set: \ - This is the first lock operation for this mutex. Enable \ - elision as it is not enabled so far. \ - Note: It can happen that multiple threads are calling e.g. \ - pthread_mutex_lock at the same time as the first lock \ - operation for this mutex. Then elision is enabled for this \ - mutex by multiple threads. Storing with relaxed MO is enough \ - as all threads will store the same new value for the kind of \ - the mutex. But we have to ensure that we always use the \ - elision path regardless if this thread has enabled elision or \ - another one. \ - \ - - PTHREAD_MUTEX_ELISION_NP flag is set: \ - Elision was already enabled for this mutex by a previous lock \ - operation. See case above. Just use the elision path. \ - \ - - PTHREAD_MUTEX_NO_ELISION_NP flag is set: \ - Elision was explicitly disabled by pthread_mutexattr_settype. \ - Do not use the elision path. \ - Note: The flag PTHREAD_MUTEX_NO_ELISION_NP will never be \ - changed after mutex initialization. */ \ - int mutex_kind = atomic_load_relaxed (&((m)->__data.__kind)); \ - if ((mutex_kind & PTHREAD_MUTEX_ELISION_FLAGS_NP) == 0) \ - { \ - mutex_kind |= PTHREAD_MUTEX_ELISION_NP; \ - atomic_store_relaxed (&((m)->__data.__kind), mutex_kind); \ - } \ - if ((mutex_kind & PTHREAD_MUTEX_ELISION_NP) != 0) \ - { \ - s; \ - } \ - } diff --git a/sysdeps/unix/sysv/linux/powerpc/lowlevellock.h b/sysdeps/unix/sysv/linux/powerpc/lowlevellock.h deleted file mode 100644 index 65962020c6..0000000000 --- a/sysdeps/unix/sysv/linux/powerpc/lowlevellock.h +++ /dev/null @@ -1,49 +0,0 @@ -/* PowerPC specific lock definitions. - Copyright (C) 2015-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#ifndef _POWERPC_LOWLEVELLOCK_H -#define _POWERPC_LOWLEVELLOCK_H 1 - -#include - -/* Transactional lock elision definitions. */ -extern int __lll_clocklock_elision - (int *futex, short *adapt_count, - clockid_t clockid, const struct __timespec64 *timeout, int private) - attribute_hidden; - -#define lll_clocklock_elision(futex, adapt_count, clockid, timeout, private) \ - __lll_clocklock_elision (&(futex), &(adapt_count), clockid, timeout, private) - -extern int __lll_lock_elision (int *futex, short *adapt_count, int private) - attribute_hidden; - -extern int __lll_unlock_elision (int *lock, short *adapt_count, int private) - attribute_hidden; - -extern int __lll_trylock_elision(int *lock, short *adapt_count) - attribute_hidden; - -#define lll_lock_elision(futex, adapt_count, private) \ - __lll_lock_elision (&(futex), &(adapt_count), private) -#define lll_unlock_elision(futex, adapt_count, private) \ - __lll_unlock_elision (&(futex), &(adapt_count), private) -#define lll_trylock_elision(futex, adapt_count) \ - __lll_trylock_elision (&(futex), &(adapt_count)) - -#endif diff --git a/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_cond_lock.c b/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_cond_lock.c deleted file mode 100644 index f33bac7ca1..0000000000 --- a/sysdeps/unix/sysv/linux/powerpc/pthread_mutex_cond_lock.c +++ /dev/null @@ -1,22 +0,0 @@ -/* Copyright (C) 2015-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -/* The cond lock is not actually elided yet, but we still need to handle - already elided locks. */ -#include - -#include diff --git a/sysdeps/unix/sysv/linux/s390/Makefile b/sysdeps/unix/sysv/linux/s390/Makefile index d9db1b5422..3de9579e23 100644 --- a/sysdeps/unix/sysv/linux/s390/Makefile +++ b/sysdeps/unix/sysv/linux/s390/Makefile @@ -12,9 +12,6 @@ gen-as-const-headers += ucontext_i.sym endif ifeq ($(subdir),nptl) -libpthread-sysdep_routines += elision-lock elision-unlock elision-timed \ - elision-trylock - elision-CFLAGS = -mhtm -msoft-float CFLAGS-elision-lock.c = $(elision-CFLAGS) CFLAGS-elision-timed.c = $(elision-CFLAGS) diff --git a/sysdeps/unix/sysv/linux/s390/elision-conf.c b/sysdeps/unix/sysv/linux/s390/elision-conf.c index c05c046e56..a7a90e73eb 100644 --- a/sysdeps/unix/sysv/linux/s390/elision-conf.c +++ b/sysdeps/unix/sysv/linux/s390/elision-conf.c @@ -51,13 +51,6 @@ struct elision_config __elision_aconf = .skip_trylock_internal_abort = 3, }; -/* Force elision for all new locks. This is used to decide whether existing - DEFAULT locks should be automatically upgraded to elision in - pthread_mutex_lock(). Disabled for suid programs. Only used when elision - is available. */ - -int __pthread_force_elision attribute_hidden = 0; - #if HAVE_TUNABLES static inline void __always_inline @@ -104,10 +97,8 @@ TUNABLE_CALLBACK_FNDECL (skip_trylock_internal_abort, int32_t); /* Initialize elison. */ -static void -elision_init (int argc __attribute__ ((unused)), - char **argv __attribute__ ((unused)), - char **environ) +void +__lll_elision_init (void) { #if HAVE_TUNABLES /* Elision depends on tunables and must be explicitly turned on by setting @@ -130,17 +121,3 @@ elision_init (int argc __attribute__ ((unused)), if (!__pthread_force_elision) __elision_aconf.try_tbegin = 0; /* Disable elision on rwlocks. */ } - -#ifdef SHARED -# define INIT_SECTION ".init_array" -# define MAYBE_CONST -#else -# define INIT_SECTION ".preinit_array" -# define MAYBE_CONST const -#endif - -void (*MAYBE_CONST __pthread_init_array []) (int, char **, char **) - __attribute__ ((section (INIT_SECTION), aligned (sizeof (void *)))) = -{ - &elision_init -}; diff --git a/sysdeps/unix/sysv/linux/s390/elision-conf.h b/sysdeps/unix/sysv/linux/s390/elision-conf.h index 846c5ff297..cb8a784ad0 100644 --- a/sysdeps/unix/sysv/linux/s390/elision-conf.h +++ b/sysdeps/unix/sysv/linux/s390/elision-conf.h @@ -21,6 +21,9 @@ #include #include +#define ENABLE_ELISION_SUPPORT 1 +#define ELISION_UNLOCK_NEEDS_ADAPT_COUNT 1 + /* Should make sure there is no false sharing on this. */ struct elision_config @@ -34,9 +37,4 @@ struct elision_config extern struct elision_config __elision_aconf attribute_hidden; -extern int __pthread_force_elision attribute_hidden; - -/* Tell the test suite to test elision for this architecture. */ -#define HAVE_ELISION 1 - #endif diff --git a/sysdeps/unix/sysv/linux/s390/elision-lock.c b/sysdeps/unix/sysv/linux/s390/elision-lock.c index 26474c6398..d5dd6023a0 100644 --- a/sysdeps/unix/sysv/linux/s390/elision-lock.c +++ b/sysdeps/unix/sysv/linux/s390/elision-lock.c @@ -23,12 +23,6 @@ #include #include -#if !defined(LLL_LOCK) && !defined(EXTRAARG) -/* Make sure the configuration code is always linked in for static - libraries. */ -#include "elision-conf.c" -#endif - #ifndef EXTRAARG #define EXTRAARG #endif @@ -123,3 +117,4 @@ __lll_lock_elision (int *futex, short *adapt_count, EXTRAARG int private) succeed. */ return LLL_LOCK ((*futex), private); } +libc_hidden_def (__lll_lock_elision) diff --git a/sysdeps/unix/sysv/linux/s390/elision-trylock.c b/sysdeps/unix/sysv/linux/s390/elision-trylock.c index 484c3cc913..2470955c89 100644 --- a/sysdeps/unix/sysv/linux/s390/elision-trylock.c +++ b/sysdeps/unix/sysv/linux/s390/elision-trylock.c @@ -95,3 +95,4 @@ __lll_trylock_elision (int *futex, short *adapt_count) succeed. */ return lll_trylock (*futex); } +libc_hidden_def (__lll_trylock_elision) diff --git a/sysdeps/unix/sysv/linux/s390/elision-unlock.c b/sysdeps/unix/sysv/linux/s390/elision-unlock.c index 5ac6758df2..87a5bd7cbf 100644 --- a/sysdeps/unix/sysv/linux/s390/elision-unlock.c +++ b/sysdeps/unix/sysv/linux/s390/elision-unlock.c @@ -59,3 +59,4 @@ __lll_unlock_elision(int *futex, short *adapt_count, int private) } return 0; } +libc_hidden_def (__lll_unlock_elision) diff --git a/sysdeps/unix/sysv/linux/s390/force-elision.h b/sysdeps/unix/sysv/linux/s390/force-elision.h deleted file mode 100644 index 154740d37f..0000000000 --- a/sysdeps/unix/sysv/linux/s390/force-elision.h +++ /dev/null @@ -1,62 +0,0 @@ -/* Automatic enabling of elision for mutexes - Copyright (C) 2014-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -/* Automatically enable elision for existing user lock kinds. */ -#define FORCE_ELISION(m, s) \ - if (__pthread_force_elision) \ - { \ - /* See concurrency notes regarding __kind in \ - struct __pthread_mutex_s in \ - sysdeps/nptl/bits/thread-shared-types.h. \ - \ - There are the following cases for the kind of a mutex \ - (The mask PTHREAD_MUTEX_ELISION_FLAGS_NP covers the flags \ - PTHREAD_MUTEX_ELISION_NP and PTHREAD_MUTEX_NO_ELISION_NP where \ - only one of both flags can be set): \ - - both flags are not set: \ - This is the first lock operation for this mutex. Enable \ - elision as it is not enabled so far. \ - Note: It can happen that multiple threads are calling e.g. \ - pthread_mutex_lock at the same time as the first lock \ - operation for this mutex. Then elision is enabled for this \ - mutex by multiple threads. Storing with relaxed MO is enough \ - as all threads will store the same new value for the kind of \ - the mutex. But we have to ensure that we always use the \ - elision path regardless if this thread has enabled elision or \ - another one. \ - \ - - PTHREAD_MUTEX_ELISION_NP flag is set: \ - Elision was already enabled for this mutex by a previous lock \ - operation. See case above. Just use the elision path. \ - \ - - PTHREAD_MUTEX_NO_ELISION_NP flag is set: \ - Elision was explicitly disabled by pthread_mutexattr_settype. \ - Do not use the elision path. \ - Note: The flag PTHREAD_MUTEX_NO_ELISION_NP will never be \ - changed after mutex initialization. */ \ - int mutex_kind = atomic_load_relaxed (&((m)->__data.__kind)); \ - if ((mutex_kind & PTHREAD_MUTEX_ELISION_FLAGS_NP) == 0) \ - { \ - mutex_kind |= PTHREAD_MUTEX_ELISION_NP; \ - atomic_store_relaxed (&((m)->__data.__kind), mutex_kind); \ - } \ - if ((mutex_kind & PTHREAD_MUTEX_ELISION_NP) != 0) \ - { \ - s; \ - } \ - } diff --git a/sysdeps/unix/sysv/linux/s390/lowlevellock.h b/sysdeps/unix/sysv/linux/s390/lowlevellock.h deleted file mode 100644 index 98d78f5270..0000000000 --- a/sysdeps/unix/sysv/linux/s390/lowlevellock.h +++ /dev/null @@ -1,49 +0,0 @@ -/* Copyright (C) 2003-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - Contributed by Martin Schwidefsky , 2003. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#ifndef _S390_LOWLEVELLOCK_H -#define _S390_LOWLEVELLOCK_H 1 - -#include - -/* Transactional lock elision definitions. */ -extern int __lll_clocklock_elision - (int *futex, short *adapt_count, - clockid_t clockid, const struct __timespec64 *timeout, int private) - attribute_hidden; - -# define lll_clocklock_elision(futex, adapt_count, clockid, timeout, private) \ - __lll_clocklock_elision (&(futex), &(adapt_count), clockid, timeout, private) - -extern int __lll_lock_elision (int *futex, short *adapt_count, int private) - attribute_hidden; - -extern int __lll_unlock_elision(int *futex, short *adapt_count, int private) - attribute_hidden; - -extern int __lll_trylock_elision(int *futex, short *adapt_count) - attribute_hidden; - -# define lll_lock_elision(futex, adapt_count, private) \ - __lll_lock_elision (&(futex), &(adapt_count), private) -# define lll_unlock_elision(futex, adapt_count, private) \ - __lll_unlock_elision (&(futex), &(adapt_count), private) -# define lll_trylock_elision(futex, adapt_count) \ - __lll_trylock_elision(&(futex), &(adapt_count)) - -#endif /* lowlevellock.h */ diff --git a/sysdeps/unix/sysv/linux/s390/pthread_mutex_cond_lock.c b/sysdeps/unix/sysv/linux/s390/pthread_mutex_cond_lock.c deleted file mode 100644 index 8398ff6425..0000000000 --- a/sysdeps/unix/sysv/linux/s390/pthread_mutex_cond_lock.c +++ /dev/null @@ -1,22 +0,0 @@ -/* Copyright (C) 2014-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -/* The cond lock is not actually elided yet, but we still need to handle - already elided locks. */ -#include - -#include diff --git a/sysdeps/unix/sysv/linux/s390/pthread_mutex_timedlock.c b/sysdeps/unix/sysv/linux/s390/pthread_mutex_timedlock.c deleted file mode 100644 index 8538f157a0..0000000000 --- a/sysdeps/unix/sysv/linux/s390/pthread_mutex_timedlock.c +++ /dev/null @@ -1,22 +0,0 @@ -/* Elided version of pthread_mutex_timedlock. - Copyright (C) 2014-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include -#include - -#include diff --git a/sysdeps/unix/sysv/linux/s390/pthread_mutex_trylock.c b/sysdeps/unix/sysv/linux/s390/pthread_mutex_trylock.c deleted file mode 100644 index c4de1363cb..0000000000 --- a/sysdeps/unix/sysv/linux/s390/pthread_mutex_trylock.c +++ /dev/null @@ -1,22 +0,0 @@ -/* Elided version of pthread_mutex_trylock. - Copyright (C) 2014-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include -#include - -#include diff --git a/sysdeps/unix/sysv/linux/x86/Makefile b/sysdeps/unix/sysv/linux/x86/Makefile index 6bfd6bec49..9dfdd689a9 100644 --- a/sysdeps/unix/sysv/linux/x86/Makefile +++ b/sysdeps/unix/sysv/linux/x86/Makefile @@ -12,8 +12,6 @@ sysdep_headers += sys/elf.h sys/perm.h sys/reg.h sys/vm86.h sys/debugreg.h sys/i endif ifeq ($(subdir),nptl) -libpthread-sysdep_routines += elision-lock elision-unlock elision-timed \ - elision-trylock CFLAGS-elision-lock.c += -mrtm CFLAGS-elision-unlock.c += -mrtm CFLAGS-elision-timed.c += -mrtm diff --git a/sysdeps/unix/sysv/linux/x86/elision-conf.c b/sysdeps/unix/sysv/linux/x86/elision-conf.c index e8ffe022f0..a5a4043d15 100644 --- a/sysdeps/unix/sysv/linux/x86/elision-conf.c +++ b/sysdeps/unix/sysv/linux/x86/elision-conf.c @@ -48,13 +48,6 @@ struct elision_config __elision_aconf = .skip_trylock_internal_abort = 3, }; -/* Force elision for all new locks. This is used to decide whether existing - DEFAULT locks should be automatically upgraded to elision in - pthread_mutex_lock(). Disabled for suid programs. Only used when elision - is available. */ - -int __pthread_force_elision attribute_hidden = 0; - #if HAVE_TUNABLES static __always_inline void do_set_elision_enable (int32_t elision_enable) @@ -98,10 +91,8 @@ TUNABLE_CALLBACK_FNDECL (skip_trylock_internal_abort, int32_t); /* Initialize elision. */ -static void -elision_init (int argc __attribute__ ((unused)), - char **argv __attribute__ ((unused)), - char **environ) +void +__lll_elision_init (void) { #if HAVE_TUNABLES /* Elision depends on tunables and must be explicitly turned on by setting @@ -122,15 +113,3 @@ elision_init (int argc __attribute__ ((unused)), if (!__pthread_force_elision) __elision_aconf.retry_try_xbegin = 0; /* Disable elision on rwlocks. */ } - -#ifdef SHARED -# define INIT_SECTION ".init_array" -#else -# define INIT_SECTION ".preinit_array" -#endif - -void (*const __pthread_init_array []) (int, char **, char **) - __attribute__ ((section (INIT_SECTION), aligned (sizeof (void *)))) = -{ - &elision_init -}; diff --git a/sysdeps/unix/sysv/linux/x86/elision-conf.h b/sysdeps/unix/sysv/linux/x86/elision-conf.h index bc5e73bd10..6016820312 100644 --- a/sysdeps/unix/sysv/linux/x86/elision-conf.h +++ b/sysdeps/unix/sysv/linux/x86/elision-conf.h @@ -21,6 +21,9 @@ #include #include +#define ENABLE_ELISION_SUPPORT 1 +#define ELISION_UNLOCK_NEEDS_ADAPT_COUNT 0 + /* Should make sure there is no false sharing on this. */ struct elision_config @@ -33,9 +36,4 @@ struct elision_config extern struct elision_config __elision_aconf attribute_hidden; -extern int __pthread_force_elision attribute_hidden; - -/* Tell the test suite to test elision for this architecture. */ -#define HAVE_ELISION 1 - #endif diff --git a/sysdeps/unix/sysv/linux/x86/elision-lock.c b/sysdeps/unix/sysv/linux/x86/elision-lock.c index 16d4f6e4f1..ea5f373c67 100644 --- a/sysdeps/unix/sysv/linux/x86/elision-lock.c +++ b/sysdeps/unix/sysv/linux/x86/elision-lock.c @@ -22,12 +22,6 @@ #include "hle.h" #include -#if !defined(LLL_LOCK) && !defined(EXTRAARG) -/* Make sure the configuration code is always linked in for static - libraries. */ -#include "elision-conf.c" -#endif - #ifndef EXTRAARG #define EXTRAARG #endif @@ -105,3 +99,4 @@ __lll_lock_elision (int *futex, short *adapt_count, EXTRAARG int private) /* Use a normal lock here. */ return LLL_LOCK ((*futex), private); } +libc_hidden_def (__lll_lock_elision) diff --git a/sysdeps/unix/sysv/linux/x86/elision-trylock.c b/sysdeps/unix/sysv/linux/x86/elision-trylock.c index ef9aa23819..5e689f577f 100644 --- a/sysdeps/unix/sysv/linux/x86/elision-trylock.c +++ b/sysdeps/unix/sysv/linux/x86/elision-trylock.c @@ -73,3 +73,4 @@ __lll_trylock_elision (int *futex, short *adapt_count) return lll_trylock (*futex); } +libc_hidden_def (__lll_trylock_elision) diff --git a/sysdeps/unix/sysv/linux/x86/elision-unlock.c b/sysdeps/unix/sysv/linux/x86/elision-unlock.c index edd8b1fd08..eb877bc18c 100644 --- a/sysdeps/unix/sysv/linux/x86/elision-unlock.c +++ b/sysdeps/unix/sysv/linux/x86/elision-unlock.c @@ -31,3 +31,4 @@ __lll_unlock_elision(int *lock, int private) lll_unlock ((*lock), private); return 0; } +libc_hidden_def (__lll_unlock_elision) diff --git a/sysdeps/unix/sysv/linux/x86/force-elision.h b/sysdeps/unix/sysv/linux/x86/force-elision.h deleted file mode 100644 index 3c0a0e0ac5..0000000000 --- a/sysdeps/unix/sysv/linux/x86/force-elision.h +++ /dev/null @@ -1,62 +0,0 @@ -/* force-elision.h: Automatic enabling of elision for mutexes - Copyright (C) 2013-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -/* Automatically enable elision for existing user lock kinds. */ -#define FORCE_ELISION(m, s) \ - if (__pthread_force_elision) \ - { \ - /* See concurrency notes regarding __kind in \ - struct __pthread_mutex_s in \ - sysdeps/nptl/bits/thread-shared-types.h. \ - \ - There are the following cases for the kind of a mutex \ - (The mask PTHREAD_MUTEX_ELISION_FLAGS_NP covers the flags \ - PTHREAD_MUTEX_ELISION_NP and PTHREAD_MUTEX_NO_ELISION_NP where \ - only one of both flags can be set): \ - - both flags are not set: \ - This is the first lock operation for this mutex. Enable \ - elision as it is not enabled so far. \ - Note: It can happen that multiple threads are calling e.g. \ - pthread_mutex_lock at the same time as the first lock \ - operation for this mutex. Then elision is enabled for this \ - mutex by multiple threads. Storing with relaxed MO is enough \ - as all threads will store the same new value for the kind of \ - the mutex. But we have to ensure that we always use the \ - elision path regardless if this thread has enabled elision or \ - another one. \ - \ - - PTHREAD_MUTEX_ELISION_NP flag is set: \ - Elision was already enabled for this mutex by a previous lock \ - operation. See case above. Just use the elision path. \ - \ - - PTHREAD_MUTEX_NO_ELISION_NP flag is set: \ - Elision was explicitly disabled by pthread_mutexattr_settype. \ - Do not use the elision path. \ - Note: The flag PTHREAD_MUTEX_NO_ELISION_NP will never be \ - changed after mutex initialization. */ \ - int mutex_kind = atomic_load_relaxed (&((m)->__data.__kind)); \ - if ((mutex_kind & PTHREAD_MUTEX_ELISION_FLAGS_NP) == 0) \ - { \ - mutex_kind |= PTHREAD_MUTEX_ELISION_NP; \ - atomic_store_relaxed (&((m)->__data.__kind), mutex_kind); \ - } \ - if ((mutex_kind & PTHREAD_MUTEX_ELISION_NP) != 0) \ - { \ - s; \ - } \ - } diff --git a/sysdeps/unix/sysv/linux/x86/lowlevellock.h b/sysdeps/unix/sysv/linux/x86/lowlevellock.h index 435a190ec9..7205bcadfd 100644 --- a/sysdeps/unix/sysv/linux/x86/lowlevellock.h +++ b/sysdeps/unix/sysv/linux/x86/lowlevellock.h @@ -82,30 +82,6 @@ __lll_cas_lock (int *futex) __lll_unlock (&(lock), private); \ })) -extern int __lll_clocklock_elision (int *futex, short *adapt_count, - clockid_t clockid, - const struct __timespec64 *timeout, - int private) attribute_hidden; - -#define lll_clocklock_elision(futex, adapt_count, clockid, timeout, private) \ - __lll_clocklock_elision (&(futex), &(adapt_count), clockid, timeout, private) - -extern int __lll_lock_elision (int *futex, short *adapt_count, int private) - attribute_hidden; - -extern int __lll_unlock_elision (int *lock, int private) - attribute_hidden; - -extern int __lll_trylock_elision (int *lock, short *adapt_count) - attribute_hidden; - -#define lll_lock_elision(futex, adapt_count, private) \ - __lll_lock_elision (&(futex), &(adapt_count), private) -#define lll_unlock_elision(futex, adapt_count, private) \ - __lll_unlock_elision (&(futex), private) -#define lll_trylock_elision(futex, adapt_count) \ - __lll_trylock_elision (&(futex), &(adapt_count)) - #endif /* !__ASSEMBLER__ */ #endif /* lowlevellock.h */ diff --git a/sysdeps/unix/sysv/linux/x86/pthread_mutex_cond_lock.c b/sysdeps/unix/sysv/linux/x86/pthread_mutex_cond_lock.c deleted file mode 100644 index 7604f8706f..0000000000 --- a/sysdeps/unix/sysv/linux/x86/pthread_mutex_cond_lock.c +++ /dev/null @@ -1,22 +0,0 @@ -/* Copyright (C) 2013-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -/* The cond lock is not actually elided yet, but we still need to handle - already elided locks. */ -#include - -#include diff --git a/sysdeps/unix/sysv/linux/x86/pthread_mutex_timedlock.c b/sysdeps/unix/sysv/linux/x86/pthread_mutex_timedlock.c deleted file mode 100644 index 3fbfe02690..0000000000 --- a/sysdeps/unix/sysv/linux/x86/pthread_mutex_timedlock.c +++ /dev/null @@ -1,22 +0,0 @@ -/* Elided version of pthread_mutex_timedlock. - Copyright (C) 2011-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include -#include "force-elision.h" - -#include "nptl/pthread_mutex_timedlock.c" diff --git a/sysdeps/unix/sysv/linux/x86/pthread_mutex_trylock.c b/sysdeps/unix/sysv/linux/x86/pthread_mutex_trylock.c deleted file mode 100644 index c7abb5ad4f..0000000000 --- a/sysdeps/unix/sysv/linux/x86/pthread_mutex_trylock.c +++ /dev/null @@ -1,22 +0,0 @@ -/* Elided version of pthread_mutex_trylock. - Copyright (C) 2011-2021 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library; if not, see - . */ - -#include -#include "force-elision.h" - -#include "nptl/pthread_mutex_trylock.c"