From patchwork Fri May 9 13:58:44 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Liebler X-Patchwork-Id: 347425 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 0972914011D for ; Fri, 9 May 2014 23:59:20 +1000 (EST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:to:from:subject:date:message-id:references :mime-version:content-type:in-reply-to; q=dns; s=default; b=gG+Z Yo4aMgTkhX985Dl74nHW6N3bjfJ3YkLx0WPwCSliqVaL4X49MnvLsmdzlokk7q7p LJf61x1aLvxiGD8xcZz3+T0XEI0Mwsd18r+7d212d4iSaWlyI4hRJ8nw+O1O57T0 72cKs5niX2Ef+apMeFZqUWLsGdFskMYhko/NUQ4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:to:from:subject:date:message-id:references :mime-version:content-type:in-reply-to; s=default; bh=XDIbBgERi5 2R7ipxUOtrtFQig0I=; b=aZUnDahAGCsyGq0PoUhjVXQ5gTJ8LH6/benPw1+Kuy 9AdvmGU09Xfsxt3ZQ1xKLiJZ4YXwDcEGo7FmxCi3F7/wuG43l1K6RaIbrgRlmg7T ALvVkZWqZF5QDk2MBvnLJamSZCQBY/NvfBlVPhSl4HbSWB66DyHH5Y2uXJB9iuYl U= Received: (qmail 30548 invoked by alias); 9 May 2014 13:59:14 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 30532 invoked by uid 89); 9 May 2014 13:59:13 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.4 required=5.0 tests=AWL, BAYES_00, RCVD_IN_DNSWL_NONE, RP_MATCHES_RCVD, SPF_HELO_PASS, SPF_PASS autolearn=ham version=3.3.2 X-HELO: plane.gmane.org To: libc-alpha@sourceware.org From: Stefan Liebler Subject: Re: [PATCH] S/390: Port of lock elision to System/z Date: Fri, 09 May 2014 15:58:44 +0200 Lines: 730 Message-ID: References: <535979FD.9060407@linux.vnet.ibm.com> <536108A2.6040903@linux.vnet.ibm.com> <536B520D.4000108@linux.vnet.ibm.com> Mime-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 In-Reply-To: <536B520D.4000108@linux.vnet.ibm.com> Hi, thanks for the review. I´ve changed the mentioned points. The Changelog remains the same. Bye On 05/08/2014 11:44 AM, Andreas Krebbel wrote: > On 04/30/2014 04:28 PM, Stefan Liebler wrote: >> Hi, >> >> like mentioned in the discussion before, the clobbering of >> FPRs/cc/memory is not needed anymore. In an older patch-version, the >> transaction was either started by the builtin_tbegin or an asm("tbegin"). >> In the latter case, the clobbering of FPRs was needed. >> There was also a non-transactional store, which is not needed anymore. >> The objdumps of this patch-version with and without the clobbering does >> not differ. >> >> The file nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_unlock.c is not >> added anymore, because it only includes the nptl/pthread_mutex_unlock.c. >> >> Ok? >> >> Bye >> >> --- >> 2014-04-30 Dominik Vogt >> Stefan Liebler >> >> * config.make.in (enable-lock-elision): New Makefile variable. >> * configure.ac: Likewise. >> * configure: Regenerate. >> * sysdeps/s390/configure.ac: >> Add check for gcc transactions support. >> * sysdeps/s390/configure: Regenerate. >> * nptl/sysdeps/unix/sysv/linux/s390/Makefile: New file. >> Build elision files if enabled. >> * nptl/sysdeps/unix/sysv/linux/s390/elision-conf.c: New file. >> Add lock elision support for s390. >> * nptl/sysdeps/unix/sysv/linux/s390/elision-conf.h: Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/elision-lock.c: Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/elision-timed.c: Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/elision-trylock.c: Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/elision-unlock.c: Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/force-elision.h: Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_cond_lock.c: >> Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_lock.c: >> Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_timedlock.c: >> Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_trylock.c: >> Likewise. >> * nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h: >> (__lll_timedlock_elision, __lll_lock_elision) >> (__lll_unlock_elision, __lll_trylock_elision) >> (lll_timedlock_elision, lll_lock_elision) >> (lll_unlock_elision, lll_trylock_elision): Add. >> * nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h >> (pthread_mutex_t): Add lock elision support for s390. >> --- >> > > Just some (very) minor nitpicking: > > nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h: > > +#ifdef ENABLE_LOCK_ELISION > +extern int __lll_lock_elision (int *futex, short *adapt_count, int private) > + attribute_hidden; > + > +extern int __lll_unlock_elision(int *lock, int private) > + attribute_hidden; > + > +extern int __lll_trylock_elision(int *lock, short *adapt_count) > + attribute_hidden; > > What's the reason for calling the first argument futex once and lock > in the other cases? No. Now the argument is called futex in all cases. Same for nptl/sysdeps/unix/sysv/linux/s390/elision-unlock.c. > > nptl/sysdeps/unix/sysv/linux/s390/elision-conf.c: > > +{ > + /* Set when the CPU supports elision. When false elision is never > attempted. */ > ... when the CPU and the kernel support transactional execution. ... > > + int elision_available = (GLRO (dl_hwcap) & HWCAP_S390_TE) ? 1 : 0; > ok > > nptl/sysdeps/unix/sysv/linux/s390/elision-lock.c: > > +int > +__lll_lock_elision (int *futex, short *adapt_count, EXTRAARG int private) > +{ > + if (*adapt_count <= 0) > + { > ... > + { > + else > + { > + /* Lost updates are possible, but harmless. Due to races this might lead > + to *adapt_count becoming less than zero. */ > + (*adapt_count)--; > + } > + > + use_lock: > + return LLL_LOCK ((*futex), private); > +} > > Perhaps something like this to reduce indentation of the whole function?! > > +int > +__lll_lock_elision (int *futex, short *adapt_count, EXTRAARG int private) > +{ > + if (*adapt_count > 0) > + { > + /* Lost updates are possible, but harmless. Due to races this might lead > + to *adapt_count becoming less than zero. */ > + (*adapt_count)--; > + goto use_lock; > + } > ... > + > + use_lock: > + return LLL_LOCK ((*futex), private); > +} ok > > + /* Same logic as above, but for for a number of tepmorary failures in a > + row. */ > > ... temporary ... > ok > nptl/sysdeps/unix/sysv/linux/s390/elision-trylock.c: > > + /* Implement POSIX semantics by forbiding nesting elided trylocks. > ... forbidding ... > > + __builtin_tabort (_HTM_FIRST_USER_ABORT_CODE + 1); > > Here you want to set the lowest order bit in order to enforce a > persistent abort. While the code works fine since > _HTM_FIRST_USER_ABORT_CODE is even (_HTM_FIRST_USER_ABORT_CODE | 1) > would appear more natural to me. The same appears in elision-lock.c. ok > > Bye, > > -Andreas- > > diff --git a/config.make.in b/config.make.in index 132d179..6bcab8a 100644 --- a/config.make.in +++ b/config.make.in @@ -93,6 +93,7 @@ build-nscd = @build_nscd@ use-nscd = @use_nscd@ build-hardcoded-path-in-tests= @hardcoded_path_in_tests@ build-pt-chown = @build_pt_chown@ +enable-lock-elision = @enable_lock_elision@ # Build tools. CC = @CC@ diff --git a/configure b/configure index 8b0b222..ecc282b 100755 --- a/configure +++ b/configure @@ -651,6 +651,7 @@ libc_cv_nss_crypt all_warnings force_install bindnow +enable_lock_elision hardcoded_path_in_tests oldest_abi use_default_link @@ -3476,6 +3477,7 @@ else enable_lock_elision=no fi + if test "$enable_lock_elision" = yes ; then $as_echo "#define ENABLE_LOCK_ELISION 1" >>confdefs.h diff --git a/configure.ac b/configure.ac index 97a9591..babfe57 100644 --- a/configure.ac +++ b/configure.ac @@ -184,6 +184,7 @@ AC_ARG_ENABLE([lock-elision], [Enable lock elision for pthread mutexes by default]), [enable_lock_elision=$enableval], [enable_lock_elision=no]) +AC_SUBST(enable_lock_elision) if test "$enable_lock_elision" = yes ; then AC_DEFINE(ENABLE_LOCK_ELISION) fi diff --git a/nptl/sysdeps/unix/sysv/linux/s390/Makefile b/nptl/sysdeps/unix/sysv/linux/s390/Makefile new file mode 100644 index 0000000..269832f --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/Makefile @@ -0,0 +1,10 @@ +ifeq ($(enable-lock-elision),yes) +libpthread-sysdep_routines += elision-lock elision-unlock elision-timed \ + elision-trylock + +elision-CFLAGS = -mhtm +CFLAGS-elision-lock.c = $(elision-CFLAGS) +CFLAGS-elision-timed.c = $(elision-CFLAGS) +CFLAGS-elision-trylock.c = $(elision-CFLAGS) +CFLAGS-elision-unlock.c = $(elision-CFLAGS) +endif diff --git a/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h b/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h index 8264de0..d70f8b3 100644 --- a/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h +++ b/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h @@ -89,14 +89,37 @@ typedef union binary compatibility. */ int __kind; #if __WORDSIZE == 64 +# ifdef ENABLE_LOCK_ELISION + short __spins; + short __elision; + /* Mutex __spins initializer used by PTHREAD_MUTEX_INITIALIZER. */ +# define __PTHREAD_SPINS 0, 0 +# else int __spins; + /* Mutex __spins initializer used by PTHREAD_MUTEX_INITIALIZER. */ +# define __PTHREAD_SPINS 0 +# endif __pthread_list_t __list; # define __PTHREAD_MUTEX_HAVE_PREV 1 #else unsigned int __nusers; __extension__ union { +# ifdef ENABLE_LOCK_ELISION + struct + { + short __espins; + short __elision; + } _d; +# define __spins _d.__espins +# define __elision _d.__elision + /* Mutex __spins initializer used by PTHREAD_MUTEX_INITIALIZER. */ +# define __PTHREAD_SPINS { 0, 0 } +# else int __spins; + /* Mutex __spins initializer used by PTHREAD_MUTEX_INITIALIZER. */ +# define __PTHREAD_SPINS 0 +# endif __pthread_slist_t __list; }; #endif @@ -105,8 +128,6 @@ typedef union long int __align; } pthread_mutex_t; -/* Mutex __spins initializer used by PTHREAD_MUTEX_INITIALIZER. */ -#define __PTHREAD_SPINS 0 typedef union { diff --git a/nptl/sysdeps/unix/sysv/linux/s390/elision-conf.c b/nptl/sysdeps/unix/sysv/linux/s390/elision-conf.c new file mode 100644 index 0000000..69c0483 --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/elision-conf.c @@ -0,0 +1,82 @@ +/* Lock elision tunable parameters. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include +#include + +/* Reasonable initial tuning values, may be revised in the future. + This is a conservative initial value. */ + +struct elision_config __elision_aconf = + { + /* How often to not attempt to use elision if a transaction aborted + because the lock is already acquired. Expressed in number of lock + acquisition attempts. */ + .skip_lock_busy = 3, + /* How often to not attempt to use elision if a transaction aborted due + to reasons other than other threads' memory accesses. Expressed in + number of lock acquisition attempts. */ + .skip_lock_internal_abort = 3, + /* How often to not attempt to use elision if a lock used up all retries + without success. Expressed in number of lock acquisition attempts. */ + .skip_lock_out_of_tbegin_retries = 3, + /* How often we try using elision if there is chance for the transaction + to finish execution (e.g., it wasn't aborted due to the lock being + already acquired. */ + .try_tbegin = 3, + /* Same as SKIP_LOCK_INTERNAL_ABORT but for trylock. */ + .skip_trylock_internal_abort = 3, + }; + +/* Force elision for all new locks. This is used to decide whether existing + DEFAULT locks should be automatically upgraded to elision in + pthread_mutex_lock(). Disabled for suid programs. Only used when elision + is available. */ + +int __pthread_force_elision attribute_hidden = 0; + +/* Initialize elison. */ + +static void +elision_init (int argc __attribute__ ((unused)), + char **argv __attribute__ ((unused)), + char **environ) +{ + /* Set when the CPU and the kernel supports transactional execution. + When false elision is never attempted. */ + int elision_available = (GLRO (dl_hwcap) & HWCAP_S390_TE) ? 1 : 0; + + __pthread_force_elision = __libc_enable_secure ? 0 : elision_available; +} + +#ifdef SHARED +# define INIT_SECTION ".init_array" +# define MAYBE_CONST +#else +# define INIT_SECTION ".preinit_array" +# define MAYBE_CONST const +#endif + +void (*MAYBE_CONST __pthread_init_array []) (int, char **, char **) + __attribute__ ((section (INIT_SECTION), aligned (sizeof (void *)))) = +{ + &elision_init +}; diff --git a/nptl/sysdeps/unix/sysv/linux/s390/elision-conf.h b/nptl/sysdeps/unix/sysv/linux/s390/elision-conf.h new file mode 100644 index 0000000..d9e9794 --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/elision-conf.h @@ -0,0 +1,44 @@ +/* Lock elision tunable parameters. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ +#ifdef ENABLE_LOCK_ELISION +#ifndef _ELISION_CONF_H +#define _ELISION_CONF_H 1 + +#include +#include + +/* Should make sure there is no false sharing on this. */ + +struct elision_config +{ + int skip_lock_busy; + int skip_lock_internal_abort; + int skip_lock_out_of_tbegin_retries; + int try_tbegin; + int skip_trylock_internal_abort; +}; + +extern struct elision_config __elision_aconf attribute_hidden; + +extern int __pthread_force_elision attribute_hidden; + +/* Tell the test suite to test elision for this architecture. */ +#define HAVE_ELISION 1 + +#endif +#endif diff --git a/nptl/sysdeps/unix/sysv/linux/s390/elision-lock.c b/nptl/sysdeps/unix/sysv/linux/s390/elision-lock.c new file mode 100644 index 0000000..ba5338f --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/elision-lock.c @@ -0,0 +1,119 @@ +/* Elided pthread mutex lock. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include +#include +#include + +#if !defined(LLL_LOCK) && !defined(EXTRAARG) +/* Make sure the configuration code is always linked in for static + libraries. */ +#include "elision-conf.c" +#endif + +#ifndef EXTRAARG +#define EXTRAARG +#endif +#ifndef LLL_LOCK +#define LLL_LOCK(a,b) lll_lock(a,b), 0 +#endif + +#define aconf __elision_aconf + +/* Adaptive lock using transactions. + By default the lock region is run as a transaction, and when it + aborts or the lock is busy the lock adapts itself. */ + +int +__lll_lock_elision (int *futex, short *adapt_count, EXTRAARG int private) +{ + if (*adapt_count > 0) + { + /* Lost updates are possible, but harmless. Due to races this might lead + to *adapt_count becoming less than zero. */ + (*adapt_count)--; + goto use_lock; + } + + __asm__ volatile (".machinemode \"zarch_nohighgprs\"\n\t" + ".machine \"all\"" + : : : "memory"); + + int try_tbegin; + for (try_tbegin = aconf.try_tbegin; + try_tbegin > 0; + try_tbegin--) + { + unsigned status; + if (__builtin_expect + ((status = __builtin_tbegin((void *)0)) == _HTM_TBEGIN_STARTED, 1)) + { + if (*futex == 0) + return 0; + /* Lock was busy. Fall back to normal locking. */ + if (__builtin_expect (__builtin_tx_nesting_depth (), 1)) + { + /* In a non-nested transaction there is no need to abort, + which is expensive. */ + __builtin_tend (); + if (aconf.skip_lock_busy > 0) + *adapt_count = aconf.skip_lock_busy; + goto use_lock; + } + else /* nesting depth is > 1 */ + { + /* A nested transaction will abort eventually because it + cannot make any progress before *futex changes back to 0. + So we may as well abort immediately. + This persistently aborts the outer transaction to force + the outer mutex use the default lock instead of retrying + with transactions until the try_tbegin of the outer mutex + is zero. + The adapt_count of this inner mutex is not changed, + because using the default lock with the inner mutex + would abort the outer transaction. + */ + __builtin_tabort (_HTM_FIRST_USER_ABORT_CODE | 1); + } + } + else + { + if (status != _HTM_TBEGIN_TRANSIENT) + { + /* A persistent abort (cc 1 or 3) indicates that a retry is + probably futile. Use the normal locking now and for the + next couple of calls. + Be careful to avoid writing to the lock. */ + if (aconf.skip_lock_internal_abort > 0) + *adapt_count = aconf.skip_lock_internal_abort; + goto use_lock; + } + } + } + + /* Same logic as above, but for for a number of temporary failures in a + row. */ + if (aconf.skip_lock_out_of_tbegin_retries > 0 && aconf.try_tbegin > 0) + *adapt_count = aconf.skip_lock_out_of_tbegin_retries; + + use_lock: + return LLL_LOCK ((*futex), private); +} diff --git a/nptl/sysdeps/unix/sysv/linux/s390/elision-timed.c b/nptl/sysdeps/unix/sysv/linux/s390/elision-timed.c new file mode 100644 index 0000000..a8d8b2a --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/elision-timed.c @@ -0,0 +1,26 @@ +/* Lock elision timed lock. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#define __lll_lock_elision __lll_timedlock_elision +#define EXTRAARG const struct timespec *t, +#undef LLL_LOCK +#define LLL_LOCK(a, b) lll_timedlock(a, t, b) +#include "elision-lock.c" diff --git a/nptl/sysdeps/unix/sysv/linux/s390/elision-trylock.c b/nptl/sysdeps/unix/sysv/linux/s390/elision-trylock.c new file mode 100644 index 0000000..61447d6 --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/elision-trylock.c @@ -0,0 +1,94 @@ +/* Elided pthread mutex trylock. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include +#include + +#define aconf __elision_aconf + +/* Try to elide a futex trylock. FUTEX is the futex variable. ADAPT_COUNT is + the adaptation counter in the mutex. */ + +int +__lll_trylock_elision (int *futex, short *adapt_count) +{ + __asm__ volatile (".machinemode \"zarch_nohighgprs\"\n\t" + ".machine \"all\"" + : : : "memory"); + + /* Implement POSIX semantics by forbiding nesting elided trylocks. + Sorry. After the abort the code is re-executed + non transactional and if the lock was already locked + return an error. */ + if (__builtin_tx_nesting_depth () > 0) + { + /* Note that this abort may terminate an outermost transaction that + was created outside glibc. + This persistently aborts the current transactions to force + them to use the default lock instead of retrying transactions + until their try_tbegin is zero. + */ + __builtin_tabort (_HTM_FIRST_USER_ABORT_CODE | 1); + } + + /* Only try a transaction if it's worth it. */ + if (*adapt_count <= 0) + { + unsigned status; + + if (__builtin_expect + ((status = __builtin_tbegin ((void *)0)) == _HTM_TBEGIN_STARTED, 1)) + { + if (*futex == 0) + return 0; + /* Lock was busy. Fall back to normal locking. */ + /* Since we are in a non-nested transaction there is no need to abort, + which is expensive. */ + __builtin_tend (); + /* Note: Changing the adapt_count here might abort a transaction on a + different cpu, but that could happen anyway when the futex is + acquired, so there's no need to check the nesting depth here. */ + if (aconf.skip_lock_busy > 0) + *adapt_count = aconf.skip_lock_busy; + } + else + { + if (status != _HTM_TBEGIN_TRANSIENT) + { + /* A persistent abort (cc 1 or 3) indicates that a retry is + probably futile. Use the normal locking now and for the + next couple of calls. + Be careful to avoid writing to the lock. */ + if (aconf.skip_trylock_internal_abort > 0) + *adapt_count = aconf.skip_trylock_internal_abort; + } + } + /* Could do some retries here. */ + } + else + { + /* Lost updates are possible, but harmless. Due to races this might lead + to *adapt_count becoming less than zero. */ + (*adapt_count)--; + } + + return lll_trylock (*futex); +} diff --git a/nptl/sysdeps/unix/sysv/linux/s390/elision-unlock.c b/nptl/sysdeps/unix/sysv/linux/s390/elision-unlock.c new file mode 100644 index 0000000..9ceae3e --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/elision-unlock.c @@ -0,0 +1,38 @@ +/* Commit an elided pthread lock. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +int +__lll_unlock_elision(int *futex, int private) +{ + /* If the lock is free, we elided the lock earlier. This does not + necessarily mean that we are in a transaction, because the user code may + have closed the transaction, but that is impossible to detect reliably. */ + if (*futex == 0) + { + __asm__ volatile (".machinemode \"zarch_nohighgprs\"\n\t" + ".machine \"all\"" + : : : "memory"); + __builtin_tend(); + } + else + lll_unlock ((*futex), private); + return 0; +} diff --git a/nptl/sysdeps/unix/sysv/linux/s390/force-elision.h b/nptl/sysdeps/unix/sysv/linux/s390/force-elision.h new file mode 100644 index 0000000..8fd7684 --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/force-elision.h @@ -0,0 +1,33 @@ +/* Automatic enabling of elision for mutexes + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifdef ENABLE_LOCK_ELISION +/* Check for elision on this lock without upgrading. */ +#define DO_ELISION(m) \ + (__pthread_force_elision \ + && (m->__data.__kind & PTHREAD_MUTEX_NO_ELISION_NP) == 0) \ + +/* Automatically enable elision for existing user lock kinds. */ +#define FORCE_ELISION(m, s) \ + if (__pthread_force_elision \ + && (m->__data.__kind & PTHREAD_MUTEX_ELISION_FLAGS_NP) == 0) \ + { \ + mutex->__data.__kind |= PTHREAD_MUTEX_ELISION_NP; \ + s; \ + } +#endif diff --git a/nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h b/nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h index 864dcbc..cabff30 100644 --- a/nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h +++ b/nptl/sysdeps/unix/sysv/linux/s390/lowlevellock.h @@ -285,6 +285,15 @@ __lll_timedlock (int *futex, const struct timespec *abstime, int private) #define lll_timedlock(futex, abstime, private) \ __lll_timedlock (&(futex), abstime, private) +#ifdef ENABLE_LOCK_ELISION +extern int __lll_timedlock_elision + (int *futex, short *adapt_count, const struct timespec *timeout, int private) + attribute_hidden; + +# define lll_timedlock_elision(futex, adapt_count, timeout, private) \ + __lll_timedlock_elision(&(futex), &(adapt_count), timeout, private) +#endif + static inline int __attribute__ ((always_inline)) __lll_robust_timedlock (int *futex, const struct timespec *abstime, @@ -360,4 +369,22 @@ extern int __lll_timedwait_tid (int *, const struct timespec *) __res; \ }) +#ifdef ENABLE_LOCK_ELISION +extern int __lll_lock_elision (int *futex, short *adapt_count, int private) + attribute_hidden; + +extern int __lll_unlock_elision(int *futex, int private) + attribute_hidden; + +extern int __lll_trylock_elision(int *futex, short *adapt_count) + attribute_hidden; + +# define lll_lock_elision(futex, adapt_count, private) \ + __lll_lock_elision (&(futex), &(adapt_count), private) +# define lll_unlock_elision(futex, private) \ + __lll_unlock_elision (&(futex), private) +# define lll_trylock_elision(futex, adapt_count) \ + __lll_trylock_elision(&(futex), &(adapt_count)) +#endif + #endif /* lowlevellock.h */ diff --git a/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_cond_lock.c b/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_cond_lock.c new file mode 100644 index 0000000..6fc0f96 --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_cond_lock.c @@ -0,0 +1,22 @@ +/* Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* The cond lock is not actually elided yet, but we still need to handle + already elided locks. */ +#include + +#include diff --git a/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_lock.c b/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_lock.c new file mode 100644 index 0000000..6fd6a98 --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_lock.c @@ -0,0 +1,22 @@ +/* Elided version of pthread_mutex_lock. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +#include diff --git a/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_timedlock.c b/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_timedlock.c new file mode 100644 index 0000000..d0e6537 --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_timedlock.c @@ -0,0 +1,22 @@ +/* Elided version of pthread_mutex_timedlock. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +#include diff --git a/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_trylock.c b/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_trylock.c new file mode 100644 index 0000000..ea8a8ff --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/s390/pthread_mutex_trylock.c @@ -0,0 +1,22 @@ +/* Elided version of pthread_mutex_trylock. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +#include diff --git a/sysdeps/s390/configure b/sysdeps/s390/configure index c2d05f7..6948cc2 100644 --- a/sysdeps/s390/configure +++ b/sysdeps/s390/configure @@ -68,5 +68,41 @@ if test $ac_verc_fail = yes; then fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __builtin_tbegin" >&5 +$as_echo_n "checking for __builtin_tbegin... " >&6; } +if ${libc_cv_gcc_builtin_tbegin+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat > conftest.c <<\EOF +#include +void testtransaction () +{ + if (__builtin_tbegin (0) == _HTM_TBEGIN_STARTED) + { + __builtin_tend (); + } +} +EOF +if { ac_try='${CC-cc} -mhtm -O2 -S conftest.c -o - | grep -w tbegin > /dev/null' + { { eval echo "\"\$as_me\":${as_lineno-$LINENO}: \"$ac_try\""; } >&5 + (eval $ac_try) 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; } ; +then + libc_cv_gcc_builtin_tbegin=yes +else + libc_cv_gcc_builtin_tbegin=no +fi +rm -f conftest* +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $libc_cv_gcc_builtin_tbegin" >&5 +$as_echo "$libc_cv_gcc_builtin_tbegin" >&6; } + +if test "$enable_lock_elision" = yes && test "$libc_cv_gcc_builtin_tbegin" = no ; then + critic_missing="$critic_missing The used GCC has no support for __builtin_tbegin, which is needed for lock-elision on target S390." +fi + test -n "$critic_missing" && as_fn_error $? " *** $critic_missing" "$LINENO" 5 diff --git a/sysdeps/s390/configure.ac b/sysdeps/s390/configure.ac index 59cfdd1..493e9a4 100644 --- a/sysdeps/s390/configure.ac +++ b/sysdeps/s390/configure.ac @@ -10,5 +10,31 @@ AC_CHECK_PROG_VER(AS, $AS, --version, [GNU assembler.* \([0-9]*\.[0-9.]*\)], [2.2[4-9]*|2.[3-9][0-9]*|[3-9].*|[1-9][0-9]*], critic_missing="$critic_missing The program AS is required in version >= 2.24 for target S390.") + +AC_CACHE_CHECK(for __builtin_tbegin, libc_cv_gcc_builtin_tbegin, [dnl +cat > conftest.c <<\EOF +#include +void testtransaction () +{ + if (__builtin_tbegin (0) == _HTM_TBEGIN_STARTED) + { + __builtin_tend (); + } +} +EOF +dnl +dnl test, if the tbegin instruction is used by __builtin_tbegin +if AC_TRY_COMMAND([${CC-cc} -mhtm -O2 -S conftest.c -o - | grep -w tbegin > /dev/null]) ; +then + libc_cv_gcc_builtin_tbegin=yes +else + libc_cv_gcc_builtin_tbegin=no +fi +rm -f conftest* ]) + +if test "$enable_lock_elision" = yes && test "$libc_cv_gcc_builtin_tbegin" = no ; then + critic_missing="$critic_missing The used GCC has no support for __builtin_tbegin, which is needed for lock-elision on target S390." +fi + test -n "$critic_missing" && AC_MSG_ERROR([ *** $critic_missing])