From patchwork Thu Jun 12 00:23:40 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andi Kleen X-Patchwork-Id: 358940 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id D7793140096 for ; Thu, 12 Jun 2014 10:24:16 +1000 (EST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=s0jwP0e4xJbI/gGR/wyT6GdL7CEWcqK dtanXFfZUwk4WXaFNmmYiWEyArbkBMvh1TI8XRifVjdFSq1/Kj4C5NmgtuusqlmY FLJs5gHgePpIdLybIKMSrXEkMEvL/X0EiJ17juysiG+P42+JJn6xt4OpYHsRAfZ6 hfb2vE42BTeo= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id:in-reply-to :references; s=default; bh=8f4w8ubPCZeEOTsHFfHrBc9QfDw=; b=Xwpw/ J3hHTPHEcGCrN7w1rurDyOQF0qBobazP+0xWbqvxNAQC8gyjK2YNSRD8Pg7IB8xk AQz3hACu+fAivN5a0Njw29TotAJIylx220ZElmOvEt/O6S5FKHdKRqS76dlH6F9d mjbsdw8LAe4mzSaZXYQmSHVQMHUsxAFZSj1smQ= Received: (qmail 23321 invoked by alias); 12 Jun 2014 00:23:52 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 23237 invoked by uid 89); 12 Jun 2014 00:23:51 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-4.0 required=5.0 tests=AWL, BAYES_00, RP_MATCHES_RCVD autolearn=ham version=3.3.2 X-HELO: mga01.intel.com X-ExtLoop1: 1 From: Andi Kleen To: libc-alpha@sourceware.org Cc: carlos@redhat.com, Andi Kleen Subject: [PATCH 2/3] Add adaptive elision to rwlocks v2 Date: Wed, 11 Jun 2014 17:23:40 -0700 Message-Id: <1402532621-26622-3-git-send-email-andi@firstfloor.org> In-Reply-To: <1402532621-26622-1-git-send-email-andi@firstfloor.org> References: <1402532621-26622-1-git-send-email-andi@firstfloor.org> From: Andi Kleen This patch relies on the C version of the rwlocks posted earlier. With C rwlocks it is very straight forward to do adaptive elision using TSX. It is based on the infrastructure added earlier for mutexes, but uses its own elision macros. The macros are fairly general purpose and could be used for other elision purposes too. This version is much cleaner than the earlier assembler based version, and in particular implements adaptation which makes it safer. I changed the behavior slightly to not require any changes in the test suite and fully conform to all expected behaviors (generally at the cost of not eliding in various situations). In particular this means the timedlock variants are not elided. Nested trylock aborts. v2: Address Roland's feedback. Add ACCESS_ONCE usage. I didn't use the nested function inlines, it seemed too fragile. Also it still uses macros to be generic. v3: Move __PTHREAD_RWLOCK_ELISION_EXTRA to each types file and remove central default. nptl/: 2014-06-11 Andi Kleen * pthread_rwlock_rdlock.c: Include elide.h. (pthread_rwlock_rdlock): Add elision. * pthread_rwlock_wrlock.c: Include elide.h. (pthread_rwlock_wrlock): Add elision. * pthread_rwlock_trywrlock.c: Include elide.h. (pthread_rwlock_trywrlock): Add elision. * pthread_rwlock_tryrdlock.c: Include elide.h. (pthread_rwlock_tryrdlock): Add elision. * pthread_rwlock_unlock.c: Include elide.h. (pthread_rwlock_tryrdlock): Add elision unlock. * sysdeps/pthread/pthread.h: (__PTHREAD_RWLOCK_ELISION_EXTRA): Handle new define (PTHREAD_RWLOCK_INITIALIZER, PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP): Handle new elision field. /: 2014-06-11 Andi Kleen * sysdeps/arm/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/sh/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/tile/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/a/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/aarch64/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/alpha/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/hppa/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/ia64/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/m68k/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/microblaze/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/mips/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/powerpc/nptl/bits/pthreadtypes.h (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/x86/elision-conf.c: (elision_init): Set try_xbegin to zero when no RTM. * sysdeps/x86/nptl/elide.h: New file. Add generic elision macros. * sysdeps/x86/nptl/bits/pthreadtypes.h (pthread_rwlock_t): Change __pad1 to __rwelision. (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. --- nptl/pthread_rwlock_rdlock.c | 7 ++ nptl/pthread_rwlock_tryrdlock.c | 7 ++ nptl/pthread_rwlock_trywrlock.c | 7 ++ nptl/pthread_rwlock_unlock.c | 6 ++ nptl/pthread_rwlock_wrlock.c | 7 ++ nptl/sysdeps/pthread/pthread.h | 6 +- .../unix/sysv/linux/s390/bits/pthreadtypes.h | 2 + .../unix/sysv/linux/sparc/bits/pthreadtypes.h | 2 + sysdeps/arm/nptl/bits/pthreadtypes.h | 2 + sysdeps/sh/nptl/bits/pthreadtypes.h | 2 + sysdeps/tile/nptl/bits/pthreadtypes.h | 2 + .../sysv/linux/aarch64/nptl/bits/pthreadtypes.h | 2 + sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h | 2 + .../unix/sysv/linux/hppa/nptl/bits/pthreadtypes.h | 2 + .../unix/sysv/linux/ia64/nptl/bits/pthreadtypes.h | 2 + .../unix/sysv/linux/m68k/nptl/bits/pthreadtypes.h | 2 + .../sysv/linux/microblaze/nptl/bits/pthreadtypes.h | 2 + .../unix/sysv/linux/mips/nptl/bits/pthreadtypes.h | 2 + .../unix/sysv/linux/powerpc/bits/pthreadtypes.h | 2 + sysdeps/unix/sysv/linux/x86/elision-conf.c | 2 + sysdeps/x86/nptl/bits/pthreadtypes.h | 7 +- sysdeps/x86/nptl/elide.h | 109 +++++++++++++++++++++ 22 files changed, 179 insertions(+), 5 deletions(-) create mode 100644 sysdeps/x86/nptl/elide.h diff --git a/nptl/pthread_rwlock_rdlock.c b/nptl/pthread_rwlock_rdlock.c index 1df0327..6eb9e09 100644 --- a/nptl/pthread_rwlock_rdlock.c +++ b/nptl/pthread_rwlock_rdlock.c @@ -22,6 +22,7 @@ #include #include #include +#include /* Acquire read lock for RWLOCK. Slow path. */ @@ -102,6 +103,12 @@ __pthread_rwlock_rdlock (pthread_rwlock_t *rwlock) LIBC_PROBE (rdlock_entry, 1, rwlock); + if (ELIDE_LOCK (rwlock->__data.__rwelision, + rwlock->__data.__lock == 0 + && rwlock->__data.__writer == 0 + && rwlock->__data.__nr_readers == 0)) + return 0; + /* Make sure we are alone. */ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); diff --git a/nptl/pthread_rwlock_tryrdlock.c b/nptl/pthread_rwlock_tryrdlock.c index f7b1e6b..cf8d68e 100644 --- a/nptl/pthread_rwlock_tryrdlock.c +++ b/nptl/pthread_rwlock_tryrdlock.c @@ -19,6 +19,7 @@ #include #include "pthreadP.h" #include +#include int @@ -26,6 +27,12 @@ __pthread_rwlock_tryrdlock (pthread_rwlock_t *rwlock) { int result = EBUSY; + if (ELIDE_TRYLOCK (rwlock->__data.__rwelision, + rwlock->__data.__lock == 0 + && rwlock->__data.__nr_readers == 0 + && rwlock->__data.__writer, 0)) + return 0; + lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); if (rwlock->__data.__writer == 0 diff --git a/nptl/pthread_rwlock_trywrlock.c b/nptl/pthread_rwlock_trywrlock.c index 106f157..0291fc9 100644 --- a/nptl/pthread_rwlock_trywrlock.c +++ b/nptl/pthread_rwlock_trywrlock.c @@ -19,6 +19,7 @@ #include #include "pthreadP.h" #include +#include int @@ -26,6 +27,12 @@ __pthread_rwlock_trywrlock (pthread_rwlock_t *rwlock) { int result = EBUSY; + if (ELIDE_TRYLOCK (rwlock->__data.__rwelision, + rwlock->__data.__lock == 0 + && rwlock->__data.__nr_readers == 0 + && rwlock->__data.__writer, 1)) + return 0; + lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); if (rwlock->__data.__writer == 0 && rwlock->__data.__nr_readers == 0) diff --git a/nptl/pthread_rwlock_unlock.c b/nptl/pthread_rwlock_unlock.c index d492383..3ebddeb 100644 --- a/nptl/pthread_rwlock_unlock.c +++ b/nptl/pthread_rwlock_unlock.c @@ -22,6 +22,8 @@ #include #include #include +#include + /* Unlock RWLOCK. */ int @@ -29,6 +31,10 @@ __pthread_rwlock_unlock (pthread_rwlock_t *rwlock) { LIBC_PROBE (rwlock_unlock, 1, rwlock); + if (ELIDE_UNLOCK (rwlock->__data.__writer == 0 + && rwlock->__data.__nr_readers == 0)) + return 0; + lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); if (rwlock->__data.__writer) rwlock->__data.__writer = 0; diff --git a/nptl/pthread_rwlock_wrlock.c b/nptl/pthread_rwlock_wrlock.c index de54e51..91ad82a 100644 --- a/nptl/pthread_rwlock_wrlock.c +++ b/nptl/pthread_rwlock_wrlock.c @@ -22,6 +22,7 @@ #include #include #include +#include /* Acquire write lock for RWLOCK. */ @@ -91,6 +92,12 @@ __pthread_rwlock_wrlock (pthread_rwlock_t *rwlock) { LIBC_PROBE (wrlock_entry, 1, rwlock); + if (ELIDE_LOCK (rwlock->__data.__rwelision, + rwlock->__data.__lock == 0 + && rwlock->__data.__writer == 0 + && rwlock->__data.__nr_readers == 0)) + return 0; + /* Make sure we are alone. */ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); diff --git a/nptl/sysdeps/pthread/pthread.h b/nptl/sysdeps/pthread/pthread.h index 40a3e21..682a1ae 100644 --- a/nptl/sysdeps/pthread/pthread.h +++ b/nptl/sysdeps/pthread/pthread.h @@ -132,17 +132,17 @@ enum /* Read-write lock initializers. */ # define PTHREAD_RWLOCK_INITIALIZER \ - { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } } + { { 0, 0, 0, 0, 0, 0, 0, 0, __PTHREAD_RWLOCK_ELISION_EXTRA, 0, 0 } } # ifdef __USE_GNU # ifdef __PTHREAD_RWLOCK_INT_FLAGS_SHARED # define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \ - { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \ + { { 0, 0, 0, 0, 0, 0, 0, 0, __PTHREAD_RWLOCK_ELISION_EXTRA, 0, \ PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP } } # else # if __BYTE_ORDER == __LITTLE_ENDIAN # define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \ { { 0, 0, 0, 0, 0, 0, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP, \ - 0, 0, 0, 0 } } + 0, __PTHREAD_RWLOCK_ELISION_EXTRA, 0, 0 } } # else # define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \ { { 0, 0, 0, 0, 0, 0, 0, 0, 0, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,\ diff --git a/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h b/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h index d70f8b3..c9f1c83 100644 --- a/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h +++ b/nptl/sysdeps/unix/sysv/linux/s390/bits/pthreadtypes.h @@ -214,6 +214,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/nptl/sysdeps/unix/sysv/linux/sparc/bits/pthreadtypes.h b/nptl/sysdeps/unix/sysv/linux/sparc/bits/pthreadtypes.h index 8c0340a..0d225a3 100644 --- a/nptl/sysdeps/unix/sysv/linux/sparc/bits/pthreadtypes.h +++ b/nptl/sysdeps/unix/sysv/linux/sparc/bits/pthreadtypes.h @@ -194,6 +194,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/arm/nptl/bits/pthreadtypes.h b/sysdeps/arm/nptl/bits/pthreadtypes.h index 64b9e09..3ece5b9 100644 --- a/sysdeps/arm/nptl/bits/pthreadtypes.h +++ b/sysdeps/arm/nptl/bits/pthreadtypes.h @@ -155,6 +155,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/sh/nptl/bits/pthreadtypes.h b/sysdeps/sh/nptl/bits/pthreadtypes.h index a177f28..363a948 100644 --- a/sysdeps/sh/nptl/bits/pthreadtypes.h +++ b/sysdeps/sh/nptl/bits/pthreadtypes.h @@ -155,6 +155,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/tile/nptl/bits/pthreadtypes.h b/sysdeps/tile/nptl/bits/pthreadtypes.h index 5ca3391..f808bb2 100644 --- a/sysdeps/tile/nptl/bits/pthreadtypes.h +++ b/sysdeps/tile/nptl/bits/pthreadtypes.h @@ -194,6 +194,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/aarch64/nptl/bits/pthreadtypes.h b/sysdeps/unix/sysv/linux/aarch64/nptl/bits/pthreadtypes.h index 1a44bb6..5ad0e70 100644 --- a/sysdeps/unix/sysv/linux/aarch64/nptl/bits/pthreadtypes.h +++ b/sysdeps/unix/sysv/linux/aarch64/nptl/bits/pthreadtypes.h @@ -141,6 +141,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h b/sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h index 3dbe612..55f1067 100644 --- a/sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h +++ b/sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h @@ -142,6 +142,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/hppa/nptl/bits/pthreadtypes.h b/sysdeps/unix/sysv/linux/hppa/nptl/bits/pthreadtypes.h index a79c195..d6fdc2c 100644 --- a/sysdeps/unix/sysv/linux/hppa/nptl/bits/pthreadtypes.h +++ b/sysdeps/unix/sysv/linux/hppa/nptl/bits/pthreadtypes.h @@ -196,6 +196,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/ia64/nptl/bits/pthreadtypes.h b/sysdeps/unix/sysv/linux/ia64/nptl/bits/pthreadtypes.h index 9468329..71e4785 100644 --- a/sysdeps/unix/sysv/linux/ia64/nptl/bits/pthreadtypes.h +++ b/sysdeps/unix/sysv/linux/ia64/nptl/bits/pthreadtypes.h @@ -143,6 +143,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/m68k/nptl/bits/pthreadtypes.h b/sysdeps/unix/sysv/linux/m68k/nptl/bits/pthreadtypes.h index 1e1fed8..fd46c78 100644 --- a/sysdeps/unix/sysv/linux/m68k/nptl/bits/pthreadtypes.h +++ b/sysdeps/unix/sysv/linux/m68k/nptl/bits/pthreadtypes.h @@ -146,6 +146,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/microblaze/nptl/bits/pthreadtypes.h b/sysdeps/unix/sysv/linux/microblaze/nptl/bits/pthreadtypes.h index 9c7e620..9434453 100644 --- a/sysdeps/unix/sysv/linux/microblaze/nptl/bits/pthreadtypes.h +++ b/sysdeps/unix/sysv/linux/microblaze/nptl/bits/pthreadtypes.h @@ -150,6 +150,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/mips/nptl/bits/pthreadtypes.h b/sysdeps/unix/sysv/linux/mips/nptl/bits/pthreadtypes.h index cefd2b6..843031e 100644 --- a/sysdeps/unix/sysv/linux/mips/nptl/bits/pthreadtypes.h +++ b/sysdeps/unix/sysv/linux/mips/nptl/bits/pthreadtypes.h @@ -203,6 +203,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h b/sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h index eda3d1a..4e9c518 100644 --- a/sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h +++ b/sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h @@ -194,6 +194,8 @@ typedef union long int __align; } pthread_rwlock_t; +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + typedef union { char __size[__SIZEOF_PTHREAD_RWLOCKATTR_T]; diff --git a/sysdeps/unix/sysv/linux/x86/elision-conf.c b/sysdeps/unix/sysv/linux/x86/elision-conf.c index e6f5d6d..28e48d9 100644 --- a/sysdeps/unix/sysv/linux/x86/elision-conf.c +++ b/sysdeps/unix/sysv/linux/x86/elision-conf.c @@ -66,6 +66,8 @@ elision_init (int argc __attribute__ ((unused)), #ifdef ENABLE_LOCK_ELISION __pthread_force_elision = __libc_enable_secure ? 0 : __elision_available; #endif + if (!HAS_RTM) + __elision_aconf.retry_try_xbegin = 0; /* Disable elision on rwlocks */ } #ifdef SHARED diff --git a/sysdeps/x86/nptl/bits/pthreadtypes.h b/sysdeps/x86/nptl/bits/pthreadtypes.h index b4329f6..b04c32b 100644 --- a/sysdeps/x86/nptl/bits/pthreadtypes.h +++ b/sysdeps/x86/nptl/bits/pthreadtypes.h @@ -184,11 +184,13 @@ typedef union unsigned int __nr_writers_queued; int __writer; int __shared; - unsigned long int __pad1; + signed char __rwelision; + unsigned char __pad1[7]; unsigned long int __pad2; /* FLAGS must stay at this position in the structure to maintain binary compatibility. */ unsigned int __flags; +# define __PTHREAD_RWLOCK_ELISION_EXTRA 0, {0, 0, 0, 0, 0, 0, 0 } # define __PTHREAD_RWLOCK_INT_FLAGS_SHARED 1 } __data; # else @@ -204,7 +206,8 @@ typedef union binary compatibility. */ unsigned char __flags; unsigned char __shared; - unsigned char __pad1; + signed char __rwelision; +# define __PTHREAD_RWLOCK_ELISION_EXTRA 0 unsigned char __pad2; int __writer; } __data; diff --git a/sysdeps/x86/nptl/elide.h b/sysdeps/x86/nptl/elide.h new file mode 100644 index 0000000..19f27e5 --- /dev/null +++ b/sysdeps/x86/nptl/elide.h @@ -0,0 +1,109 @@ +/* elide.h: Generic lock elision support. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ +#ifndef ELIDE_H +#define ELIDE_H 1 + +#include +#include + +#define ACCESS_ONCE(x) (* (volatile typeof(x) *) &(x)) + +/* Adapt elision with ADAPT_COUNT and STATUS and decide retries. */ + +static inline bool +elision_adapt(uint8_t *adapt_count, unsigned int status) +{ + if (status & _XABORT_RETRY) + return false; + if ((status & _XABORT_EXPLICIT) + && _XABORT_CODE (status) == _ABORT_LOCK_BUSY) + { + /* Right now we skip here. Better would be to wait a bit + and retry. This likely needs some spinning. Be careful + to avoid writing the lock. */ + if (*adapt_count != __elision_aconf.skip_lock_busy) + ACCESS_ONCE (*adapt_count) = __elision_aconf.skip_lock_busy; + } + /* Internal abort. There is no chance for retry. + Use the normal locking and next time use lock. + Be careful to avoid writing to the lock. */ + else if (*adapt_count != __elision_aconf.skip_lock_internal_abort) + ACCESS_ONCE (*adapt_count) = __elision_aconf.skip_lock_internal_abort; + return true; +} + +/* is_lock_free must be executed inside the transaction */ + +/* Returns true if lock defined by IS_LOCK_FREE was elided. + ADAPT_COUNT is a pointer to per-lock state variable. */ + +#define ELIDE_LOCK(adapt_count, is_lock_free) \ + ({ \ + int ret = 0; \ + \ + if ((adapt_count) <= 0) \ + { \ + for (int i = __elision_aconf.retry_try_xbegin; i > 0; i--) \ + { \ + unsigned int status; \ + if ((status = _xbegin ()) == _XBEGIN_STARTED) \ + { \ + if (is_lock_free) \ + { \ + ret = 1; \ + break; \ + } \ + _xabort (_ABORT_LOCK_BUSY); \ + } \ + if (!elision_adapt (&(adapt_count), status)) \ + break; \ + } \ + } \ + else \ + (adapt_count)--; /* missing updates ok */ \ + ret; \ + }) + +/* Returns true if lock defined by IS_LOCK_FREE was try-elided. + ADAPT_COUNT is a pointer to per-lock state variable. */ + +#define ELIDE_TRYLOCK(adapt_count, is_lock_free, write) ({ \ + int ret = 0; \ + if (__elision_aconf.retry_try_xbegin > 0) \ + { \ + if (write) \ + _xabort (_ABORT_NESTED_TRYLOCK); \ + ret = ELIDE_LOCK (adapt_count, is_lock_free); \ + } \ + ret; \ + }) + +/* Returns true if lock defined by IS_LOCK_FREE was elided. */ + +#define ELIDE_UNLOCK(is_lock_free) \ + ({ \ + int ret = 0; \ + if (is_lock_free) \ + { \ + _xend (); \ + ret = 1; \ + } \ + ret; \ + }) + +#endif