From patchwork Wed Oct 25 14:10:03 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 830298 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=sourceware.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=libc-alpha-return-86341-incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.b="vxXmCnUm"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yMX9Z4Wpjz9t2M for ; Thu, 26 Oct 2017 01:10:58 +1100 (AEDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=bqoAznnYVpqDMzcuofzUJWy69FrtUtK 9YcO1pQEBNlHqakS3CGtbetlI4CDk7zCu1yC7QkMtoTQvn+JDlrI8bijBQeYyS3B oDAtpAsDG11dGuOhl65EB6W3XrDUPNUlxt6267qzxE9Ovpv91Czk0bDeUGJmKaBn rrASiyTXivHU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id:in-reply-to :references; s=default; bh=TBzEvY0lyQFE5aui3TkxZBh39XM=; b=vxXmC nUmSQ4yJbKfQ2qIBlW4zqyo4QjYGSboh5Udk8GpLbQTaA9lcfMfOPxnKx0Np3MQo k3yr4rn1kSJ2l8l/Ob3Nc7KlkwSvZyPhBj8LJfqCFU9hGksWYtUZZoN++JfNoytl N9wTSt7iQAMi/83HoD1Wi93enWE3hiWBxtyWug= Received: (qmail 49776 invoked by alias); 25 Oct 2017 14:10:31 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 49671 invoked by uid 89); 25 Oct 2017 14:10:30 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-25.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_LAZY_DOMAIN_SECURITY, RP_MATCHES_RCVD, SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=acquisitions, ceiling X-HELO: mx1.redhat.com DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 6B8286A7E4 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=longman@redhat.com From: Waiman Long To: GLIBC Devel Cc: Torvald Riegel , Carlos O'Donell , Thomas Gleixner , Waiman Long Subject: [RFC] [PATCH 1/2] nptl: Enable pthread mutex to use the TP futex Date: Wed, 25 Oct 2017 10:10:03 -0400 Message-Id: <1508940604-7233-2-git-send-email-longman@redhat.com> In-Reply-To: <1508940604-7233-1-git-send-email-longman@redhat.com> References: <1508940604-7233-1-git-send-email-longman@redhat.com> This patch adds a new protocol attribute PTHREAD_THROUGHPUT_NP to the pthread mutex code that enables the use of the new TP futexes. Signed-off-by: Waiman Long --- nptl/pthreadP.h | 18 ++++++++++ nptl/pthread_mutex_init.c | 27 +++++++++++++++ nptl/pthread_mutex_lock.c | 49 ++++++++++++++++++++++---- nptl/pthread_mutex_timedlock.c | 52 ++++++++++++++++++++++++---- nptl/pthread_mutex_trylock.c | 20 +++++++++-- nptl/pthread_mutex_unlock.c | 20 ++++++++--- nptl/pthread_mutexattr_setprotocol.c | 1 + sysdeps/nptl/pthread.h | 3 ++ sysdeps/unix/sysv/linux/hppa/pthread.h | 3 +- sysdeps/unix/sysv/linux/lowlevellock-futex.h | 2 ++ 10 files changed, 173 insertions(+), 22 deletions(-) diff --git a/nptl/pthreadP.h b/nptl/pthreadP.h index dbf46b0..e74010b 100644 --- a/nptl/pthreadP.h +++ b/nptl/pthreadP.h @@ -107,6 +107,24 @@ enum PTHREAD_MUTEX_TIMED_NP | PTHREAD_MUTEX_ELISION_NP, PTHREAD_MUTEX_TIMED_NO_ELISION_NP = PTHREAD_MUTEX_TIMED_NP | PTHREAD_MUTEX_NO_ELISION_NP, + + PTHREAD_MUTEX_THROUGHPUT_NP = 8, + PTHREAD_MUTEX_TP_NORMAL_NP + = PTHREAD_MUTEX_THROUGHPUT_NP | PTHREAD_MUTEX_NORMAL, + PTHREAD_MUTEX_TP_RECURSIVE_NP + = PTHREAD_MUTEX_THROUGHPUT_NP | PTHREAD_MUTEX_RECURSIVE_NP, + PTHREAD_MUTEX_TP_ERRORCHECK_NP + = PTHREAD_MUTEX_THROUGHPUT_NP | PTHREAD_MUTEX_ERRORCHECK_NP, + PTHREAD_MUTEX_TP_ADAPTIVE_NP + = PTHREAD_MUTEX_THROUGHPUT_NP | PTHREAD_MUTEX_ADAPTIVE_NP, + PTHREAD_MUTEX_TP_ROBUST_NORMAL_NP + = PTHREAD_MUTEX_THROUGHPUT_NP | PTHREAD_MUTEX_ROBUST_NORMAL_NP, + PTHREAD_MUTEX_TP_ROBUST_RECURSIVE_NP + = PTHREAD_MUTEX_THROUGHPUT_NP | PTHREAD_MUTEX_ROBUST_RECURSIVE_NP, + PTHREAD_MUTEX_TP_ROBUST_ERRORCHECK_NP + = PTHREAD_MUTEX_THROUGHPUT_NP | PTHREAD_MUTEX_ROBUST_ERRORCHECK_NP, + PTHREAD_MUTEX_TP_ROBUST_ADAPTIVE_NP + = PTHREAD_MUTEX_THROUGHPUT_NP | PTHREAD_MUTEX_ROBUST_ADAPTIVE_NP, }; #define PTHREAD_MUTEX_PSHARED_BIT 128 diff --git a/nptl/pthread_mutex_init.c b/nptl/pthread_mutex_init.c index 6f2fc80..d96570f 100644 --- a/nptl/pthread_mutex_init.c +++ b/nptl/pthread_mutex_init.c @@ -51,6 +51,24 @@ prio_inherit_missing (void) return true; } +static bool +tp_futex_missing (void) +{ +#ifdef __NR_futex + static int tp_futex_supported; + if (__glibc_unlikely (tp_futex_supported == 0)) + { + int lock = 0; + INTERNAL_SYSCALL_DECL (err); + int ret = INTERNAL_SYSCALL (futex, err, 4, &lock, FUTEX_UNLOCK, 0, 0); + assert (INTERNAL_SYSCALL_ERROR_P (ret, err)); + tp_futex_supported = INTERNAL_SYSCALL_ERRNO (ret, err) == ENOSYS ? -1 : 1; + } + return __glibc_unlikely (tp_futex_supported < 0); +#endif + return true; +} + int __pthread_mutex_init (pthread_mutex_t *mutex, const pthread_mutexattr_t *mutexattr) @@ -76,6 +94,11 @@ __pthread_mutex_init (pthread_mutex_t *mutex, return ENOTSUP; break; + case PTHREAD_THROUGHPUT_NP << PTHREAD_MUTEXATTR_PROTOCOL_SHIFT: + if (__glibc_unlikely (tp_futex_missing ())) + return ENOTSUP; + break; + default: /* XXX: For now we don't support robust priority protected mutexes. */ if (imutexattr->mutexkind & PTHREAD_MUTEXATTR_FLAG_ROBUST) @@ -123,6 +146,10 @@ __pthread_mutex_init (pthread_mutex_t *mutex, mutex->__data.__lock = ceiling << PTHREAD_MUTEX_PRIO_CEILING_SHIFT; break; + case PTHREAD_THROUGHPUT_NP << PTHREAD_MUTEXATTR_PROTOCOL_SHIFT: + mutex->__data.__kind |= PTHREAD_MUTEX_THROUGHPUT_NP; + break; + default: break; } diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index 7f8254b..afdf058 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -339,7 +339,7 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, NULL); break; - /* The PI support requires the Linux futex system call. If that's not + /* The PI/TP support requires the Linux futex system call. If that's not available, pthread_mutex_init should never have allowed the type to be set. So it will get the default case for an invalid type. */ #ifdef __NR_futex @@ -351,13 +351,37 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) case PTHREAD_MUTEX_PI_ROBUST_ERRORCHECK_NP: case PTHREAD_MUTEX_PI_ROBUST_NORMAL_NP: case PTHREAD_MUTEX_PI_ROBUST_ADAPTIVE_NP: + case PTHREAD_MUTEX_TP_RECURSIVE_NP: + case PTHREAD_MUTEX_TP_ERRORCHECK_NP: + case PTHREAD_MUTEX_TP_NORMAL_NP: + case PTHREAD_MUTEX_TP_ADAPTIVE_NP: + case PTHREAD_MUTEX_TP_ROBUST_RECURSIVE_NP: + case PTHREAD_MUTEX_TP_ROBUST_ERRORCHECK_NP: + case PTHREAD_MUTEX_TP_ROBUST_NORMAL_NP: + case PTHREAD_MUTEX_TP_ROBUST_ADAPTIVE_NP: { int kind = mutex->__data.__kind & PTHREAD_MUTEX_KIND_MASK_NP; int robust = mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP; + int lock_op, unlock_op, futex_val, e; + + if (mutex->__data.__kind & PTHREAD_MUTEX_THROUGHPUT_NP) + { + /* We will try 5 userspace locking attempts before doing kernel + lock */ + lock_op = FUTEX_LOCK; + unlock_op = FUTEX_UNLOCK; + futex_val = 5; + } + else + { + lock_op = FUTEX_LOCK_PI; + unlock_op = FUTEX_UNLOCK_PI; + futex_val = 1; + } if (robust) { - /* Note: robust PI futexes are signaled by setting bit 0. */ + /* Note: robust PI/TP futexes are signaled by setting bit 0. */ THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, (void *) (((uintptr_t) &mutex->__data.__list.__next) | 1)); @@ -411,9 +435,22 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) ? PTHREAD_ROBUST_MUTEX_PSHARED (mutex) : PTHREAD_MUTEX_PSHARED (mutex)); INTERNAL_SYSCALL_DECL (__err); - int e = INTERNAL_SYSCALL (futex, __err, 4, &mutex->__data.__lock, - __lll_private_flag (FUTEX_LOCK_PI, - private), 1, 0); + do + { + /* We need to do userspace locking for TP futexes */ + e = INTERNAL_SYSCALL (futex, __err, 4, + &mutex->__data.__lock, + __lll_private_flag (lock_op, private), + futex_val, 0); + if (lock_op != FUTEX_LOCK + || !INTERNAL_SYSCALL_ERROR_P (e, __err) + || INTERNAL_SYSCALL_ERRNO (e, __err) != EAGAIN) + break; + oldval = atomic_compare_and_exchange_val_acq + (&mutex->__data.__lock, newval, 0); + futex_val--; + } + while (oldval != 0 && (oldval & FUTEX_TID_MASK) != id); if (INTERNAL_SYSCALL_ERROR_P (e, __err) && (INTERNAL_SYSCALL_ERRNO (e, __err) == ESRCH @@ -474,7 +511,7 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) INTERNAL_SYSCALL_DECL (__err); INTERNAL_SYSCALL (futex, __err, 4, &mutex->__data.__lock, - __lll_private_flag (FUTEX_UNLOCK_PI, + __lll_private_flag (unlock_op, PTHREAD_ROBUST_MUTEX_PSHARED (mutex)), 0, 0); diff --git a/nptl/pthread_mutex_timedlock.c b/nptl/pthread_mutex_timedlock.c index 45f3454..7d80b83 100644 --- a/nptl/pthread_mutex_timedlock.c +++ b/nptl/pthread_mutex_timedlock.c @@ -337,13 +337,38 @@ __pthread_mutex_timedlock (pthread_mutex_t *mutex, case PTHREAD_MUTEX_PI_ROBUST_ERRORCHECK_NP: case PTHREAD_MUTEX_PI_ROBUST_NORMAL_NP: case PTHREAD_MUTEX_PI_ROBUST_ADAPTIVE_NP: + case PTHREAD_MUTEX_TP_RECURSIVE_NP: + case PTHREAD_MUTEX_TP_ERRORCHECK_NP: + case PTHREAD_MUTEX_TP_NORMAL_NP: + case PTHREAD_MUTEX_TP_ADAPTIVE_NP: + case PTHREAD_MUTEX_TP_ROBUST_RECURSIVE_NP: + case PTHREAD_MUTEX_TP_ROBUST_ERRORCHECK_NP: + case PTHREAD_MUTEX_TP_ROBUST_NORMAL_NP: + case PTHREAD_MUTEX_TP_ROBUST_ADAPTIVE_NP: { int kind = mutex->__data.__kind & PTHREAD_MUTEX_KIND_MASK_NP; int robust = mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP; + int lock_op, unlock_op, futex_val, e; + + if (mutex->__data.__kind & PTHREAD_MUTEX_THROUGHPUT_NP) + { + /* We will try 5 userspace locking attempts before doing kernel + lock */ + + lock_op = FUTEX_LOCK; + unlock_op = FUTEX_UNLOCK; + futex_val = 5; + } + else + { + lock_op = FUTEX_LOCK_PI; + unlock_op = FUTEX_UNLOCK_PI; + futex_val = 1; + } if (robust) { - /* Note: robust PI futexes are signaled by setting bit 0. */ + /* Note: robust PI/TP futexes are signaled by setting bit 0. */ THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, (void *) (((uintptr_t) &mutex->__data.__list.__next) | 1)); @@ -371,7 +396,7 @@ __pthread_mutex_timedlock (pthread_mutex_t *mutex, access. */ THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, NULL); - /* Just bump the counter. */ + /* Just bump the counter. */ if (__glibc_unlikely (mutex->__data.__count + 1 == 0)) /* Overflow of the counter. */ return EAGAIN; @@ -397,10 +422,23 @@ __pthread_mutex_timedlock (pthread_mutex_t *mutex, : PTHREAD_MUTEX_PSHARED (mutex)); INTERNAL_SYSCALL_DECL (__err); - int e = INTERNAL_SYSCALL (futex, __err, 4, &mutex->__data.__lock, - __lll_private_flag (FUTEX_LOCK_PI, - private), 1, - abstime); + do + { + /* We need to do userspace locking for TP futexes */ + e = INTERNAL_SYSCALL (futex, __err, 4, + &mutex->__data.__lock, + __lll_private_flag (lock_op, private), + futex_val, abstime); + if (lock_op != FUTEX_LOCK + || !INTERNAL_SYSCALL_ERROR_P (e, __err) + || INTERNAL_SYSCALL_ERRNO (e, __err) != EAGAIN) + break; + oldval = atomic_compare_and_exchange_val_acq + (&mutex->__data.__lock, id, 0); + futex_val--; + } + while (oldval != 0 && (oldval & FUTEX_TID_MASK) != id); + if (INTERNAL_SYSCALL_ERROR_P (e, __err)) { if (INTERNAL_SYSCALL_ERRNO (e, __err) == ETIMEDOUT) @@ -479,7 +517,7 @@ __pthread_mutex_timedlock (pthread_mutex_t *mutex, INTERNAL_SYSCALL_DECL (__err); INTERNAL_SYSCALL (futex, __err, 4, &mutex->__data.__lock, - __lll_private_flag (FUTEX_UNLOCK_PI, + __lll_private_flag (unlock_op, PTHREAD_ROBUST_MUTEX_PSHARED (mutex)), 0, 0); diff --git a/nptl/pthread_mutex_trylock.c b/nptl/pthread_mutex_trylock.c index ec7da61..9f0189f 100644 --- a/nptl/pthread_mutex_trylock.c +++ b/nptl/pthread_mutex_trylock.c @@ -186,7 +186,7 @@ __pthread_mutex_trylock (pthread_mutex_t *mutex) return 0; - /* The PI support requires the Linux futex system call. If that's not + /* The PI/TP support requires the Linux futex system call. If that's not available, pthread_mutex_init should never have allowed the type to be set. So it will get the default case for an invalid type. */ #ifdef __NR_futex @@ -198,12 +198,21 @@ __pthread_mutex_trylock (pthread_mutex_t *mutex) case PTHREAD_MUTEX_PI_ROBUST_ERRORCHECK_NP: case PTHREAD_MUTEX_PI_ROBUST_NORMAL_NP: case PTHREAD_MUTEX_PI_ROBUST_ADAPTIVE_NP: + case PTHREAD_MUTEX_TP_RECURSIVE_NP: + case PTHREAD_MUTEX_TP_ERRORCHECK_NP: + case PTHREAD_MUTEX_TP_NORMAL_NP: + case PTHREAD_MUTEX_TP_ADAPTIVE_NP: + case PTHREAD_MUTEX_TP_ROBUST_RECURSIVE_NP: + case PTHREAD_MUTEX_TP_ROBUST_ERRORCHECK_NP: + case PTHREAD_MUTEX_TP_ROBUST_NORMAL_NP: + case PTHREAD_MUTEX_TP_ROBUST_ADAPTIVE_NP: { int kind = mutex->__data.__kind & PTHREAD_MUTEX_KIND_MASK_NP; int robust = mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP; + bool tp_futex = mutex->__data.__kind & PTHREAD_MUTEX_THROUGHPUT_NP; if (robust) - /* Note: robust PI futexes are signaled by setting bit 0. */ + /* Note: robust PI/TP futexes are signaled by setting bit 0. */ THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, (void *) (((uintptr_t) &mutex->__data.__list.__next) | 1)); @@ -247,6 +256,9 @@ __pthread_mutex_trylock (pthread_mutex_t *mutex) return EBUSY; } + if (tp_futex) + goto futex_dead; + assert (robust); /* The mutex owner died. The kernel will now take care of @@ -272,6 +284,7 @@ __pthread_mutex_trylock (pthread_mutex_t *mutex) if (__glibc_unlikely (oldval & FUTEX_OWNER_DIED)) { +futex_dead: atomic_and (&mutex->__data.__lock, ~FUTEX_OWNER_DIED); /* We got the mutex. */ @@ -298,7 +311,8 @@ __pthread_mutex_trylock (pthread_mutex_t *mutex) INTERNAL_SYSCALL_DECL (__err); INTERNAL_SYSCALL (futex, __err, 4, &mutex->__data.__lock, - __lll_private_flag (FUTEX_UNLOCK_PI, + __lll_private_flag (tp_futex ? FUTEX_UNLOCK + : FUTEX_UNLOCK_PI, PTHREAD_ROBUST_MUTEX_PSHARED (mutex)), 0, 0); diff --git a/nptl/pthread_mutex_unlock.c b/nptl/pthread_mutex_unlock.c index c7e6795..58cbffa 100644 --- a/nptl/pthread_mutex_unlock.c +++ b/nptl/pthread_mutex_unlock.c @@ -176,6 +176,7 @@ __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) be set. So it will get the default case for an invalid type. */ #ifdef __NR_futex case PTHREAD_MUTEX_PI_RECURSIVE_NP: + case PTHREAD_MUTEX_TP_RECURSIVE_NP: /* Recursive mutex. */ if (mutex->__data.__owner != THREAD_GETMEM (THREAD_SELF, tid)) return EPERM; @@ -183,9 +184,10 @@ __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) if (--mutex->__data.__count != 0) /* We still hold the mutex. */ return 0; - goto continue_pi_non_robust; + goto continue_pi_tp_non_robust; case PTHREAD_MUTEX_PI_ROBUST_RECURSIVE_NP: + case PTHREAD_MUTEX_TP_ROBUST_RECURSIVE_NP: /* Recursive mutex. */ if ((mutex->__data.__lock & FUTEX_TID_MASK) == THREAD_GETMEM (THREAD_SELF, tid) @@ -206,7 +208,7 @@ __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) /* We still hold the mutex. */ return 0; - goto continue_pi_robust; + goto continue_pi_tp_robust; case PTHREAD_MUTEX_PI_ERRORCHECK_NP: case PTHREAD_MUTEX_PI_NORMAL_NP: @@ -214,6 +216,12 @@ __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) case PTHREAD_MUTEX_PI_ROBUST_ERRORCHECK_NP: case PTHREAD_MUTEX_PI_ROBUST_NORMAL_NP: case PTHREAD_MUTEX_PI_ROBUST_ADAPTIVE_NP: + case PTHREAD_MUTEX_TP_ERRORCHECK_NP: + case PTHREAD_MUTEX_TP_NORMAL_NP: + case PTHREAD_MUTEX_TP_ADAPTIVE_NP: + case PTHREAD_MUTEX_TP_ROBUST_ERRORCHECK_NP: + case PTHREAD_MUTEX_TP_ROBUST_NORMAL_NP: + case PTHREAD_MUTEX_TP_ROBUST_ADAPTIVE_NP: if ((mutex->__data.__lock & FUTEX_TID_MASK) != THREAD_GETMEM (THREAD_SELF, tid) || ! lll_islocked (mutex->__data.__lock)) @@ -230,7 +238,7 @@ __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) != 0) { - continue_pi_robust: + continue_pi_tp_robust: /* Remove mutex from the list. Note: robust PI futexes are signaled by setting bit 0. */ THREAD_SETMEM (THREAD_SELF, robust_head.list_op_pending, @@ -242,7 +250,7 @@ __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) DEQUEUE_MUTEX (mutex); } - continue_pi_non_robust: + continue_pi_tp_non_robust: mutex->__data.__owner = newowner; if (decr) /* One less user. */ @@ -260,6 +268,8 @@ __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) kernel take care of the situation. Use release MO in the CAS to synchronize with acquire MO in lock acquisitions. */ int l = atomic_load_relaxed (&mutex->__data.__lock); + int unlock_op = (mutex->__data.__kind & PTHREAD_MUTEX_THROUGHPUT_NP) + ? FUTEX_UNLOCK : FUTEX_UNLOCK_PI; do { if (((l & FUTEX_WAITERS) != 0) @@ -267,7 +277,7 @@ __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) { INTERNAL_SYSCALL_DECL (__err); INTERNAL_SYSCALL (futex, __err, 2, &mutex->__data.__lock, - __lll_private_flag (FUTEX_UNLOCK_PI, private)); + __lll_private_flag (unlock_op, private)); break; } } diff --git a/nptl/pthread_mutexattr_setprotocol.c b/nptl/pthread_mutexattr_setprotocol.c index 301fd0a..52f71b9 100644 --- a/nptl/pthread_mutexattr_setprotocol.c +++ b/nptl/pthread_mutexattr_setprotocol.c @@ -26,6 +26,7 @@ pthread_mutexattr_setprotocol (pthread_mutexattr_t *attr, int protocol) { if (protocol != PTHREAD_PRIO_NONE && protocol != PTHREAD_PRIO_INHERIT + && protocol != PTHREAD_THROUGHPUT_NP && __builtin_expect (protocol != PTHREAD_PRIO_PROTECT, 0)) return EINVAL; diff --git a/sysdeps/nptl/pthread.h b/sysdeps/nptl/pthread.h index 632ea7b..b2fe9e6 100644 --- a/sysdeps/nptl/pthread.h +++ b/sysdeps/nptl/pthread.h @@ -79,6 +79,9 @@ enum PTHREAD_PRIO_NONE, PTHREAD_PRIO_INHERIT, PTHREAD_PRIO_PROTECT +#ifdef __USE_GNU + , PTHREAD_THROUGHPUT_NP +#endif }; #endif diff --git a/sysdeps/unix/sysv/linux/hppa/pthread.h b/sysdeps/unix/sysv/linux/hppa/pthread.h index d197374..37f9fdf 100644 --- a/sysdeps/unix/sysv/linux/hppa/pthread.h +++ b/sysdeps/unix/sysv/linux/hppa/pthread.h @@ -78,7 +78,8 @@ enum { PTHREAD_PRIO_NONE, PTHREAD_PRIO_INHERIT, - PTHREAD_PRIO_PROTECT + PTHREAD_PRIO_PROTECT, + PTHREAD_THROUGHPUT_NP }; #endif diff --git a/sysdeps/unix/sysv/linux/lowlevellock-futex.h b/sysdeps/unix/sysv/linux/lowlevellock-futex.h index bb4fbae..89b1674 100644 --- a/sysdeps/unix/sysv/linux/lowlevellock-futex.h +++ b/sysdeps/unix/sysv/linux/lowlevellock-futex.h @@ -38,6 +38,8 @@ #define FUTEX_WAKE_BITSET 10 #define FUTEX_WAIT_REQUEUE_PI 11 #define FUTEX_CMP_REQUEUE_PI 12 +#define FUTEX_LOCK 13 +#define FUTEX_UNLOCK 14 #define FUTEX_PRIVATE_FLAG 128 #define FUTEX_CLOCK_REALTIME 256 From patchwork Wed Oct 25 14:10:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 830297 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=sourceware.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=libc-alpha-return-86340-incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.b="agYNZ1MG"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yMX9L37pwz9t2M for ; Thu, 26 Oct 2017 01:10:46 +1100 (AEDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=KYMDar0imod9IEwmgCfdz3lL2jd6oqJ 4Les8pcvyD3K3g1F+6vB4JarEXus7rtEDbWIcmguA5jrdLKOeLXvWusvfB8/aKaf y6cu11QAf5JEiLjv3RnUZp0Bdwr/530dTpm8sM2tHwz6RIG10hu7ca1ZNl4PHLyf ucvc7Z7YMr4c= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id:in-reply-to :references; s=default; bh=dHiA6gXcWjDMAO18mkBxltp5jNk=; b=agYNZ 1MGrmIkakb/kMpof19QeW8oRMa+97eM6+IS3rjF4IGyBGQY3I9mybdZQ4JTAJGzq wlC3xOPRAYQc+x94ZAERBglyGE/GMaxR9TLl3/tVOmoAXj7/Z6A60882mG5+xO1z /JSLJv4NxfSLTCDmbXjALyoRkin+YsL21f4Vak= Received: (qmail 49550 invoked by alias); 25 Oct 2017 14:10:29 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 49445 invoked by uid 89); 25 Oct 2017 14:10:29 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RP_MATCHES_RCVD, SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=waiter, 2625 X-HELO: mx1.redhat.com DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 5DF3210476 Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx09.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=longman@redhat.com From: Waiman Long To: GLIBC Devel Cc: Torvald Riegel , Carlos O'Donell , Thomas Gleixner , Waiman Long Subject: [RFC] [PATCH 2/2] nptl: Enable pthread rwlock to use the TP futex Date: Wed, 25 Oct 2017 10:10:04 -0400 Message-Id: <1508940604-7233-3-git-send-email-longman@redhat.com> In-Reply-To: <1508940604-7233-1-git-send-email-longman@redhat.com> References: <1508940604-7233-1-git-send-email-longman@redhat.com> This patch adds a new lock kind (PTHREAD_RWLOCK_USE_TP_FUTEX_NP) to the pthread_rwlockattr_setkind_np() to enable the pthread rwlock to use the TP futex, if available, for supporting rwlock. Signed-off-by: Waiman Long --- ChangeLog | 26 +++ nptl/pthread_rwlock_rdlock.c | 5 +- nptl/pthread_rwlock_timedrdlock.c | 5 +- nptl/pthread_rwlock_timedwrlock.c | 5 +- nptl/pthread_rwlock_tp.c | 235 +++++++++++++++++++++++++++ nptl/pthread_rwlock_tryrdlock.c | 5 + nptl/pthread_rwlock_trywrlock.c | 5 + nptl/pthread_rwlock_unlock.c | 14 ++ nptl/pthread_rwlock_wrlock.c | 5 +- nptl/pthread_rwlockattr_setkind_np.c | 21 ++- sysdeps/nptl/pthread.h | 1 + sysdeps/unix/sysv/linux/lowlevellock-futex.h | 7 + 12 files changed, 329 insertions(+), 5 deletions(-) create mode 100644 nptl/pthread_rwlock_tp.c diff --git a/ChangeLog b/ChangeLog index ea655fc..49255db 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,29 @@ +2017-10-24 Waiman Long + + * nptl/pthreadP.h: Add pthread mutex macros to support TP futexes. + * nptl/pthread_mutex_init.c: Add test for the presence of TP futexes. + * nptl/pthread_mutex_lock.c: Add TP futexes support. + * nptl/pthread_mutex_timedlock.c: Add TP futexes support. + * nptl/pthread_mutex_trylock.c: Add TP futexes support. + * nptl/pthread_mutex_unlock.c: Add TP futexes support. + * nptl/pthread_mutexattr_setprotocol.c: Make it accept a new protocol + PTHREAD_THROUGHPUT_NP. + * nptl/pthread_rwlock_rdlock.c: Add TP futexes support. + * nptl/pthread_rwlock_timedrdlock.c: Add TP futexes support. + * nptl/pthread_rwlock_timedwrlock.c: Add TP futexes support. + * nptl/pthread_rwlock_tp.c: New file to make TP futex syscalls. + * nptl/pthread_rwlock_tryrdlock.c: Add TP futexes support. + * nptl/pthread_rwlock_trywrlock.c: Add TP futexes support. + * nptl/pthread_rwlock_unlock.c: Add TP futexes support. + * nptl/pthread_rwlock_wrlock.c: Add TP futexes support. + * nptl/pthread_rwlockattr_setkind_np.c: Make it accept a new kind + PTHREAD_RWLOCK_USE_TP_FUTEX_NP. + * sysdeps/nptl/pthread.h: Add PTHREAD_THROUGHPUT_NP & + PTHREAD_RWLOCK_USE_TP_FUTEX_NP. + * sysdeps/unix/sysv/linux/hppa/pthread.h: Add PTHREAD_THROUGHPUT_NP. + * sysdeps/unix/sysv/linux/lowlevellock-futex.h: Add new TP futexes + related macros. + 2017-10-19 Valery Reznic H.J. Lu diff --git a/nptl/pthread_rwlock_rdlock.c b/nptl/pthread_rwlock_rdlock.c index e07581b..fe408b1 100644 --- a/nptl/pthread_rwlock_rdlock.c +++ b/nptl/pthread_rwlock_rdlock.c @@ -17,6 +17,7 @@ . */ #include "pthread_rwlock_common.c" +#include "pthread_rwlock_tp.c" /* See pthread_rwlock_common.c. */ int @@ -24,7 +25,9 @@ __pthread_rwlock_rdlock (pthread_rwlock_t *rwlock) { LIBC_PROBE (rdlock_entry, 1, rwlock); - int result = __pthread_rwlock_rdlock_full (rwlock, NULL); + int result = (rwlock->__data.__flags == PTHREAD_RWLOCK_USE_TP_FUTEX_NP) + ? __pthread_rwlock_rdlock_tp (rwlock, NULL) + : __pthread_rwlock_rdlock_full (rwlock, NULL); LIBC_PROBE (rdlock_acquire_read, 1, rwlock); return result; } diff --git a/nptl/pthread_rwlock_timedrdlock.c b/nptl/pthread_rwlock_timedrdlock.c index 9f084f8..d818184 100644 --- a/nptl/pthread_rwlock_timedrdlock.c +++ b/nptl/pthread_rwlock_timedrdlock.c @@ -17,6 +17,7 @@ . */ #include "pthread_rwlock_common.c" +#include "pthread_rwlock_tp.c" /* See pthread_rwlock_common.c. */ int @@ -33,5 +34,7 @@ pthread_rwlock_timedrdlock (pthread_rwlock_t *rwlock, || abstime->tv_nsec < 0)) return EINVAL; - return __pthread_rwlock_rdlock_full (rwlock, abstime); + return (rwlock->__data.__flags == PTHREAD_RWLOCK_USE_TP_FUTEX_NP) + ? __pthread_rwlock_rdlock_tp (rwlock, abstime) + : __pthread_rwlock_rdlock_full (rwlock, abstime); } diff --git a/nptl/pthread_rwlock_timedwrlock.c b/nptl/pthread_rwlock_timedwrlock.c index 5626505..e1dc3a4 100644 --- a/nptl/pthread_rwlock_timedwrlock.c +++ b/nptl/pthread_rwlock_timedwrlock.c @@ -17,6 +17,7 @@ . */ #include "pthread_rwlock_common.c" +#include "pthread_rwlock_tp.c" /* See pthread_rwlock_common.c. */ int @@ -33,5 +34,7 @@ pthread_rwlock_timedwrlock (pthread_rwlock_t *rwlock, || abstime->tv_nsec < 0)) return EINVAL; - return __pthread_rwlock_wrlock_full (rwlock, abstime); + return (rwlock->__data.__flags == PTHREAD_RWLOCK_USE_TP_FUTEX_NP) + ? __pthread_rwlock_wrlock_tp (rwlock, abstime) + : __pthread_rwlock_wrlock_full (rwlock, abstime); } diff --git a/nptl/pthread_rwlock_tp.c b/nptl/pthread_rwlock_tp.c new file mode 100644 index 0000000..1ecde7b --- /dev/null +++ b/nptl/pthread_rwlock_tp.c @@ -0,0 +1,235 @@ +/* POSIX reader--writer lock: TP futex specific code + Copyright (C) 2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * The __writers_futex is used as the target of the TP futex syscalls. + */ + +#ifdef __NR_futex + +#define FUTEX_FLAGS_MASK (~FUTEX_TID_MASK) +#define lll_futex_tp_lock(futexp, val, timeout, private) \ + lll_futex_syscall(4, futexp, \ + __lll_private_flag(FUTEX_LOCK, private), \ + val, timeout) +#define lll_futex_tp_unlock(futexp, private) \ + lll_futex_syscall(4, futexp, \ + __lll_private_flag(FUTEX_UNLOCK, private), \ + 0, NULL) +#define lll_futex_tp_lock_shared(futexp, timeout, private) \ + lll_futex_syscall(4, futexp, \ + __lll_private_flag(FUTEX_LOCK_SHARED, private),\ + 0, timeout) +#define lll_futex_tp_unlock_shared(futexp, private) \ + lll_futex_syscall(4, futexp, \ + __lll_private_flag(FUTEX_UNLOCK_SHARED, private),\ + 0, NULL) + +static __always_inline int +__pthread_rwlock_private (pthread_rwlock_t *rwlock) +{ + return rwlock->__data.__shared != 0 ? FUTEX_SHARED : FUTEX_PRIVATE; +} + +/* Return 1 if the lock acquired, 0 otherwise */ +static __always_inline int +__pthread_rwlock_tryrdlock_tp(pthread_rwlock_t *rwlock) +{ + unsigned int *futex = &rwlock->__data.__writers_futex; + unsigned int oldval = atomic_compare_and_exchange_val_acq + (futex, FUTEX_SHARED_FLAG + 1, 0); + if (oldval == 0) + return 1; + + while (1) + { + unsigned int new, old = oldval; + + /* + * Try to increment the reader count only if + * 1) the FUTEX_SHARED_FLAG bit is set; and + * 2) none of the flags bits or FUTEX_SHARED_UNLOCK is set. + */ + if (!old) + new = FUTEX_SHARED_FLAG + 1; + else if ((old & FUTEX_SHARED_FLAG) && + !(old & (FUTEX_FLAGS_MASK|FUTEX_SHARED_UNLOCK))) + new = old + 1; + else + return 0; + + oldval = atomic_compare_and_exchange_val_acq(futex, new, old); + if (old == oldval) + return 1; + } + /* Unreachable */ +} + +/* Return 1 if the lock acquired, 0 otherwise */ +static __always_inline int +__pthread_rwlock_trywrlock_tp(pthread_rwlock_t *rwlock) +{ + pid_t id = THREAD_GETMEM (THREAD_SELF, tid); + unsigned int *futex = &rwlock->__data.__writers_futex; + + return atomic_compare_and_exchange_val_acq(futex, id, 0) == 0; +} + +static __always_inline int +__pthread_rwlock_rdlock_tp(pthread_rwlock_t *rwlock, + const struct timespec *abstime) +{ + int private = __pthread_rwlock_private(rwlock); + unsigned int *futex = &rwlock->__data.__writers_futex; + + if (__pthread_rwlock_tryrdlock_tp(rwlock)) + return 0; + + return lll_futex_tp_lock_shared(futex, abstime, private); +} + +static __always_inline int +__pthread_rwlock_wrlock_tp(pthread_rwlock_t *rwlock, + const struct timespec *abstime) +{ + pid_t id = THREAD_GETMEM (THREAD_SELF, tid); + unsigned int *futex = &rwlock->__data.__writers_futex; + int private = __pthread_rwlock_private(rwlock); + int uslockcnt = 4; /* Do 4 userspace locks before doing kernel lock */ + + if (__pthread_rwlock_trywrlock_tp(rwlock)) + return 0; + + while (1) + { + int err = lll_futex_tp_lock(futex, uslockcnt, abstime, private); + + /* It is possible, though extremely unlikely, that a lock + handoff has happened and an error is returned. So we + need to check for that here. */ + if (!err || ((*futex & FUTEX_TID_MASK) == id)) + return 0; + + if (err == ETIMEDOUT) + return err; + + if ((err == EAGAIN) && __pthread_rwlock_trywrlock_tp(rwlock)) + return 0; + + uslockcnt--; + } + /* Unreachable */ +} + +static __always_inline void +__pthread_rwlock_rdunlock_tp(pthread_rwlock_t *rwlock) +{ + int val = atomic_fetch_add_release(&rwlock->__data.__writers_futex, -1) - 1; + unsigned int *futex = &rwlock->__data.__writers_futex; + int private = __pthread_rwlock_private(rwlock); + + while (1) + { + int oldval; + + /* Return if not the last reader, not in shared locking + mode or the unlock bit has been set. */ + if ((val & (FUTEX_SCNT_MASK|FUTEX_SHARED_UNLOCK)) || + !(val & FUTEX_SHARED_FLAG)) + return; + + if (val & FUTEX_FLAGS_MASK) + { + /* Only one task that can set the FUTEX_SHARED_UNLOCK + bit will do the unlock to wake up the waiters. */ + oldval = atomic_compare_and_exchange_val_rel + (futex, val|FUTEX_SHARED_UNLOCK, val); + if (oldval == val) + break; + } + else + { + /* Try to clear the futex when there is no waiter. */ + oldval = atomic_compare_and_exchange_val_rel(futex, 0, val); + if (oldval == val) + return; + } + val = oldval; + } + + lll_futex_tp_unlock_shared(futex, private); +} + +static __always_inline void +__pthread_rwlock_wrunlock_tp(pthread_rwlock_t *rwlock) +{ + pid_t id = THREAD_GETMEM (THREAD_SELF, tid); + unsigned int *futex = &rwlock->__data.__writers_futex; + int private = __pthread_rwlock_private(rwlock); + + if (atomic_compare_and_exchange_val_rel(futex, 0, id) == id) + return; + + lll_futex_tp_unlock(futex, private); +} + +#else /* __NR_futex */ + +static __always_inline int +__pthread_rwlock_tryrdlock_tp(pthread_rwlock_t *rwlock) +{ +} + +static __always_inline int +__pthread_rwlock_trywrlock_tp(pthread_rwlock_t *rwlock) +{ +} + + +static __always_inline void +__pthread_rwlock_rdlock_tp(pthread_rwlock_t *rwlock, + const struct timespec *abstime) +{ +} + +static __always_inline void +__pthread_rwlock_wrlock_tp(pthread_rwlock_t *rwlock, + const struct timespec *abstime) +{ +} + +static __always_inline void +__pthread_rwlock_rdunlock_tp(pthread_rwlock_t *rwlock) +{ +} + +static __always_inline void +__pthread_rwlock_wrunlock_tp(pthread_rwlock_t *rwlock) +{ +} + +#endif /* __NR_futex */ diff --git a/nptl/pthread_rwlock_tryrdlock.c b/nptl/pthread_rwlock_tryrdlock.c index 6c3014c..11f3a16 100644 --- a/nptl/pthread_rwlock_tryrdlock.c +++ b/nptl/pthread_rwlock_tryrdlock.c @@ -21,6 +21,7 @@ #include #include #include "pthread_rwlock_common.c" +#include "pthread_rwlock_tp.c" /* See pthread_rwlock_common.c for an overview. */ @@ -39,6 +40,10 @@ __pthread_rwlock_tryrdlock (pthread_rwlock_t *rwlock) below. */ unsigned int r = atomic_load_relaxed (&rwlock->__data.__readers); unsigned int rnew; + + if (rwlock->__data.__flags == PTHREAD_RWLOCK_USE_TP_FUTEX_NP) + return __pthread_rwlock_tryrdlock_tp(rwlock) ? 0 : EBUSY; + do { if ((r & PTHREAD_RWLOCK_WRPHASE) == 0) diff --git a/nptl/pthread_rwlock_trywrlock.c b/nptl/pthread_rwlock_trywrlock.c index 0d9ccaf..1ca4c3a 100644 --- a/nptl/pthread_rwlock_trywrlock.c +++ b/nptl/pthread_rwlock_trywrlock.c @@ -19,6 +19,7 @@ #include #include "pthreadP.h" #include +#include "pthread_rwlock_tp.c" /* See pthread_rwlock_common.c for an overview. */ int @@ -37,6 +38,10 @@ __pthread_rwlock_trywrlock (pthread_rwlock_t *rwlock) unsigned int r = atomic_load_relaxed (&rwlock->__data.__readers); bool prefer_writer = (rwlock->__data.__flags != PTHREAD_RWLOCK_PREFER_READER_NP); + + if (rwlock->__data.__flags == PTHREAD_RWLOCK_USE_TP_FUTEX_NP) + return __pthread_rwlock_trywrlock_tp(rwlock) ? 0 : EBUSY; + while (((r & PTHREAD_RWLOCK_WRLOCKED) == 0) && (((r >> PTHREAD_RWLOCK_READER_SHIFT) == 0) || (prefer_writer && ((r & PTHREAD_RWLOCK_WRPHASE) != 0)))) diff --git a/nptl/pthread_rwlock_unlock.c b/nptl/pthread_rwlock_unlock.c index ef46e88..153c2e2 100644 --- a/nptl/pthread_rwlock_unlock.c +++ b/nptl/pthread_rwlock_unlock.c @@ -24,6 +24,7 @@ #include #include "pthread_rwlock_common.c" +#include "pthread_rwlock_tp.c" /* See pthread_rwlock_common.c for an overview. */ int @@ -31,6 +32,19 @@ __pthread_rwlock_unlock (pthread_rwlock_t *rwlock) { LIBC_PROBE (rwlock_unlock, 1, rwlock); + if (rwlock->__data.__flags == PTHREAD_RWLOCK_USE_TP_FUTEX_NP) + { + unsigned int *futex = &rwlock->__data.__writers_futex; + + /* For TP futex, the FUTEX_SHARED bit is used to distinguish between a + writer and reader-owned lock. */ + if (atomic_load_relaxed (futex) & FUTEX_SHARED_FLAG) + __pthread_rwlock_rdunlock_tp (rwlock); + else + __pthread_rwlock_wrunlock_tp (rwlock); + return 0; + } + /* We distinguish between having acquired a read vs. a write lock by looking at the writer TID. If it's equal to our TID, we must be the writer because nobody else can have stored this value. Also, if we are a diff --git a/nptl/pthread_rwlock_wrlock.c b/nptl/pthread_rwlock_wrlock.c index 335fcd1..acaee21 100644 --- a/nptl/pthread_rwlock_wrlock.c +++ b/nptl/pthread_rwlock_wrlock.c @@ -17,6 +17,7 @@ . */ #include "pthread_rwlock_common.c" +#include "pthread_rwlock_tp.c" /* See pthread_rwlock_common.c. */ int @@ -24,7 +25,9 @@ __pthread_rwlock_wrlock (pthread_rwlock_t *rwlock) { LIBC_PROBE (wrlock_entry, 1, rwlock); - int result = __pthread_rwlock_wrlock_full (rwlock, NULL); + int result = (rwlock->__data.__flags == PTHREAD_RWLOCK_USE_TP_FUTEX_NP) + ? __pthread_rwlock_wrlock_tp (rwlock, NULL) + : __pthread_rwlock_wrlock_full (rwlock, NULL); LIBC_PROBE (wrlock_acquire_write, 1, rwlock); return result; } diff --git a/nptl/pthread_rwlockattr_setkind_np.c b/nptl/pthread_rwlockattr_setkind_np.c index b3cdc7f..2d35f8b 100644 --- a/nptl/pthread_rwlockattr_setkind_np.c +++ b/nptl/pthread_rwlockattr_setkind_np.c @@ -17,6 +17,7 @@ . */ #include +#include #include "pthreadP.h" @@ -25,7 +26,25 @@ pthread_rwlockattr_setkind_np (pthread_rwlockattr_t *attr, int pref) { struct pthread_rwlockattr *iattr; - if (pref != PTHREAD_RWLOCK_PREFER_READER_NP + if (pref == PTHREAD_RWLOCK_USE_TP_FUTEX_NP) + { +#ifdef __NR_futex + static int tp_futex_supported; + if (tp_futex_supported == 0) + { + int lock = 0; + INTERNAL_SYSCALL_DECL (err); + int ret = INTERNAL_SYSCALL (futex, err, 4, &lock, FUTEX_UNLOCK_SHARED, 0, 0); + assert (INTERNAL_SYSCALL_ERROR_P (ret, err)); + tp_futex_supported = INTERNAL_SYSCALL_ERRNO (ret, err) == ENOSYS ? -1 : 1; + } + if (tp_futex_supported < 0) + return ENOTSUP; +#else + return ENOTSUP; +#endif + } + else if (pref != PTHREAD_RWLOCK_PREFER_READER_NP && pref != PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP && __builtin_expect (pref != PTHREAD_RWLOCK_PREFER_WRITER_NP, 0)) return EINVAL; diff --git a/sysdeps/nptl/pthread.h b/sysdeps/nptl/pthread.h index b2fe9e6..a1e88c9 100644 --- a/sysdeps/nptl/pthread.h +++ b/sysdeps/nptl/pthread.h @@ -120,6 +120,7 @@ enum PTHREAD_RWLOCK_PREFER_READER_NP, PTHREAD_RWLOCK_PREFER_WRITER_NP, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP, + PTHREAD_RWLOCK_USE_TP_FUTEX_NP, PTHREAD_RWLOCK_DEFAULT_NP = PTHREAD_RWLOCK_PREFER_READER_NP }; diff --git a/sysdeps/unix/sysv/linux/lowlevellock-futex.h b/sysdeps/unix/sysv/linux/lowlevellock-futex.h index 89b1674..b093d9b 100644 --- a/sysdeps/unix/sysv/linux/lowlevellock-futex.h +++ b/sysdeps/unix/sysv/linux/lowlevellock-futex.h @@ -40,11 +40,18 @@ #define FUTEX_CMP_REQUEUE_PI 12 #define FUTEX_LOCK 13 #define FUTEX_UNLOCK 14 +#define FUTEX_LOCK_SHARED 15 +#define FUTEX_UNLOCK_SHARED 16 #define FUTEX_PRIVATE_FLAG 128 #define FUTEX_CLOCK_REALTIME 256 #define FUTEX_BITSET_MATCH_ANY 0xffffffff +/* Bits used by TP futex */ +#define FUTEX_SHARED_FLAG 0x20000000 +#define FUTEX_SHARED_UNLOCK 0x10000000 +#define FUTEX_SCNT_MASK 0x00ffffff + /* Values for 'private' parameter of locking macros. Yes, the definition seems to be backwards. But it is not. The bit will be reversed before passing to the system call. */