From patchwork Wed Nov 10 00:16:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 1553197 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=DRR22vyU; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4HplsB5m2Tz9sRK for ; Wed, 10 Nov 2021 11:21:50 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3D3DB3858032 for ; Wed, 10 Nov 2021 00:21:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3D3DB3858032 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1636503708; bh=RhIyIvEPXt4NrcKAnhcpzVZiZZ6v1gKsR8tBGOULI54=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=DRR22vyU2N0jdqRRWOmq1gA9rhm8iL0WQU+4lz43kb021rTbNeiSs9kYxHXyXqGhI SaE4iLOy6rv540OpXaWTo/kNfrkNJvsCPUkuqU7WZukI1Vnep/4uhlitbGYHCF13/d hx1+BuDMBF0mqlHiztD6tkOxeObndESaU3JNU6mE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by sourceware.org (Postfix) with ESMTPS id 751D93857C76 for ; Wed, 10 Nov 2021 00:16:17 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 751D93857C76 Received: by mail-pj1-x102c.google.com with SMTP id t5-20020a17090a4e4500b001a0a284fcc2so273201pjl.2 for ; Tue, 09 Nov 2021 16:16:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RhIyIvEPXt4NrcKAnhcpzVZiZZ6v1gKsR8tBGOULI54=; b=of67cuFy+EUsLjr+26o3GTZqDiVyiyh8EhV4HpOqY2l0CAqh4fkltINBAlNsIEs1uh 4ATf72ZE52+z0GRFUe/4hoUouPrh0a2nE19TJZ/2WE1j5p59CvPPDO4s342KEwG63dWf TEcTWIsXfcUPf9SGLnS/+A2ehPrCYfVRuLuDa94dcEDe+q4tPU6iZKzYu4h3SymrhnvQ 0P1rXjqzNCA1YyIQ1a29Sqc/B08pra0F/BXe8LXFDVSWAQ+M3HPDJilAgC/NO6mh8wyE yY3VYcPPbu0Hp+mjrb6PvIwHukrQtc4Rb+6SQIn4xXxOnuz18FiA1AghC+VmaOi/+ym8 xKYQ== X-Gm-Message-State: AOAM533HAlu0aOZYLEiujlK8G9NA3p5R0ajSJ/lGG4DXdIAk09bSZ7ma GvYIKhWS3taC55DgrvrikN8= X-Google-Smtp-Source: ABdhPJxTTz8kWz8yik3olm0HZYSwUpSlfzEy8rsqNtEnAAcRBqYvq2CkrWjryuVVuQl6MkB6vXD0SQ== X-Received: by 2002:a17:903:4043:b0:142:4f21:6976 with SMTP id n3-20020a170903404300b001424f216976mr11563336pla.62.1636503376575; Tue, 09 Nov 2021 16:16:16 -0800 (PST) Received: from gnu-cfl-2.localdomain ([172.58.35.133]) by smtp.gmail.com with ESMTPSA id mg12sm3651801pjb.10.2021.11.09.16.16.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Nov 2021 16:16:15 -0800 (PST) Received: from gnu-cfl-2.lan (localhost [IPv6:::1]) by gnu-cfl-2.localdomain (Postfix) with ESMTP id 7F5281A0985; Tue, 9 Nov 2021 16:16:14 -0800 (PST) To: libc-alpha@sourceware.org Subject: [PATCH v4 1/3] Reduce CAS in low level locks [BZ #28537] Date: Tue, 9 Nov 2021 16:16:12 -0800 Message-Id: <20211110001614.2087610-2-hjl.tools@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211110001614.2087610-1-hjl.tools@gmail.com> References: <20211110001614.2087610-1-hjl.tools@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3030.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Cc: Florian Weimer , Arjan van de Ven , Andreas Schwab Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" CAS instruction is expensive. From the x86 CPU's point of view, getting a cache line for writing is more expensive than reading. See Appendix A.2 Spinlock in: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/xeon-lock-scaling-analysis-paper.pdf The full compare and swap will grab the cache line exclusive and cause excessive cache line bouncing. 1. Change low level locks to do an atomic load and skip CAS if compare may fail to reduce cache line bouncing on contended locks. 2. In __lll_lock, replace atomic_compare_and_exchange_bool_acq with atomic_compare_and_exchange_val_acq and pass down the result to __lll_lock_wait and __lll_lock_wait_private to avoid the redundant load there. 3. Drop __glibc_unlikely in __lll_trylock and lll_cond_trylock since we don't know if it's actually rare; in the contended case it is clearly not rare. --- nptl/lowlevellock.c | 12 ++++++------ sysdeps/nptl/lowlevellock.h | 29 ++++++++++++++++++++--------- 2 files changed, 26 insertions(+), 15 deletions(-) diff --git a/nptl/lowlevellock.c b/nptl/lowlevellock.c index 8c740529c4..d1965c01ca 100644 --- a/nptl/lowlevellock.c +++ b/nptl/lowlevellock.c @@ -22,30 +22,30 @@ #include void -__lll_lock_wait_private (int *futex) +__lll_lock_wait_private (int *futex, int futex_value) { - if (atomic_load_relaxed (futex) == 2) + if (futex_value == 2) goto futex; while (atomic_exchange_acquire (futex, 2) != 0) { futex: - LIBC_PROBE (lll_lock_wait_private, 1, futex); + LIBC_PROBE (lll_lock_wait_private, 2, futex, futex_value); futex_wait ((unsigned int *) futex, 2, LLL_PRIVATE); /* Wait if *futex == 2. */ } } libc_hidden_def (__lll_lock_wait_private) void -__lll_lock_wait (int *futex, int private) +__lll_lock_wait (int *futex, int futex_value, int private) { - if (atomic_load_relaxed (futex) == 2) + if (futex_value == 2) goto futex; while (atomic_exchange_acquire (futex, 2) != 0) { futex: - LIBC_PROBE (lll_lock_wait, 1, futex); + LIBC_PROBE (lll_lock_wait, 2, futex, futex_value); futex_wait ((unsigned int *) futex, 2, private); /* Wait if *futex == 2. */ } } diff --git a/sysdeps/nptl/lowlevellock.h b/sysdeps/nptl/lowlevellock.h index 4d95114ed3..4235d13de9 100644 --- a/sysdeps/nptl/lowlevellock.h +++ b/sysdeps/nptl/lowlevellock.h @@ -66,7 +66,11 @@ 0. Otherwise leave lock unchanged and return non-zero to indicate that the lock was not acquired. */ #define __lll_trylock(lock) \ - __glibc_unlikely (atomic_compare_and_exchange_bool_acq ((lock), 1, 0)) + (__extension__ ({ \ + __typeof (*(lock)) __lock_value = atomic_load_relaxed (lock); \ + (__lock_value != 0 \ + || atomic_compare_and_exchange_bool_acq ((lock), 1, 0)); \ + })) #define lll_trylock(lock) \ __lll_trylock (&(lock)) @@ -74,11 +78,15 @@ return 0. Otherwise leave lock unchanged and return non-zero to indicate that the lock was not acquired. */ #define lll_cond_trylock(lock) \ - __glibc_unlikely (atomic_compare_and_exchange_bool_acq (&(lock), 2, 0)) + (__extension__ ({ \ + __typeof (lock) __lock_value = atomic_load_relaxed (&(lock)); \ + (__lock_value != 0 \ + || atomic_compare_and_exchange_bool_acq (&(lock), 2, 0)); \ + })) -extern void __lll_lock_wait_private (int *futex); +extern void __lll_lock_wait_private (int *futex, int futex_value); libc_hidden_proto (__lll_lock_wait_private) -extern void __lll_lock_wait (int *futex, int private); +extern void __lll_lock_wait (int *futex, int futex_value, int private); libc_hidden_proto (__lll_lock_wait) /* This is an expression rather than a statement even though its value is @@ -95,13 +103,15 @@ libc_hidden_proto (__lll_lock_wait) ((void) \ ({ \ int *__futex = (futex); \ - if (__glibc_unlikely \ - (atomic_compare_and_exchange_bool_acq (__futex, 1, 0))) \ + int __futex_value = atomic_load_relaxed (futex); \ + if (__futex_value != 0 \ + || ((__futex_value = atomic_compare_and_exchange_val_acq \ + (__futex, 1, 0) != 0))) \ { \ if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \ - __lll_lock_wait_private (__futex); \ + __lll_lock_wait_private (futex, __futex_value); \ else \ - __lll_lock_wait (__futex, private); \ + __lll_lock_wait (futex, __futex_value, private); \ } \ })) #define lll_lock(futex, private) \ @@ -120,7 +130,8 @@ libc_hidden_proto (__lll_lock_wait) ({ \ int *__futex = (futex); \ if (__glibc_unlikely (atomic_exchange_acq (__futex, 2) != 0)) \ - __lll_lock_wait (__futex, private); \ + __lll_lock_wait (__futex, atomic_load_relaxed (__futex), \ + private); \ })) #define lll_cond_lock(futex, private) __lll_cond_lock (&(futex), private) From patchwork Wed Nov 10 00:16:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 1553196 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=ojY3gkTL; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4HplrM69P7z9sRK for ; Wed, 10 Nov 2021 11:21:07 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 94AF5385801E for ; Wed, 10 Nov 2021 00:21:05 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 94AF5385801E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1636503665; bh=aVqux7lBP9ur7GWlQ+k+NpS2JY66dIN7KrJeBckVMZw=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=ojY3gkTLF/z+4oAKPsUGetjz8ujeINnyErP3vWs5FfFJ88u8J3R1qH0kY0R1iw1/X mdnEx8klWQGbW76sES2kFnZsIuF2C5mX9HiRRBKQDQRguVkOf0Vp1fYcA4wqqhn0Ol xz/0BGoxnOBv8K0lJpq6VddSqrjnk7ZiVfhDE4bo= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by sourceware.org (Postfix) with ESMTPS id 1B31C3857C72 for ; Wed, 10 Nov 2021 00:16:17 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 1B31C3857C72 Received: by mail-pg1-x529.google.com with SMTP id b4so574039pgh.10 for ; Tue, 09 Nov 2021 16:16:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aVqux7lBP9ur7GWlQ+k+NpS2JY66dIN7KrJeBckVMZw=; b=QfCkcNVUToFqmXwmAFDpUSaUTHII7sotCM6rJSZod7yGnXYLdGTTeVbzro/QkRDnIZ XY08In8MPaz3YhA4rw499oHtjLFBw9vcLzKRmUHh1eVoUPYNZYI6nEtCeibFzW9fzBuT 3YUklwDtvyIlGpMzGliEEE9cZkTPvZ4HrH5rM3kp86EzZg/BpQqUIFUWT/vOSw34Flms kXDRd0r2LSYbYvYjJL/i+D+IzlLYK//XQtQglEHFldXTBZYUIXUf/xftkLvOyDOQCQMs DfmeNuVbdaArpH1BswCrMCFzyzgUJPRJeui3Bh8qtB558x5cQ0O7unSMpr6kfg6bAUAv k+3w== X-Gm-Message-State: AOAM533GkL5tH/qYkhzqAVIk4QoxHZ08Bg69XOnATn/c8mRAXXlOmRdl I5Yzm8SV6Q1M8DwJQFPWIGA= X-Google-Smtp-Source: ABdhPJzwbS6v1Jx/m7PtfAEckpmkQfHwGO5PghzaGcJM2p/w4mEFwPrKgZ7rMZ9aSe1LM8M0HjRGZA== X-Received: by 2002:a63:b252:: with SMTP id t18mr9252764pgo.19.1636503376226; Tue, 09 Nov 2021 16:16:16 -0800 (PST) Received: from gnu-cfl-2.localdomain ([172.58.35.133]) by smtp.gmail.com with ESMTPSA id 59sm3736928pjz.34.2021.11.09.16.16.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Nov 2021 16:16:15 -0800 (PST) Received: from gnu-cfl-2.lan (localhost [IPv6:::1]) by gnu-cfl-2.localdomain (Postfix) with ESMTP id 8A51F1A0987; Tue, 9 Nov 2021 16:16:14 -0800 (PST) To: libc-alpha@sourceware.org Subject: [PATCH v4 2/3] Reduce CAS in __pthread_mutex_lock_full [BZ #28537] Date: Tue, 9 Nov 2021 16:16:13 -0800 Message-Id: <20211110001614.2087610-3-hjl.tools@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211110001614.2087610-1-hjl.tools@gmail.com> References: <20211110001614.2087610-1-hjl.tools@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3030.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Cc: Florian Weimer , Arjan van de Ven , Andreas Schwab Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" Change __pthread_mutex_lock_full to do an atomic load and skip CAS if compare may fail to reduce cache line bouncing on contended locks. --- nptl/pthread_mutex_lock.c | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index 2bd41767e0..1126ecba95 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -204,6 +204,10 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) our TID | assume_other_futex_waiters. */ if (__glibc_likely (oldval == 0)) { + oldval = atomic_load_relaxed (&mutex->__data.__lock); + if (oldval != 0) + goto tid_failed; + oldval = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, id | assume_other_futex_waiters, 0); @@ -213,6 +217,13 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) if ((oldval & FUTEX_OWNER_DIED) != 0) { + int currval = atomic_load_relaxed (&mutex->__data.__lock); + if (currval != oldval) + { + oldval = currval; + continue; + } + /* The previous owner died. Try locking the mutex. */ int newval = id; #ifdef NO_INCR @@ -259,6 +270,7 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) return EOWNERDEAD; } + tid_failed: /* Check whether we already hold the mutex. */ if (__glibc_unlikely ((oldval & FUTEX_TID_MASK) == id)) { @@ -411,11 +423,15 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) # ifdef NO_INCR newval |= FUTEX_WAITERS; # endif + oldval = atomic_load_relaxed (&mutex->__data.__lock); + if (oldval != 0) + goto locked_mutex; oldval = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, newval, 0); if (oldval != 0) { + locked_mutex: /* The mutex is locked. The kernel will now take care of everything. */ int private = (robust @@ -554,6 +570,10 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) ceilval = ceiling << PTHREAD_MUTEX_PRIO_CEILING_SHIFT; oldprio = ceiling; + oldval = atomic_load_relaxed (&mutex->__data.__lock); + if (oldval != ceilval) + goto ceilval_failed; + oldval = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, #ifdef NO_INCR @@ -568,10 +588,13 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) do { - oldval - = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, - ceilval | 2, - ceilval | 1); + oldval = atomic_load_relaxed (&mutex->__data.__lock); + ceilval_failed: + if (oldval == (ceilval | 1)) + oldval + = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, + ceilval | 2, + ceilval | 1); if ((oldval & PTHREAD_MUTEX_PRIO_CEILING_MASK) != ceilval) break; @@ -581,9 +604,10 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) ceilval | 2, PTHREAD_MUTEX_PSHARED (mutex)); } - while (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, - ceilval | 2, ceilval) - != ceilval); + while (atomic_load_relaxed (&mutex->__data.__lock) != ceilval + || (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, + ceilval | 2, ceilval) + != ceilval)); } while ((oldval & PTHREAD_MUTEX_PRIO_CEILING_MASK) != ceilval); From patchwork Wed Nov 10 00:16:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 1553194 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=VWnpTEH9; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Hplpl704nz9sRK for ; Wed, 10 Nov 2021 11:19:43 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D45D73858018 for ; Wed, 10 Nov 2021 00:19:40 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D45D73858018 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1636503580; bh=4h04qgwOtsVHEm2XST+J6XAxLFcZdm/Q3FHtH/yruSk=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=VWnpTEH9h6VaL1RdXcS+bPuECCMHyNxvdypQHkyZzYKlzWmObHpAg3Iloanigobpt Iat4p7nTcHn0C5twq/wRob4EQPCZkjsl6SnuH6JtEIeHE/lqHmYrrjhC40graydmKV xhROdLCy+tsuIMcTPa0qBR2Y7QkUra/gzBTOKUYY= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) by sourceware.org (Postfix) with ESMTPS id E5A083858033 for ; Wed, 10 Nov 2021 00:16:16 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org E5A083858033 Received: by mail-pf1-x432.google.com with SMTP id g18so989176pfk.5 for ; Tue, 09 Nov 2021 16:16:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4h04qgwOtsVHEm2XST+J6XAxLFcZdm/Q3FHtH/yruSk=; b=q3rzN2pgn11gN6VftRXuxhDLASV+kSzP5Qyhj40c81cLXhZB0iElasth79c3cb42TF 2FNUtLjZBhRJd60ytcYWmFK9uAOjcyMiG8NLOIZPuASs9SPAxe3bs+5oZB+vvbMATAMJ 9EgLNc3p0Vw9Fovbs1ZdncxnMzT7WP8NwtGySseH/gL5RTokqQ/0Rx7wfLTR9VEv8q8n HBE9Pzqt07iClqP1nHWn9DF9ntb+bMJ3rmGonKaNjU4cDe8x3UG4VkPsq5MXhAvXWZHK GDUfAFzwQSG33Agv2xWnwSVJvLcyZCqfncfMrvUaVCx6y4HxRbRaIDCHOgvJmQm09fpj ZX4Q== X-Gm-Message-State: AOAM531qHMdO4Asl23PoZ0rO9JVOefDUrXyzd4b6gi4imJuu3ppYxjPT 1IczcOwmT7CpFYaabMM+bypiJ7yJ+ww= X-Google-Smtp-Source: ABdhPJzULfLMGtzqAUSASw1A67BHDOcOX6Z5KHsXIuBqvTuQdgS/yn+P6HD1ZfMKIUncGf9ledTG+A== X-Received: by 2002:a63:5151:: with SMTP id r17mr9170924pgl.56.1636503376096; Tue, 09 Nov 2021 16:16:16 -0800 (PST) Received: from gnu-cfl-2.localdomain ([172.58.35.133]) by smtp.gmail.com with ESMTPSA id g17sm2410368pfv.136.2021.11.09.16.16.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Nov 2021 16:16:15 -0800 (PST) Received: from gnu-cfl-2.lan (localhost [IPv6:::1]) by gnu-cfl-2.localdomain (Postfix) with ESMTP id 8C31E1A0ED2; Tue, 9 Nov 2021 16:16:14 -0800 (PST) To: libc-alpha@sourceware.org Subject: [PATCH v4 3/3] Optimize CAS in __pthread_mutex_lock_full [BZ #28537] Date: Tue, 9 Nov 2021 16:16:14 -0800 Message-Id: <20211110001614.2087610-4-hjl.tools@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211110001614.2087610-1-hjl.tools@gmail.com> References: <20211110001614.2087610-1-hjl.tools@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3030.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Cc: Florian Weimer , Arjan van de Ven , Andreas Schwab Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" 1. Do an atomic load and skip CAS if compare may fail to reduce cache line bouncing on contended locks. 2. Replace atomic_compare_and_exchange_bool_acq with atomic_compare_and_exchange_val_acq to avoid the extra load. --- nptl/pthread_mutex_lock.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index 1126ecba95..9a18073f29 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -309,12 +309,17 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) meantime. */ if ((oldval & FUTEX_WAITERS) == 0) { - if (atomic_compare_and_exchange_bool_acq (&mutex->__data.__lock, - oldval | FUTEX_WAITERS, - oldval) - != 0) + int currval = atomic_load_relaxed (&mutex->__data.__lock); + if (currval != oldval) + { + oldval = currval; + continue; + } + int newval = atomic_compare_and_exchange_val_acq + (&mutex->__data.__lock, oldval | FUTEX_WAITERS, oldval); + if (newval != oldval) { - oldval = mutex->__data.__lock; + oldval = newval; continue; } oldval |= FUTEX_WAITERS;