From patchwork Thu Jul 2 07:48:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1321024 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49y9fc0ybVz9sDX for ; Thu, 2 Jul 2020 18:06:48 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=h1o1LcgG; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 49y9fb6YkjzDqNs for ; Thu, 2 Jul 2020 18:06:47 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::444; helo=mail-pf1-x444.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=h1o1LcgG; dkim-atps=neutral Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 49y9Gh5W0ZzDqVj for ; Thu, 2 Jul 2020 17:49:32 +1000 (AEST) Received: by mail-pf1-x444.google.com with SMTP id t11so7320684pfq.11 for ; Thu, 02 Jul 2020 00:49:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9NJ1Ulnd8FUmc15smZh6pt0VA5vxUSOm2TGGwsdqYoM=; b=h1o1LcgG1xlbxsTXx56FrrXDQFQguL0GPMYYSt5jqmcQS9iaN8toSEHNjvoLQHl5g2 kzDzzyNXxq0XYbgxtm1X7QREncpnWLB1Eu58VIUB0QuwN2jXCVFxKvOCnuLd0vhkqZNf 7XS8ybdePQ/GGeEP9XrvMzir35iJR2SgXnl9wB7NtEhEMpbdkvDZCh2jdW5ci9ZxZRVq wmMJGIH33PvOU3axKN/hDYhkKBb8CGOyBYeL0lG2SXjANW5pAr9EBa8AmAoEI2P0z2df pwJwVQQ+KNZdfS632ULuiUJ7WUxOD2vhmhwGdR20X/XaNBtIygo3+0MK/U7nnju8pVc8 vPmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9NJ1Ulnd8FUmc15smZh6pt0VA5vxUSOm2TGGwsdqYoM=; b=pUdXhHBoT5XafEdR92XTVUCLKINvSaK3SbFl9XaOQXqwcIYwPfIcAEIDaf4yelQtMD gnQvnsFOAlNN/kSd57v81YJPYwT9y59jBuZNOLnOTgAmeEQebP0ShwA6TNPGmqc3QTYm jOKg+BZNn/7wAUwbmEsSOMHt4p4AwJP6OohEe2NkWat8mNAJd2DribsfgFu7Ci0YVFwK bfXNZbytAdSA665fWTAQchIwowxvMAY0P1BiuqiKEjmiHvYY56Jab8XSy/r/nts2d4mp 9ZohZof1E2xRDAZqqzmSmJDHYoVGMb6FoQALEL/pFt5xXnQyRJQo8bwpYuxTi6Vf6H39 Kr0w== X-Gm-Message-State: AOAM531bd3BmrpMX1eispBqdxkSWR/Zryvu5GFBSODJRv9BrrKNvmEMi 8jjN+tZHbbnguvQM4nufV68= X-Google-Smtp-Source: ABdhPJzwYFeDE0Zesg+dB6oDlHyvbtGF9QEI6LdVjc/BuBvcwXIzeNCPUJRMFXKCpdtG6Hbd1Ttt8g== X-Received: by 2002:a63:7741:: with SMTP id s62mr22972293pgc.332.1593676169702; Thu, 02 Jul 2020 00:49:29 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-186-125.tpgi.com.au. [61.68.186.125]) by smtp.gmail.com with ESMTPSA id 17sm6001953pfv.16.2020.07.02.00.49.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jul 2020 00:49:29 -0700 (PDT) From: Nicholas Piggin To: Subject: [PATCH 7/8] powerpc/qspinlock: optimised atomic_try_cmpxchg_lock that adds the lock hint Date: Thu, 2 Jul 2020 17:48:38 +1000 Message-Id: <20200702074839.1057733-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200702074839.1057733-1-npiggin@gmail.com> References: <20200702074839.1057733-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Peter Zijlstra , linuxppc-dev@lists.ozlabs.org, Boqun Feng , linux-kernel@vger.kernel.org, Nicholas Piggin , virtualization@lists.linux-foundation.org, Ingo Molnar , kvm-ppc@vger.kernel.org, Waiman Long , Will Deacon Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This brings the behaviour of the uncontended fast path back to roughly equivalent to simple spinlocks -- a single atomic op with lock hint. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/atomic.h | 28 ++++++++++++++++++++++++++++ arch/powerpc/include/asm/qspinlock.h | 2 +- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/include/asm/atomic.h b/arch/powerpc/include/asm/atomic.h index 498785ffc25f..f6a3d145ffb7 100644 --- a/arch/powerpc/include/asm/atomic.h +++ b/arch/powerpc/include/asm/atomic.h @@ -193,6 +193,34 @@ static __inline__ int atomic_dec_return_relaxed(atomic_t *v) #define atomic_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new)) +/* + * Don't want to override the generic atomic_try_cmpxchg_acquire, because + * we add a lock hint to the lwarx, which may not be wanted for the + * _acquire case (and is not used by the other _acquire variants so it + * would be a surprise). + */ +static __always_inline bool +atomic_try_cmpxchg_lock(atomic_t *v, int *old, int new) +{ + int r, o = *old; + + __asm__ __volatile__ ( +"1:\t" PPC_LWARX(%0,0,%2,1) " # atomic_try_cmpxchg_acquire \n" +" cmpw 0,%0,%3 \n" +" bne- 2f \n" +" stwcx. %4,0,%2 \n" +" bne- 1b \n" +"\t" PPC_ACQUIRE_BARRIER " \n" +"2: \n" + : "=&r" (r), "+m" (v->counter) + : "r" (&v->counter), "r" (o), "r" (new) + : "cr0", "memory"); + + if (unlikely(r != o)) + *old = r; + return likely(r == o); +} + /** * atomic_fetch_add_unless - add unless the number is a given value * @v: pointer of type atomic_t diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h index 997a9a32df77..7091f1ceec3d 100644 --- a/arch/powerpc/include/asm/qspinlock.h +++ b/arch/powerpc/include/asm/qspinlock.h @@ -26,7 +26,7 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) { u32 val = 0; - if (likely(atomic_try_cmpxchg_acquire(&lock->val, &val, _Q_LOCKED_VAL))) + if (likely(atomic_try_cmpxchg_lock(&lock->val, &val, _Q_LOCKED_VAL))) return; queued_spin_lock_slowpath(lock, val);