From patchwork Mon Feb 27 11:36:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Liebler X-Patchwork-Id: 732770 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3vX06q4wcyz9sN1 for ; Mon, 27 Feb 2017 22:37:07 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.b="vbun3OiE"; dkim-atps=neutral DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:subject:to:references:from:date:mime-version :in-reply-to:content-type:message-id; q=dns; s=default; b=jCQAab u7WAUD982XEMUAZcq537Twe/kU8toJm/ecsB5qTNLdgdGZIFvvTAKFgYrm43Y7Yp 3ffPlDojXoMeUdizc+ls/pOcJ2Yd+bmictUeSPsEQSSLToM42UA/ACukDmT062s6 7LZyhmO0sywohoe6x1lU+Kfzbmwpv5+1c4gLU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:subject:to:references:from:date:mime-version :in-reply-to:content-type:message-id; s=default; bh=7LjiSJd68KmW Cn1m76x7BFDOOq4=; b=vbun3OiEvgXRqOzv9prdS/TiSQ2/8Rdn17BUmYPQfIo/ gxyCahIIEzV/ZUi0m4L1qb/VpnJYModq/Lv1OtFiYNKYJnZvohWuCeRBac83YZtE ybhKPqcrIwAja9GAv3I1lS0STsbVpO3ardX6pLZqZASsPCyZbibHefK6hOWyVFI= Received: (qmail 78537 invoked by alias); 27 Feb 2017 11:36:59 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 78511 invoked by uid 89); 27 Feb 2017 11:36:58 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-23.1 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_LAZY_DOMAIN_SECURITY, KHOP_DYNAMIC, RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.2 spammy= X-HELO: mx0a-001b2d01.pphosted.com Subject: Re: [PATCH 1/2] S390: Optimize atomic macros. To: libc-alpha@sourceware.org References: <1479913753-20506-1-git-send-email-stli@linux.vnet.ibm.com> <1479991865.7146.1526.camel@localhost.localdomain> <485d2d38-26ae-b941-e55d-a58f7559049b@linux.vnet.ibm.com> <1481915703.14990.622.camel@redhat.com> From: Stefan Liebler Date: Mon, 27 Feb 2017 12:36:49 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <1481915703.14990.622.camel@redhat.com> X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 17022711-0016-0000-0000-000004468E7D X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17022711-0017-0000-0000-0000269661DA Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-02-27_08:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1702270117 This patch activates c11 atomic builtins by defining USE_ATOMIC_COMPILER_BUILTINS to 1. Note: E.g. in nptl/pthread_key_delete.c if compiled with GCCs 6 and before, an extra stack-frame is generated and the old value is stored on stack before cs instruction but it never loads this value from stack. An unreleased GCC 7 omit those stack operations. E.g. in nptl/pthread_once.c the condition code of cs instruction is evaluated by a sequence of ipm, sra, compare and jump instructions instead of one conditional jump instruction. This also occurs with an unreleased GCC 7. These shortcomings does not really hurt. Nevertheless, the gcc guys are investigating those ones and plan to fix them before GCC 7 release. The atomic_fetch_abc_def c11 builtins are now using load-and-abc instructions on z196 zarch and higher cpus instead of a loop with compare-and-swap instruction. Some of the non-c11 atomic macros from include/atomic.h are now implemented with help of the c11 atomic builtins. The other non-c11 atomic macros are using the macros defined here. Optimizations like using load-and-and for exchanging memory to zero will be done by GCC in future. ChangeLog: * sysdeps/s390/atomic-machine.h (USE_ATOMIC_COMPILER_BUILTINS): Define to 1. (__arch_compare_and_exchange_val_8_acq, __arch_compare_and_exchange_val_16_acq, __arch_compare_and_exchange_val_32_acq, __arch_compare_and_exchange_val_64_acq): Delete macro. (atomic_compare_and_exchange_val_acq, atomic_compare_and_exchange_val_rel, atomic_compare_and_exchange_bool_acq, catomic_compare_and_exchange_bool_acq, atomic_exchange_acq, atomic_exchange_rel, atomic_exchange_and_add_acq, atomic_exchange_and_add_rel, catomic_exchange_and_add, atomic_or_val, atomic_or, catomic_or, atomic_bit_test_set, atomic_and_val, atomic_and, catomic_and): Define macros with help of c11 atomic builtins. commit 187e63cf8411585313051f020a39f8b6010280a9 Author: Stefan Liebler Date: Mon Feb 27 12:23:32 2017 +0100 S390: Optimize atomic macros. This patch activates c11 atomic builtins by defining USE_ATOMIC_COMPILER_BUILTINS to 1. Note: E.g. in nptl/pthread_key_delete.c if compiled with GCCs 6 and before, an extra stack-frame is generated and the old value is stored on stack before cs instruction but it never loads this value from stack. An unreleased GCC 7 omit those stack operations. E.g. in nptl/pthread_once.c the condition code of cs instruction is evaluated by a sequence of ipm, sra, compare and jump instructions instead of one conditional jump instruction. This also occurs with an unreleased GCC 7. These shortcomings does not really hurt. Nevertheless, the gcc guys are investigating those ones and plan to fix them before GCC 7 release. The atomic_fetch_abc_def c11 builtins are now using load-and-abc instructions on z196 zarch and higher cpus instead of a loop with compare-and-swap instruction. Some of the non-c11 atomic macros from include/atomic.h are now implemented with help of the c11 atomic builtins. The other non-c11 atomic macros are using the macros defined here. ChangeLog: * sysdeps/s390/atomic-machine.h (USE_ATOMIC_COMPILER_BUILTINS): Define to 1. (__arch_compare_and_exchange_val_8_acq, __arch_compare_and_exchange_val_16_acq, __arch_compare_and_exchange_val_32_acq, __arch_compare_and_exchange_val_64_acq): Delete macro. (atomic_compare_and_exchange_val_acq, atomic_compare_and_exchange_val_rel, atomic_compare_and_exchange_bool_acq, catomic_compare_and_exchange_bool_acq, atomic_exchange_acq, atomic_exchange_rel, atomic_exchange_and_add_acq, atomic_exchange_and_add_rel, catomic_exchange_and_add, atomic_or_val, atomic_or, catomic_or, atomic_bit_test_set, atomic_and_val, atomic_and, catomic_and): Define macros with help of c11 atomic builtins. diff --git a/sysdeps/s390/atomic-machine.h b/sysdeps/s390/atomic-machine.h index 211d3d6..6f0a6fe 100644 --- a/sysdeps/s390/atomic-machine.h +++ b/sysdeps/s390/atomic-machine.h @@ -43,78 +43,117 @@ typedef uintptr_t uatomicptr_t; typedef intmax_t atomic_max_t; typedef uintmax_t uatomic_max_t; -#define USE_ATOMIC_COMPILER_BUILTINS 0 +/* Activate all c11 atomic builtins. + Note: + E.g. in nptl/pthread_key_delete.c if compiled with GCCs 6 and before, + an extra stack-frame is generated and the old value is stored on stack + before cs instruction but it never loads this value from stack. + An unreleased GCC 7 omit those stack operations. -#define __arch_compare_and_exchange_val_8_acq(mem, newval, oldval) \ - (abort (), (__typeof (*mem)) 0) + E.g. in nptl/pthread_once.c the condition code of cs instruction is + evaluated by a sequence of ipm, sra, compare and jump instructions instead + of one conditional jump instruction. This also occurs with an unreleased + GCC 7. -#define __arch_compare_and_exchange_val_16_acq(mem, newval, oldval) \ - (abort (), (__typeof (*mem)) 0) - -#define __arch_compare_and_exchange_val_32_acq(mem, newval, oldval) \ - ({ __typeof (mem) __archmem = (mem); \ - __typeof (*mem) __archold = (oldval); \ - __asm__ __volatile__ ("cs %0,%2,%1" \ - : "+d" (__archold), "=Q" (*__archmem) \ - : "d" (newval), "m" (*__archmem) : "cc", "memory" ); \ - __archold; }) + The atomic_fetch_abc_def c11 builtins are now using load-and-abc instructions + on z196 zarch and higher cpus instead of a loop with compare-and-swap + instruction. */ +#define USE_ATOMIC_COMPILER_BUILTINS 1 #ifdef __s390x__ # define __HAVE_64B_ATOMICS 1 -# define __arch_compare_and_exchange_val_64_acq(mem, newval, oldval) \ - ({ __typeof (mem) __archmem = (mem); \ - __typeof (*mem) __archold = (oldval); \ - __asm__ __volatile__ ("csg %0,%2,%1" \ - : "+d" (__archold), "=Q" (*__archmem) \ - : "d" ((long) (newval)), "m" (*__archmem) : "cc", "memory" ); \ - __archold; }) #else # define __HAVE_64B_ATOMICS 0 -/* For 31 bit we do not really need 64-bit compare-and-exchange. We can - implement them by use of the csd instruction. The straightforward - implementation causes warnings so we skip the definition for now. */ -# define __arch_compare_and_exchange_val_64_acq(mem, newval, oldval) \ - (abort (), (__typeof (*mem)) 0) #endif +/* Implement some of the non-c11 atomic macros from include/atomic.h + with help of the c11 atomic builtins. The other non-c11 atomic macros + are using the macros defined here. */ + +/* Atomically store NEWVAL in *MEM if *MEM is equal to OLDVAL. + Return the old *MEM value. */ +#define atomic_compare_and_exchange_val_acq(mem, newval, oldval) \ + ({ __atomic_check_size((mem)); \ + typeof ((__typeof (*(mem))) *(mem)) __atg1_oldval = (oldval); \ + __atomic_compare_exchange_n (mem, (void *) &__atg1_oldval, \ + newval, 1, __ATOMIC_ACQUIRE, \ + __ATOMIC_RELAXED); \ + __atg1_oldval; }) +#define atomic_compare_and_exchange_val_rel(mem, newval, oldval) \ + ({ __atomic_check_size((mem)); \ + typeof ((__typeof (*(mem))) *(mem)) __atg1_2_oldval = (oldval); \ + __atomic_compare_exchange_n (mem, (void *) &__atg1_2_oldval, \ + newval, 1, __ATOMIC_RELEASE, \ + __ATOMIC_RELAXED); \ + __atg1_2_oldval; }) + +/* Atomically store NEWVAL in *MEM if *MEM is equal to OLDVAL. + Return zero if *MEM was changed or non-zero if no exchange happened. */ +#define atomic_compare_and_exchange_bool_acq(mem, newval, oldval) \ + ({ __atomic_check_size((mem)); \ + typeof ((__typeof (*(mem))) *(mem)) __atg2_oldval = (oldval); \ + !__atomic_compare_exchange_n (mem, (void *) &__atg2_oldval, newval, \ + 1, __ATOMIC_ACQUIRE, \ + __ATOMIC_RELAXED); }) +#define catomic_compare_and_exchange_bool_acq(mem, newval, oldval) \ + atomic_compare_and_exchange_bool_acq (mem, newval, oldval) + /* Store NEWVALUE in *MEM and return the old value. */ -/* On s390, the atomic_exchange_acq is different from generic implementation, - because the generic one does not use the condition-code of cs-instruction - to determine if looping is needed. Instead it saves the old-value and - compares it against old-value returned by cs-instruction. */ -#ifdef __s390x__ -# define atomic_exchange_acq(mem, newvalue) \ - ({ __typeof (mem) __atg5_memp = (mem); \ - __typeof (*(mem)) __atg5_oldval = *__atg5_memp; \ - __typeof (*(mem)) __atg5_value = (newvalue); \ - if (sizeof (*mem) == 4) \ - __asm__ __volatile__ ("0: cs %0,%2,%1\n" \ - " jl 0b" \ - : "+d" (__atg5_oldval), "=Q" (*__atg5_memp) \ - : "d" (__atg5_value), "m" (*__atg5_memp) \ - : "cc", "memory" ); \ - else if (sizeof (*mem) == 8) \ - __asm__ __volatile__ ("0: csg %0,%2,%1\n" \ - " jl 0b" \ - : "+d" ( __atg5_oldval), "=Q" (*__atg5_memp) \ - : "d" ((long) __atg5_value), "m" (*__atg5_memp) \ - : "cc", "memory" ); \ - else \ - abort (); \ - __atg5_oldval; }) -#else -# define atomic_exchange_acq(mem, newvalue) \ - ({ __typeof (mem) __atg5_memp = (mem); \ - __typeof (*(mem)) __atg5_oldval = *__atg5_memp; \ - __typeof (*(mem)) __atg5_value = (newvalue); \ - if (sizeof (*mem) == 4) \ - __asm__ __volatile__ ("0: cs %0,%2,%1\n" \ - " jl 0b" \ - : "+d" (__atg5_oldval), "=Q" (*__atg5_memp) \ - : "d" (__atg5_value), "m" (*__atg5_memp) \ - : "cc", "memory" ); \ - else \ - abort (); \ - __atg5_oldval; }) -#endif +#define atomic_exchange_acq(mem, newvalue) \ + ({ __atomic_check_size((mem)); \ + __atomic_exchange_n (mem, newvalue, __ATOMIC_ACQUIRE); }) +#define atomic_exchange_rel(mem, newvalue) \ + ({ __atomic_check_size((mem)); \ + __atomic_exchange_n (mem, newvalue, __ATOMIC_RELEASE); }) + +/* Add VALUE to *MEM and return the old value of *MEM. */ +/* The gcc builtin uses load-and-add instruction on z196 zarch and higher cpus + instead of a loop with compare-and-swap instruction. */ +# define atomic_exchange_and_add_acq(mem, operand) \ + ({ __atomic_check_size((mem)); \ + __atomic_fetch_add ((mem), (operand), __ATOMIC_ACQUIRE); }) +# define atomic_exchange_and_add_rel(mem, operand) \ + ({ __atomic_check_size((mem)); \ + __atomic_fetch_add ((mem), (operand), __ATOMIC_RELEASE); }) +#define catomic_exchange_and_add(mem, value) \ + atomic_exchange_and_add (mem, value) + +/* Atomically *mem |= mask and return the old value of *mem. */ +/* The gcc builtin uses load-and-or instruction on z196 zarch and higher cpus + instead of a loop with compare-and-swap instruction. */ +#define atomic_or_val(mem, operand) \ + ({ __atomic_check_size((mem)); \ + __atomic_fetch_or ((mem), (operand), __ATOMIC_ACQUIRE); }) +/* Atomically *mem |= mask. */ +#define atomic_or(mem, mask) \ + do { \ + atomic_or_val (mem, mask); \ + } while (0) +#define catomic_or(mem, mask) \ + atomic_or (mem, mask) + +/* Atomically *mem |= 1 << bit and return true if the bit was set in old value + of *mem. */ +/* The load-and-or instruction is used on z196 zarch and higher cpus + instead of a loop with compare-and-swap instruction. */ +#define atomic_bit_test_set(mem, bit) \ + ({ __typeof (*(mem)) __atg14_old; \ + __typeof (mem) __atg14_memp = (mem); \ + __typeof (*(mem)) __atg14_mask = ((__typeof (*(mem))) 1 << (bit)); \ + __atg14_old = atomic_or_val (__atg14_memp, __atg14_mask); \ + __atg14_old & __atg14_mask; }) + +/* Atomically *mem &= mask and return the old value of *mem. */ +/* The gcc builtin uses load-and-and instruction on z196 zarch and higher cpus + instead of a loop with compare-and-swap instruction. */ +#define atomic_and_val(mem, operand) \ + ({ __atomic_check_size((mem)); \ + __atomic_fetch_and ((mem), (operand), __ATOMIC_ACQUIRE); }) +/* Atomically *mem &= mask. */ +#define atomic_and(mem, mask) \ + do { \ + atomic_and_val (mem, mask); \ + } while (0) +#define catomic_and(mem, mask) \ + atomic_and(mem, mask)