From patchwork Wed Sep 18 01:58:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 1163660 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gcc.gnu.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=gcc-patches-return-509145-incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b="lhvgWpdy"; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="pcobBO1G"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Y3775nqwz9sCJ for ; Wed, 18 Sep 2019 11:58:59 +1000 (AEST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=RJ5kS5Xi4f8QHdi9FFGGRNTHPbr3CalntAbKG+5wj11JnFa4ToEN5 H8+CEufcB8hmxtDdJwiX0qddCs2wreUiMKXQ1hDdy9vFPqGqcTJfZen7L+VP1ldY fMUr2U3lJ8xNM3U6Jbvvw4OkcDjrWAZXOXKyBxMuAOAumVCyE5lMeU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=rNeulZTJHdDjvRZ6EJWt7fPx+r0=; b=lhvgWpdyl/qf/GZRCOaK DM2X6duZ02IIqBrb1vhycxOx+OB1U5NM4Kn7lzh8ZQaqF+vmmu6miAb9d2+zsI9n P0SQovyBKQ9LHS8hLLbyM1qX3Bww4xSKzl66F8a+rVlZhOprFmIftxIyes11Uon1 5iw0r0YMqICy6ofKmsC1S28= Received: (qmail 120691 invoked by alias); 18 Sep 2019 01:58:27 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 120623 invoked by uid 89); 18 Sep 2019 01:58:27 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-23.2 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.1 spammy=HX-Received:90a, emit_insn X-HELO: mail-pf1-f172.google.com Received: from mail-pf1-f172.google.com (HELO mail-pf1-f172.google.com) (209.85.210.172) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 18 Sep 2019 01:58:25 +0000 Received: by mail-pf1-f172.google.com with SMTP id q21so3285002pfn.11 for ; Tue, 17 Sep 2019 18:58:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LCor1pkgTu7UhqGMy/ZO4Ubcr0JnZ56kgSWlTCKyc6Q=; b=pcobBO1GrS2eJYklbV1NAQzPyDzN2g8h7LH3LkXmv5Qm4ZmZCGU5E2wrZUZ1SbL9JH +lOMmQe9E6j3zenikDx4II+naFTIupEmIkGbXz+LxcZ6pfPopMaO2AsFfaizXA5lZr05 cB0HGAbhEBS0OHkrUsGS8S0TfyrRyAhpeuhj9OYOaFkKnj8RjTaJr3aZuTioOoafYz6Q 5OR+7zrVWGl7RQkQT3RaxBVK/5Qe63XAwObp+a0v6IPZ38/EEVDv0ERNSXzsuRrhyLpJ 5yPHwYKNXItYg7uBSDkSJK0YKu75fq8kJpMba3iOCvw5hujy62492bWdOHT3kq20kUCC VTOw== Received: from localhost.localdomain (97-113-7-119.tukw.qwest.net. [97.113.7.119]) by smtp.gmail.com with ESMTPSA id d10sm6614306pfn.50.2019.09.17.18.58.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Sep 2019 18:58:22 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: Wilco.Dijkstra@arm.com, kyrylo.tkachov@foss.arm.com, Marcus.Shawcroft@arm.com, James.Greenhalgh@arm.com Subject: [PATCH, AArch64 v4 2/6] aarch64: Implement TImode compare-and-swap Date: Tue, 17 Sep 2019 18:58:13 -0700 Message-Id: <20190918015817.24408-3-richard.henderson@linaro.org> In-Reply-To: <20190918015817.24408-1-richard.henderson@linaro.org> References: <20190918015817.24408-1-richard.henderson@linaro.org> This pattern will only be used with the __sync functions, because we do not yet have a bare TImode atomic load. * config/aarch64/aarch64.c (aarch64_gen_compare_reg): Add support for NE comparison of TImode values. (aarch64_emit_load_exclusive): Add support for TImode. (aarch64_emit_store_exclusive): Likewise. (aarch64_split_compare_and_swap): Disable strong_zero_p for TImode. * config/aarch64/atomics.md (@atomic_compare_and_swap): Change iterator from ALLI to ALLI_TI. (@atomic_compare_and_swap): New. (@atomic_compare_and_swap_lse): New. (aarch64_load_exclusive_pair): New. (aarch64_store_exclusive_pair): New. * config/aarch64/iterators.md (JUST_TI): New. --- gcc/config/aarch64/aarch64.c | 48 ++++++++++++++--- gcc/config/aarch64/atomics.md | 93 +++++++++++++++++++++++++++++++-- gcc/config/aarch64/iterators.md | 3 ++ 3 files changed, 131 insertions(+), 13 deletions(-) diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 99d51e2aef9..a5c4f55627d 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -2039,10 +2039,33 @@ emit_set_insn (rtx x, rtx y) rtx aarch64_gen_compare_reg (RTX_CODE code, rtx x, rtx y) { - machine_mode mode = SELECT_CC_MODE (code, x, y); - rtx cc_reg = gen_rtx_REG (mode, CC_REGNUM); + machine_mode cmp_mode = GET_MODE (x); + machine_mode cc_mode; + rtx cc_reg; - emit_set_insn (cc_reg, gen_rtx_COMPARE (mode, x, y)); + if (cmp_mode == TImode) + { + gcc_assert (code == NE); + + cc_mode = CCmode; + cc_reg = gen_rtx_REG (cc_mode, CC_REGNUM); + + rtx x_lo = operand_subword (x, 0, 0, TImode); + rtx y_lo = operand_subword (y, 0, 0, TImode); + emit_set_insn (cc_reg, gen_rtx_COMPARE (cc_mode, x_lo, y_lo)); + + rtx x_hi = operand_subword (x, 1, 0, TImode); + rtx y_hi = operand_subword (y, 1, 0, TImode); + emit_insn (gen_ccmpdi (cc_reg, cc_reg, x_hi, y_hi, + gen_rtx_EQ (cc_mode, cc_reg, const0_rtx), + GEN_INT (AARCH64_EQ))); + } + else + { + cc_mode = SELECT_CC_MODE (code, x, y); + cc_reg = gen_rtx_REG (cc_mode, CC_REGNUM); + emit_set_insn (cc_reg, gen_rtx_COMPARE (cc_mode, x, y)); + } return cc_reg; } @@ -2593,7 +2616,6 @@ aarch64_zero_extend_const_eq (machine_mode xmode, rtx x, gcc_assert (r != NULL); return rtx_equal_p (x, r); } - /* Return TARGET if it is nonnull and a register of mode MODE. Otherwise, return a fresh register of mode MODE if we can, @@ -16814,16 +16836,26 @@ static void aarch64_emit_load_exclusive (machine_mode mode, rtx rval, rtx mem, rtx model_rtx) { - emit_insn (gen_aarch64_load_exclusive (mode, rval, mem, model_rtx)); + if (mode == TImode) + emit_insn (gen_aarch64_load_exclusive_pair (gen_lowpart (DImode, rval), + gen_highpart (DImode, rval), + mem, model_rtx)); + else + emit_insn (gen_aarch64_load_exclusive (mode, rval, mem, model_rtx)); } /* Emit store exclusive. */ static void aarch64_emit_store_exclusive (machine_mode mode, rtx bval, - rtx rval, rtx mem, rtx model_rtx) + rtx mem, rtx rval, rtx model_rtx) { - emit_insn (gen_aarch64_store_exclusive (mode, bval, rval, mem, model_rtx)); + if (mode == TImode) + emit_insn (gen_aarch64_store_exclusive_pair + (bval, mem, operand_subword (rval, 0, 0, TImode), + operand_subword (rval, 1, 0, TImode), model_rtx)); + else + emit_insn (gen_aarch64_store_exclusive (mode, bval, mem, rval, model_rtx)); } /* Mark the previous jump instruction as unlikely. */ @@ -16950,7 +16982,7 @@ aarch64_split_compare_and_swap (rtx operands[]) CBNZ scratch, .label1 .label2: CMP rval, 0. */ - bool strong_zero_p = !is_weak && oldval == const0_rtx; + bool strong_zero_p = !is_weak && oldval == const0_rtx && mode != TImode; label1 = NULL; if (!is_weak) diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index a679270cd38..f8bdd048b37 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -21,11 +21,11 @@ ;; Instruction patterns. (define_expand "@atomic_compare_and_swap" - [(match_operand:SI 0 "register_operand") ;; bool out - (match_operand:ALLI 1 "register_operand") ;; val out - (match_operand:ALLI 2 "aarch64_sync_memory_operand") ;; memory - (match_operand:ALLI 3 "nonmemory_operand") ;; expected - (match_operand:ALLI 4 "aarch64_reg_or_zero") ;; desired + [(match_operand:SI 0 "register_operand" "") ;; bool out + (match_operand:ALLI_TI 1 "register_operand" "") ;; val out + (match_operand:ALLI_TI 2 "aarch64_sync_memory_operand" "") ;; memory + (match_operand:ALLI_TI 3 "nonmemory_operand" "") ;; expected + (match_operand:ALLI_TI 4 "aarch64_reg_or_zero" "") ;; desired (match_operand:SI 5 "const_int_operand") ;; is_weak (match_operand:SI 6 "const_int_operand") ;; mod_s (match_operand:SI 7 "const_int_operand")] ;; mod_f @@ -88,6 +88,30 @@ } ) +(define_insn_and_split "@aarch64_compare_and_swap" + [(set (reg:CC CC_REGNUM) ;; bool out + (unspec_volatile:CC [(const_int 0)] UNSPECV_ATOMIC_CMPSW)) + (set (match_operand:JUST_TI 0 "register_operand" "=&r") ;; val out + (match_operand:JUST_TI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory + (set (match_dup 1) + (unspec_volatile:JUST_TI + [(match_operand:JUST_TI 2 "aarch64_reg_or_zero" "rZ") ;; expect + (match_operand:JUST_TI 3 "aarch64_reg_or_zero" "rZ") ;; desired + (match_operand:SI 4 "const_int_operand") ;; is_weak + (match_operand:SI 5 "const_int_operand") ;; mod_s + (match_operand:SI 6 "const_int_operand")] ;; mod_f + UNSPECV_ATOMIC_CMPSW)) + (clobber (match_scratch:SI 7 "=&r"))] + "" + "#" + "&& reload_completed" + [(const_int 0)] + { + aarch64_split_compare_and_swap (operands); + DONE; + } +) + (define_insn "@aarch64_compare_and_swap_lse" [(set (match_operand:SI 0 "register_operand" "+r") ;; val out (zero_extend:SI @@ -133,6 +157,28 @@ return "casal\t%0, %2, %1"; }) +(define_insn "@aarch64_compare_and_swap_lse" + [(set (match_operand:JUST_TI 0 "register_operand" "+r") ;; val out + (match_operand:JUST_TI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory + (set (match_dup 1) + (unspec_volatile:JUST_TI + [(match_dup 0) ;; expect + (match_operand:JUST_TI 2 "register_operand" "r") ;; desired + (match_operand:SI 3 "const_int_operand")] ;; mod_s + UNSPECV_ATOMIC_CMPSW))] + "TARGET_LSE" +{ + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model)) + return "casp\t%0, %R0, %2, %R2, %1"; + else if (is_mm_acquire (model) || is_mm_consume (model)) + return "caspa\t%0, %R0, %2, %R2, %1"; + else if (is_mm_release (model)) + return "caspl\t%0, %R0, %2, %R2, %1"; + else + return "caspal\t%0, %R0, %2, %R2, %1"; +}) + (define_expand "atomic_exchange" [(match_operand:ALLI 0 "register_operand") (match_operand:ALLI 1 "aarch64_sync_memory_operand") @@ -581,6 +627,24 @@ } ) +(define_insn "aarch64_load_exclusive_pair" + [(set (match_operand:DI 0 "register_operand" "=r") + (unspec_volatile:DI + [(match_operand:TI 2 "aarch64_sync_memory_operand" "Q") + (match_operand:SI 3 "const_int_operand")] + UNSPECV_LX)) + (set (match_operand:DI 1 "register_operand" "=r") + (unspec_volatile:DI [(match_dup 2) (match_dup 3)] UNSPECV_LX))] + "" + { + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model) || is_mm_consume (model) || is_mm_release (model)) + return "ldxp\t%0, %1, %2"; + else + return "ldaxp\t%0, %1, %2"; + } +) + (define_insn "@aarch64_store_exclusive" [(set (match_operand:SI 0 "register_operand" "=&r") (unspec_volatile:SI [(const_int 0)] UNSPECV_SX)) @@ -599,6 +663,25 @@ } ) +(define_insn "aarch64_store_exclusive_pair" + [(set (match_operand:SI 0 "register_operand" "=&r") + (unspec_volatile:SI [(const_int 0)] UNSPECV_SX)) + (set (match_operand:TI 1 "aarch64_sync_memory_operand" "=Q") + (unspec_volatile:TI + [(match_operand:DI 2 "aarch64_reg_or_zero" "rZ") + (match_operand:DI 3 "aarch64_reg_or_zero" "rZ") + (match_operand:SI 4 "const_int_operand")] + UNSPECV_SX))] + "" + { + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model) || is_mm_consume (model) || is_mm_acquire (model)) + return "stxp\t%w0, %x2, %x3, %1"; + else + return "stlxp\t%w0, %x2, %x3, %1"; + } +) + (define_expand "mem_thread_fence" [(match_operand:SI 0 "const_int_operand")] "" diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md index d23f0fcbc2f..03b3ce36302 100644 --- a/gcc/config/aarch64/iterators.md +++ b/gcc/config/aarch64/iterators.md @@ -29,6 +29,9 @@ ;; Iterator for HI, SI, DI, some instructions can only work on these modes. (define_mode_iterator GPI_I16 [(HI "AARCH64_ISA_F16") SI DI]) +;; "Iterator" for just TI -- features like @pattern only work with iterators. +(define_mode_iterator JUST_TI [TI]) + ;; Iterator for QI and HI modes (define_mode_iterator SHORT [QI HI])