From patchwork Mon Apr 18 22:05:57 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bill Schmidt X-Patchwork-Id: 611919 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3qpj0K6mn3z9sXR for ; Tue, 19 Apr 2016 08:06:25 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b=tAsXEnuv; dkim-atps=neutral DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender :subject:from:to:cc:content-type:date:message-id:mime-version :content-transfer-encoding; q=dns; s=default; b=tGusZ0XfM/q/8LC1 zZ7EVZRZxw82ttoIOjoJ5L2hCaM9Z7NjIEfbbbfeIEg0BbZwKChahY1QSUlUskpo Ra40S1uGxGT7YXVltUkl4NXUO+MXdzW0fPgWvwT6rPL5qbPJlUumYbyOEqTyWQ1T 6C6cHiIlgRNC2KWMlF1E/oaSAvI= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender :subject:from:to:cc:content-type:date:message-id:mime-version :content-transfer-encoding; s=default; bh=cWXdSi5JTYk0drocA9ZoUr pG9Fg=; b=tAsXEnuvO8wkmwkpKF+0IFVE61U42nF25SzJsyDZICN9+peEMqKpA6 z633Go5/qkAGngsyyCVneJsw23BTnrNNW9zBI0VAIY8QW/RhDKp9E3qlPA1+Yq+S /4qqt4EjZBRD8/VQApAx9g9n5qvdGE+UFmayMjnRGI3X3AfcrjXWc= Received: (qmail 123068 invoked by alias); 18 Apr 2016 22:06:16 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 122467 invoked by uid 89); 18 Apr 2016 22:06:16 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=0.1 required=5.0 tests=AWL, BAYES_50, KAM_ASCII_DIVIDERS, KAM_LAZY_DOMAIN_SECURITY, RP_MATCHES_RCVD autolearn=no version=3.3.2 spammy=H*MI:108, H*M:108, gen_rtx_UNSPEC, curr_insn X-HELO: e34.co.us.ibm.com Received: from e34.co.us.ibm.com (HELO e34.co.us.ibm.com) (32.97.110.152) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (CAMELLIA256-SHA encrypted) ESMTPS; Mon, 18 Apr 2016 22:06:05 +0000 Received: from localhost by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 18 Apr 2016 16:06:02 -0600 Received: from d03dlp01.boulder.ibm.com (9.17.202.177) by e34.co.us.ibm.com (192.168.1.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Mon, 18 Apr 2016 16:06:01 -0600 X-IBM-Helo: d03dlp01.boulder.ibm.com X-IBM-MailFrom: wschmidt@linux.vnet.ibm.com X-IBM-RcptTo: gcc-patches@gcc.gnu.org Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by d03dlp01.boulder.ibm.com (Postfix) with ESMTP id B5E531FF004B for ; Mon, 18 Apr 2016 15:54:08 -0600 (MDT) Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u3IM60M228770456 for ; Mon, 18 Apr 2016 22:06:00 GMT Received: from d01av04.pok.ibm.com (localhost [127.0.0.1]) by d01av04.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u3IM5xMZ004425 for ; Mon, 18 Apr 2016 18:05:59 -0400 Received: from [9.65.78.93] (sig-9-65-78-93.ibm.com [9.65.78.93]) by d01av04.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u3IM5wJm004381; Mon, 18 Apr 2016 18:05:58 -0400 Subject: [PATCH, rs6000] Expand vec_ld and vec_st during parsing to improve performance From: Bill Schmidt To: gcc-patches@gcc.gnu.org Cc: dje.gcc@gmail.com Date: Mon, 18 Apr 2016 17:05:57 -0500 Message-ID: <1461017157.18355.108.camel@oc8801110288.ibm.com> Mime-Version: 1.0 X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16041822-0017-0000-0000-000023540C96 X-IsSubscribed: yes Hi, Expanding built-ins in the usual way (leaving them as calls until expanding into RTL) restricts the amount of optimization that can be performed on the code represented by the built-ins. This has been observed to be particularly bad for the vec_ld and vec_st built-ins on PowerPC, which represent the lvx and stvx instructions. Currently these are expanded into UNSPECs that are left untouched by the optimizers, so no redundant load or store elimination can take place. For certain idiomatic usages, this leads to very bad performance. Initially I planned to just change the UNSPEC representation to RTL that directly expresses the address masking implicit in lvx and stvx. This turns out to be only partially successful in improving performance. Among other things, by the time we reach RTL we have lost track of the __restrict__ attribute, leading to more appearances of may-alias relationships than should really be present. Instead, this patch expands the built-ins during parsing so that they are exposed to all GIMPLE optimizations as well. This works well for vec_ld and vec_st. It is also possible for programmers to instead use __builtin_altivec_lvx_ and __builtin_altivec_stvx_. These are not so easy to catch during parsing, since they are not processed by the overloaded built-in function table. For these, I am currently falling back to expansion during RTL while still exposing the address-masking semantics, which seems ok for these somewhat obscure built-ins. At some future time we may decide to handle them similarly to vec_ld and vec_st. For POWER8 little-endian only, the loads and stores during expand time require some special handling, since the POWER8 expanders want to convert these to lxvd2x/xxswapd and xxswapd/stxvd2x. To deal with this, I've added an extra pre-pass to the swap optimization phase that recognizes the lvx and stvx patterns and canonicalizes them so they'll be properly recognized. This isn't an issue for earlier or later processors, or for big-endian POWER8, so doing this as part of swap optimization is appropriate. We have a lot of existing test cases for this code, which proved very useful in discovering bugs, so I haven't seen a reason to add any new tests. The patch is fairly large, but it isn't feasible to break it up into smaller units without leaving something in a broken state. So I will have to just apologize for the size and leave it at that. Sorry! :) Bootstrapped and tested successfully on powerpc64le-unknown-linux-gnu, and on powerpc64-unknown-linux-gnu (-m32 and -m64) with no regressions. Is this ok for trunk after GCC 6 releases? Thanks, Bill 2016-04-18 Bill Schmidt * config/rs6000/altivec.md (altivec_lvx_): Remove. (altivec_lvx__internal): Document. (altivec_lvx__2op): New define_insn. (altivec_lvx__1op): Likewise. (altivec_lvx__2op_si): Likewise. (altivec_lvx__1op_si): Likewise. (altivec_stvx_): Remove. (altivec_stvx__internal): Document. (altivec_stvx__2op): New define_insn. (altivec_stvx__1op): Likewise. (altivec_stvx__2op_si): Likewise. (altivec_stvx__1op_si): Likewise. * config/rs6000/rs6000-c.c (altivec_resolve_overloaded_builtin): Expand vec_ld and vec_st during parsing. * config/rs6000/rs6000.c (altivec_expand_lvx_be): Commentary changes. (altivec_expand_stvx_be): Likewise. (altivec_expand_lv_builtin): Expand lvx built-ins to expose the address-masking behavior in RTL. (altivec_expand_stv_builtin): Expand stvx built-ins to expose the address-masking behavior in RTL. (altivec_expand_builtin): Change builtin code arguments for calls to altivec_expand_stv_builtin and altivec_expand_lv_builtin. (insn_is_swappable_p): Avoid incorrect swap optimization in the presence of lvx/stvx patterns. (alignment_with_canonical_addr): New function. (alignment_mask): Likewise. (find_alignment_op): Likewise. (combine_lvx_pattern): Likewise. (combine_stvx_pattern): Likewise. (combine_lvx_stvx_patterns): Likewise. (rs6000_analyze_swaps): Perform a pre-pass to recognize lvx and stvx patterns from expand. * config/rs6000/vector.md (vector_altivec_load_): Use new expansions. (vector_altivec_store_): Likewise. Index: gcc/config/rs6000/altivec.md =================================================================== --- gcc/config/rs6000/altivec.md (revision 235090) +++ gcc/config/rs6000/altivec.md (working copy) @@ -2514,20 +2514,9 @@ "lvxl %0,%y1" [(set_attr "type" "vecload")]) -(define_expand "altivec_lvx_" - [(parallel - [(set (match_operand:VM2 0 "register_operand" "=v") - (match_operand:VM2 1 "memory_operand" "Z")) - (unspec [(const_int 0)] UNSPEC_LVX)])] - "TARGET_ALTIVEC" -{ - if (!BYTES_BIG_ENDIAN && VECTOR_ELT_ORDER_BIG) - { - altivec_expand_lvx_be (operands[0], operands[1], mode, UNSPEC_LVX); - DONE; - } -}) - +; This version of lvx is used only in cases where we need to force an lvx +; over any other load, and we don't care about losing CSE opportunities. +; Its primary use is for prologue register saves. (define_insn "altivec_lvx__internal" [(parallel [(set (match_operand:VM2 0 "register_operand" "=v") @@ -2537,20 +2526,45 @@ "lvx %0,%y1" [(set_attr "type" "vecload")]) -(define_expand "altivec_stvx_" - [(parallel - [(set (match_operand:VM2 0 "memory_operand" "=Z") - (match_operand:VM2 1 "register_operand" "v")) - (unspec [(const_int 0)] UNSPEC_STVX)])] - "TARGET_ALTIVEC" -{ - if (!BYTES_BIG_ENDIAN && VECTOR_ELT_ORDER_BIG) - { - altivec_expand_stvx_be (operands[0], operands[1], mode, UNSPEC_STVX); - DONE; - } -}) +; The next two patterns embody what lvx should usually look like. +(define_insn "altivec_lvx__2op" + [(set (match_operand:VM2 0 "register_operand" "=v") + (mem:VM2 (and:DI (plus:DI (match_operand:DI 1 "register_operand" "b") + (match_operand:DI 2 "register_operand" "r")) + (const_int -16))))] + "TARGET_ALTIVEC && TARGET_64BIT" + "lvx %0,%1,%2" + [(set_attr "type" "vecload")]) +(define_insn "altivec_lvx__1op" + [(set (match_operand:VM2 0 "register_operand" "=v") + (mem:VM2 (and:DI (match_operand:DI 1 "register_operand" "r") + (const_int -16))))] + "TARGET_ALTIVEC && TARGET_64BIT" + "lvx %0,0,%1" + [(set_attr "type" "vecload")]) + +; 32-bit versions of the above. +(define_insn "altivec_lvx__2op_si" + [(set (match_operand:VM2 0 "register_operand" "=v") + (mem:VM2 (and:SI (plus:SI (match_operand:SI 1 "register_operand" "b") + (match_operand:SI 2 "register_operand" "r")) + (const_int -16))))] + "TARGET_ALTIVEC && TARGET_32BIT" + "lvx %0,%1,%2" + [(set_attr "type" "vecload")]) + +(define_insn "altivec_lvx__1op_si" + [(set (match_operand:VM2 0 "register_operand" "=v") + (mem:VM2 (and:SI (match_operand:SI 1 "register_operand" "r") + (const_int -16))))] + "TARGET_ALTIVEC && TARGET_32BIT" + "lvx %0,0,%1" + [(set_attr "type" "vecload")]) + +; This version of stvx is used only in cases where we need to force an stvx +; over any other store, and we don't care about losing CSE opportunities. +; Its primary use is for epilogue register restores. (define_insn "altivec_stvx__internal" [(parallel [(set (match_operand:VM2 0 "memory_operand" "=Z") @@ -2560,6 +2574,42 @@ "stvx %1,%y0" [(set_attr "type" "vecstore")]) +; The next two patterns embody what stvx should usually look like. +(define_insn "altivec_stvx__2op" + [(set (mem:VM2 (and:DI (plus:DI (match_operand:DI 1 "register_operand" "b") + (match_operand:DI 2 "register_operand" "r")) + (const_int -16))) + (match_operand:VM2 0 "register_operand" "v"))] + "TARGET_ALTIVEC && TARGET_64BIT" + "stvx %0,%1,%2" + [(set_attr "type" "vecstore")]) + +(define_insn "altivec_stvx__1op" + [(set (mem:VM2 (and:DI (match_operand:DI 1 "register_operand" "r") + (const_int -16))) + (match_operand:VM2 0 "register_operand" "v"))] + "TARGET_ALTIVEC && TARGET_64BIT" + "stvx %0,0,%1" + [(set_attr "type" "vecstore")]) + +; 32-bit versions of the above. +(define_insn "altivec_stvx__2op_si" + [(set (mem:VM2 (and:SI (plus:SI (match_operand:SI 1 "register_operand" "b") + (match_operand:SI 2 "register_operand" "r")) + (const_int -16))) + (match_operand:VM2 0 "register_operand" "v"))] + "TARGET_ALTIVEC && TARGET_32BIT" + "stvx %0,%1,%2" + [(set_attr "type" "vecstore")]) + +(define_insn "altivec_stvx__1op_si" + [(set (mem:VM2 (and:SI (match_operand:SI 1 "register_operand" "r") + (const_int -16))) + (match_operand:VM2 0 "register_operand" "v"))] + "TARGET_ALTIVEC && TARGET_32BIT" + "stvx %0,0,%1" + [(set_attr "type" "vecstore")]) + (define_expand "altivec_stvxl_" [(parallel [(set (match_operand:VM2 0 "memory_operand" "=Z") Index: gcc/config/rs6000/rs6000-c.c =================================================================== --- gcc/config/rs6000/rs6000-c.c (revision 235090) +++ gcc/config/rs6000/rs6000-c.c (working copy) @@ -4800,6 +4800,164 @@ assignment for unaligned loads and stores"); return stmt; } + /* Expand vec_ld into an expression that masks the address and + performs the load. We need to expand this early to allow + the best aliasing, as by the time we get into RTL we no longer + are able to honor __restrict__, for example. We may want to + consider this for all memory access built-ins. + + When -maltivec=be is specified, simply punt to existing + built-in processing. */ + if (fcode == ALTIVEC_BUILTIN_VEC_LD + && (BYTES_BIG_ENDIAN || !VECTOR_ELT_ORDER_BIG)) + { + tree arg0 = (*arglist)[0]; + tree arg1 = (*arglist)[1]; + + /* Strip qualifiers like "const" from the pointer arg. */ + tree arg1_type = TREE_TYPE (arg1); + tree inner_type = TREE_TYPE (arg1_type); + if (TYPE_QUALS (TREE_TYPE (arg1_type)) != 0) + { + arg1_type = build_pointer_type (build_qualified_type (inner_type, + 0)); + arg1 = fold_convert (arg1_type, arg1); + } + + /* Construct the masked address. We have to jump through some hoops + here. If the first argument to a PLUS_EXPR is a pointer, + build_binary_op will multiply the offset by the size of the + inner type of the pointer (C semantics). With vec_ld and vec_st, + the offset must be left alone. However, if we convert to a + sizetype to do the arithmetic, we get a PLUS_EXPR instead of a + POINTER_PLUS_EXPR, which interferes with aliasing (causing us, + for example, to lose "restrict" information). Thus where legal, + we pre-adjust the offset knowing that a multiply by size is + coming. When the offset isn't a multiple of the size, we are + forced to do the arithmetic in size_type for correctness, at the + cost of losing aliasing information. This, however, should be + quite rare with these operations. */ + arg0 = fold (arg0); + + /* Let existing error handling take over if we don't have a constant + offset. */ + if (TREE_CODE (arg0) == INTEGER_CST) + { + HOST_WIDE_INT off = TREE_INT_CST_LOW (arg0); + HOST_WIDE_INT size = int_size_in_bytes (inner_type); + tree addr; + + if (off % size == 0) + { + tree adjoff = build_int_cst (TREE_TYPE (arg0), off / size); + addr = build_binary_op (loc, PLUS_EXPR, arg1, adjoff, 0); + addr = build1 (NOP_EXPR, sizetype, addr); + } + else + { + tree hack_arg1 = build1 (NOP_EXPR, sizetype, arg1); + addr = build_binary_op (loc, PLUS_EXPR, hack_arg1, arg0, 0); + } + tree aligned = build_binary_op (loc, BIT_AND_EXPR, addr, + build_int_cst (sizetype, -16), 0); + + /* Find the built-in to get the return type so we can convert + the result properly (or fall back to default handling if the + arguments aren't compatible). */ + for (desc = altivec_overloaded_builtins; + desc->code && desc->code != fcode; desc++) + continue; + + for (; desc->code == fcode; desc++) + if (rs6000_builtin_type_compatible (TREE_TYPE (arg0), desc->op1) + && (rs6000_builtin_type_compatible (TREE_TYPE (arg1), + desc->op2))) + { + tree ret_type = rs6000_builtin_type (desc->ret_type); + if (TYPE_MODE (ret_type) == V2DImode) + /* Type-based aliasing analysis thinks vector long + and vector long long are different and will put them + in distinct alias classes. Force our return type + to be a may-alias type to avoid this. */ + ret_type + = build_pointer_type_for_mode (ret_type, Pmode, + true/*can_alias_all*/); + else + ret_type = build_pointer_type (ret_type); + aligned = build1 (NOP_EXPR, ret_type, aligned); + tree ret_val = build_indirect_ref (loc, aligned, RO_NULL); + return ret_val; + } + } + } + + /* Similarly for stvx. */ + if (fcode == ALTIVEC_BUILTIN_VEC_ST + && (BYTES_BIG_ENDIAN || !VECTOR_ELT_ORDER_BIG)) + { + tree arg0 = (*arglist)[0]; + tree arg1 = (*arglist)[1]; + tree arg2 = (*arglist)[2]; + + /* Construct the masked address. See handling for ALTIVEC_BUILTIN_VEC_LD + for an explanation of address arithmetic concerns. */ + arg1 = fold (arg1); + + /* Let existing error handling take over if we don't have a constant + offset. */ + if (TREE_CODE (arg1) == INTEGER_CST) + { + HOST_WIDE_INT off = TREE_INT_CST_LOW (arg1); + tree inner_type = TREE_TYPE (TREE_TYPE (arg2)); + HOST_WIDE_INT size = int_size_in_bytes (inner_type); + tree addr; + + if (off % size == 0) + { + tree adjoff = build_int_cst (TREE_TYPE (arg1), off / size); + addr = build_binary_op (loc, PLUS_EXPR, arg2, adjoff, 0); + addr = build1 (NOP_EXPR, sizetype, addr); + } + else + { + tree hack_arg2 = build1 (NOP_EXPR, sizetype, arg2); + addr = build_binary_op (loc, PLUS_EXPR, hack_arg2, arg1, 0); + } + tree aligned = build_binary_op (loc, BIT_AND_EXPR, addr, + build_int_cst (sizetype, -16), 0); + + /* Find the built-in to make sure a compatible one exists; if not + we fall back to default handling to get the error message. */ + for (desc = altivec_overloaded_builtins; + desc->code && desc->code != fcode; desc++) + continue; + + for (; desc->code == fcode; desc++) + if (rs6000_builtin_type_compatible (TREE_TYPE (arg0), desc->op1) + && rs6000_builtin_type_compatible (TREE_TYPE (arg1), desc->op2) + && rs6000_builtin_type_compatible (TREE_TYPE (arg2), + desc->op3)) + { + tree arg0_type = TREE_TYPE (arg0); + if (TYPE_MODE (arg0_type) == V2DImode) + /* Type-based aliasing analysis thinks vector long + and vector long long are different and will put them + in distinct alias classes. Force our address type + to be a may-alias type to avoid this. */ + arg0_type + = build_pointer_type_for_mode (arg0_type, Pmode, + true/*can_alias_all*/); + else + arg0_type = build_pointer_type (arg0_type); + aligned = build1 (NOP_EXPR, arg0_type, aligned); + tree stg = build_indirect_ref (loc, aligned, RO_NULL); + tree retval = build2 (MODIFY_EXPR, TREE_TYPE (stg), stg, + convert (TREE_TYPE (stg), arg0)); + return retval; + } + } + } + for (n = 0; !VOID_TYPE_P (TREE_VALUE (fnargs)) && n < nargs; fnargs = TREE_CHAIN (fnargs), n++) Index: gcc/config/rs6000/rs6000.c =================================================================== --- gcc/config/rs6000/rs6000.c (revision 235090) +++ gcc/config/rs6000/rs6000.c (working copy) @@ -13025,9 +13025,9 @@ swap_selector_for_mode (machine_mode mode) return force_reg (V16QImode, gen_rtx_CONST_VECTOR (V16QImode, gen_rtvec_v (16, perm))); } -/* Generate code for an "lvx", "lvxl", or "lve*x" built-in for a little endian target - with -maltivec=be specified. Issue the load followed by an element-reversing - permute. */ +/* Generate code for an "lvxl", or "lve*x" built-in for a little endian target + with -maltivec=be specified. Issue the load followed by an element- + reversing permute. */ void altivec_expand_lvx_be (rtx op0, rtx op1, machine_mode mode, unsigned unspec) { @@ -13043,8 +13043,8 @@ altivec_expand_lvx_be (rtx op0, rtx op1, machine_m emit_insn (gen_rtx_SET (op0, vperm)); } -/* Generate code for a "stvx" or "stvxl" built-in for a little endian target - with -maltivec=be specified. Issue the store preceded by an element-reversing +/* Generate code for a "stvxl" built-in for a little endian target with + -maltivec=be specified. Issue the store preceded by an element-reversing permute. */ void altivec_expand_stvx_be (rtx op0, rtx op1, machine_mode mode, unsigned unspec) @@ -13106,22 +13106,65 @@ altivec_expand_lv_builtin (enum insn_code icode, t op1 = copy_to_mode_reg (mode1, op1); - if (op0 == const0_rtx) + /* For LVX, express the RTL accurately by ANDing the address with -16. + LVXL and LVE*X expand to use UNSPECs to hide their special behavior, + so the raw address is fine. */ + switch (icode) { - addr = gen_rtx_MEM (blk ? BLKmode : tmode, op1); - } - else - { - op0 = copy_to_mode_reg (mode0, op0); - addr = gen_rtx_MEM (blk ? BLKmode : tmode, gen_rtx_PLUS (Pmode, op0, op1)); - } + case CODE_FOR_altivec_lvx_v2df_2op: + case CODE_FOR_altivec_lvx_v2di_2op: + case CODE_FOR_altivec_lvx_v4sf_2op: + case CODE_FOR_altivec_lvx_v4si_2op: + case CODE_FOR_altivec_lvx_v8hi_2op: + case CODE_FOR_altivec_lvx_v16qi_2op: + { + rtx rawaddr; + if (op0 == const0_rtx) + rawaddr = op1; + else + { + op0 = copy_to_mode_reg (mode0, op0); + rawaddr = gen_rtx_PLUS (Pmode, op1, op0); + } + addr = gen_rtx_AND (Pmode, rawaddr, gen_rtx_CONST_INT (Pmode, -16)); + addr = gen_rtx_MEM (blk ? BLKmode : tmode, addr); - pat = GEN_FCN (icode) (target, addr); + /* For -maltivec=be, emit the load and follow it up with a + permute to swap the elements. */ + if (!BYTES_BIG_ENDIAN && VECTOR_ELT_ORDER_BIG) + { + rtx temp = gen_reg_rtx (tmode); + emit_insn (gen_rtx_SET (temp, addr)); - if (! pat) - return 0; - emit_insn (pat); + rtx sel = swap_selector_for_mode (tmode); + rtx vperm = gen_rtx_UNSPEC (tmode, gen_rtvec (3, temp, temp, sel), + UNSPEC_VPERM); + emit_insn (gen_rtx_SET (target, vperm)); + } + else + emit_insn (gen_rtx_SET (target, addr)); + break; + } + + default: + if (op0 == const0_rtx) + addr = gen_rtx_MEM (blk ? BLKmode : tmode, op1); + else + { + op0 = copy_to_mode_reg (mode0, op0); + addr = gen_rtx_MEM (blk ? BLKmode : tmode, + gen_rtx_PLUS (Pmode, op1, op0)); + } + + pat = GEN_FCN (icode) (target, addr); + if (! pat) + return 0; + emit_insn (pat); + + break; + } + return target; } @@ -13208,7 +13251,7 @@ altivec_expand_stv_builtin (enum insn_code icode, rtx op0 = expand_normal (arg0); rtx op1 = expand_normal (arg1); rtx op2 = expand_normal (arg2); - rtx pat, addr; + rtx pat, addr, rawaddr; machine_mode tmode = insn_data[icode].operand[0].mode; machine_mode smode = insn_data[icode].operand[1].mode; machine_mode mode1 = Pmode; @@ -13220,24 +13263,69 @@ altivec_expand_stv_builtin (enum insn_code icode, || arg2 == error_mark_node) return const0_rtx; - if (! (*insn_data[icode].operand[1].predicate) (op0, smode)) - op0 = copy_to_mode_reg (smode, op0); - op2 = copy_to_mode_reg (mode2, op2); - if (op1 == const0_rtx) + /* For STVX, express the RTL accurately by ANDing the address with -16. + STVXL and STVE*X expand to use UNSPECs to hide their special behavior, + so the raw address is fine. */ + switch (icode) { - addr = gen_rtx_MEM (tmode, op2); - } - else - { - op1 = copy_to_mode_reg (mode1, op1); - addr = gen_rtx_MEM (tmode, gen_rtx_PLUS (Pmode, op1, op2)); - } + case CODE_FOR_altivec_stvx_v2df_2op: + case CODE_FOR_altivec_stvx_v2di_2op: + case CODE_FOR_altivec_stvx_v4sf_2op: + case CODE_FOR_altivec_stvx_v4si_2op: + case CODE_FOR_altivec_stvx_v8hi_2op: + case CODE_FOR_altivec_stvx_v16qi_2op: + { + if (op1 == const0_rtx) + rawaddr = op2; + else + { + op1 = copy_to_mode_reg (mode1, op1); + rawaddr = gen_rtx_PLUS (Pmode, op2, op1); + } - pat = GEN_FCN (icode) (addr, op0); - if (pat) - emit_insn (pat); + addr = gen_rtx_AND (Pmode, rawaddr, gen_rtx_CONST_INT (Pmode, -16)); + addr = gen_rtx_MEM (tmode, addr); + + op0 = copy_to_mode_reg (tmode, op0); + + /* For -maltivec=be, emit a permute to swap the elements, followed + by the store. */ + if (!BYTES_BIG_ENDIAN && VECTOR_ELT_ORDER_BIG) + { + rtx temp = gen_reg_rtx (tmode); + rtx sel = swap_selector_for_mode (tmode); + rtx vperm = gen_rtx_UNSPEC (tmode, gen_rtvec (3, op0, op0, sel), + UNSPEC_VPERM); + emit_insn (gen_rtx_SET (temp, vperm)); + emit_insn (gen_rtx_SET (addr, temp)); + } + else + emit_insn (gen_rtx_SET (addr, op0)); + + break; + } + + default: + { + if (! (*insn_data[icode].operand[1].predicate) (op0, smode)) + op0 = copy_to_mode_reg (smode, op0); + + if (op1 == const0_rtx) + addr = gen_rtx_MEM (tmode, op2); + else + { + op1 = copy_to_mode_reg (mode1, op1); + addr = gen_rtx_MEM (tmode, gen_rtx_PLUS (Pmode, op2, op1)); + } + + pat = GEN_FCN (icode) (addr, op0); + if (pat) + emit_insn (pat); + } + } + return NULL_RTX; } @@ -14073,18 +14161,18 @@ altivec_expand_builtin (tree exp, rtx target, bool switch (fcode) { case ALTIVEC_BUILTIN_STVX_V2DF: - return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v2df, exp); + return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v2df_2op, exp); case ALTIVEC_BUILTIN_STVX_V2DI: - return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v2di, exp); + return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v2di_2op, exp); case ALTIVEC_BUILTIN_STVX_V4SF: - return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v4sf, exp); + return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v4sf_2op, exp); case ALTIVEC_BUILTIN_STVX: case ALTIVEC_BUILTIN_STVX_V4SI: - return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v4si, exp); + return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v4si_2op, exp); case ALTIVEC_BUILTIN_STVX_V8HI: - return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v8hi, exp); + return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v8hi_2op, exp); case ALTIVEC_BUILTIN_STVX_V16QI: - return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v16qi, exp); + return altivec_expand_stv_builtin (CODE_FOR_altivec_stvx_v16qi_2op, exp); case ALTIVEC_BUILTIN_STVEBX: return altivec_expand_stv_builtin (CODE_FOR_altivec_stvebx, exp); case ALTIVEC_BUILTIN_STVEHX: @@ -14272,23 +14360,23 @@ altivec_expand_builtin (tree exp, rtx target, bool return altivec_expand_lv_builtin (CODE_FOR_altivec_lvxl_v16qi, exp, target, false); case ALTIVEC_BUILTIN_LVX_V2DF: - return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v2df, + return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v2df_2op, exp, target, false); case ALTIVEC_BUILTIN_LVX_V2DI: - return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v2di, + return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v2di_2op, exp, target, false); case ALTIVEC_BUILTIN_LVX_V4SF: - return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v4sf, + return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v4sf_2op, exp, target, false); case ALTIVEC_BUILTIN_LVX: case ALTIVEC_BUILTIN_LVX_V4SI: - return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v4si, + return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v4si_2op, exp, target, false); case ALTIVEC_BUILTIN_LVX_V8HI: - return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v8hi, + return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v8hi_2op, exp, target, false); case ALTIVEC_BUILTIN_LVX_V16QI: - return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v16qi, + return altivec_expand_lv_builtin (CODE_FOR_altivec_lvx_v16qi_2op, exp, target, false); case ALTIVEC_BUILTIN_LVLX: return altivec_expand_lv_builtin (CODE_FOR_altivec_lvlx, @@ -37139,7 +37227,9 @@ insn_is_swappable_p (swap_web_entry *insn_entry, r fix them up by converting them to permuting ones. Exceptions: UNSPEC_LVE, UNSPEC_LVX, and UNSPEC_STVX, which have a PARALLEL body instead of a SET; and UNSPEC_STVE, which has an UNSPEC - for the SET source. */ + for the SET source. Also we must now make an exception for lvx + and stvx when they are not in the UNSPEC_LVX/STVX form (with the + explicit "& -16") since this leads to unrecognizable insns. */ rtx body = PATTERN (insn); int i = INSN_UID (insn); @@ -37147,6 +37237,11 @@ insn_is_swappable_p (swap_web_entry *insn_entry, r { if (GET_CODE (body) == SET) { + rtx rhs = SET_SRC (body); + gcc_assert (GET_CODE (rhs) == MEM); + if (GET_CODE (XEXP (rhs, 0)) == AND) + return 0; + *special = SH_NOSWAP_LD; return 1; } @@ -37156,8 +37251,14 @@ insn_is_swappable_p (swap_web_entry *insn_entry, r if (insn_entry[i].is_store) { - if (GET_CODE (body) == SET && GET_CODE (SET_SRC (body)) != UNSPEC) + if (GET_CODE (body) == SET + && GET_CODE (SET_SRC (body)) != UNSPEC) { + rtx lhs = SET_DEST (body); + gcc_assert (GET_CODE (lhs) == MEM); + if (GET_CODE (XEXP (lhs, 0)) == AND) + return 0; + *special = SH_NOSWAP_ST; return 1; } @@ -37827,6 +37928,267 @@ dump_swap_insn_table (swap_web_entry *insn_entry) fputs ("\n", dump_file); } +/* Return RTX with its address canonicalized to (reg) or (+ reg reg). + Here RTX is an (& addr (const_int -16)). Always return a new copy + to avoid problems with combine. */ +static rtx +alignment_with_canonical_addr (rtx align) +{ + rtx canon; + rtx addr = XEXP (align, 0); + + if (REG_P (addr)) + canon = addr; + + else if (GET_CODE (addr) == PLUS) + { + rtx addrop0 = XEXP (addr, 0); + rtx addrop1 = XEXP (addr, 1); + + if (!REG_P (addrop0)) + addrop0 = force_reg (GET_MODE (addrop0), addrop0); + + if (!REG_P (addrop1)) + addrop1 = force_reg (GET_MODE (addrop1), addrop1); + + canon = gen_rtx_PLUS (GET_MODE (addr), addrop0, addrop1); + } + + else + canon = force_reg (GET_MODE (addr), addr); + + return gen_rtx_AND (GET_MODE (align), canon, GEN_INT (-16)); +} + +/* Check whether an rtx is an alignment mask, and if so, return + a fully-expanded rtx for the masking operation. */ +static rtx +alignment_mask (rtx_insn *insn) +{ + rtx body = PATTERN (insn); + + if (GET_CODE (body) != SET + || GET_CODE (SET_SRC (body)) != AND + || !REG_P (XEXP (SET_SRC (body), 0))) + return 0; + + rtx mask = XEXP (SET_SRC (body), 1); + + if (GET_CODE (mask) == CONST_INT) + { + if (INTVAL (mask) == -16) + return alignment_with_canonical_addr (SET_SRC (body)); + else + return 0; + } + + if (!REG_P (mask)) + return 0; + + struct df_insn_info *insn_info = DF_INSN_INFO_GET (insn); + df_ref use; + rtx real_mask = 0; + + FOR_EACH_INSN_INFO_USE (use, insn_info) + { + if (!rtx_equal_p (DF_REF_REG (use), mask)) + continue; + + struct df_link *def_link = DF_REF_CHAIN (use); + if (!def_link || def_link->next) + return 0; + + rtx_insn *const_insn = DF_REF_INSN (def_link->ref); + rtx const_body = PATTERN (const_insn); + if (GET_CODE (const_body) != SET) + return 0; + + real_mask = SET_SRC (const_body); + + if (GET_CODE (real_mask) != CONST_INT + || INTVAL (real_mask) != -16) + return 0; + } + + if (real_mask == 0) + return 0; + + return alignment_with_canonical_addr (SET_SRC (body)); +} + +/* Given INSN that's a load or store based at BASE_REG, look for a + feeding computation that aligns its address on a 16-byte boundary. */ +static rtx +find_alignment_op (rtx_insn *insn, rtx base_reg) +{ + df_ref base_use; + struct df_insn_info *insn_info = DF_INSN_INFO_GET (insn); + rtx and_operation = 0; + + FOR_EACH_INSN_INFO_USE (base_use, insn_info) + { + if (!rtx_equal_p (DF_REF_REG (base_use), base_reg)) + continue; + + struct df_link *base_def_link = DF_REF_CHAIN (base_use); + if (!base_def_link || base_def_link->next) + break; + + rtx_insn *and_insn = DF_REF_INSN (base_def_link->ref); + and_operation = alignment_mask (and_insn); + if (and_operation != 0) + break; + } + + return and_operation; +} + +struct del_info { bool replace; rtx_insn *replace_insn; }; + +/* If INSN is the load for an lvx pattern, put it in canonical form. */ +static void +combine_lvx_pattern (rtx_insn *insn, del_info *to_delete) +{ + rtx body = PATTERN (insn); + gcc_assert (GET_CODE (body) == SET + && GET_CODE (SET_SRC (body)) == VEC_SELECT + && GET_CODE (XEXP (SET_SRC (body), 0)) == MEM); + + rtx mem = XEXP (SET_SRC (body), 0); + rtx base_reg = XEXP (mem, 0); + + rtx and_operation = find_alignment_op (insn, base_reg); + + if (and_operation != 0) + { + df_ref def; + struct df_insn_info *insn_info = DF_INSN_INFO_GET (insn); + FOR_EACH_INSN_INFO_DEF (def, insn_info) + { + struct df_link *link = DF_REF_CHAIN (def); + if (!link || link->next) + break; + + rtx_insn *swap_insn = DF_REF_INSN (link->ref); + if (!insn_is_swap_p (swap_insn) + || insn_is_load_p (swap_insn) + || insn_is_store_p (swap_insn)) + break; + + /* Expected lvx pattern found. Change the swap to + a copy, and propagate the AND operation into the + load. */ + to_delete[INSN_UID (swap_insn)].replace = true; + to_delete[INSN_UID (swap_insn)].replace_insn = swap_insn; + + XEXP (mem, 0) = and_operation; + SET_SRC (body) = mem; + INSN_CODE (insn) = -1; /* Force re-recognition. */ + df_insn_rescan (insn); + + if (dump_file) + fprintf (dump_file, "lvx opportunity found at %d\n", + INSN_UID (insn)); + } + } +} + +/* If INSN is the store for an stvx pattern, put it in canonical form. */ +static void +combine_stvx_pattern (rtx_insn *insn, del_info *to_delete) +{ + rtx body = PATTERN (insn); + gcc_assert (GET_CODE (body) == SET + && GET_CODE (SET_DEST (body)) == MEM + && GET_CODE (SET_SRC (body)) == VEC_SELECT); + rtx mem = SET_DEST (body); + rtx base_reg = XEXP (mem, 0); + + rtx and_operation = find_alignment_op (insn, base_reg); + + if (and_operation != 0) + { + rtx src_reg = XEXP (SET_SRC (body), 0); + df_ref src_use; + struct df_insn_info *insn_info = DF_INSN_INFO_GET (insn); + FOR_EACH_INSN_INFO_USE (src_use, insn_info) + { + if (!rtx_equal_p (DF_REF_REG (src_use), src_reg)) + continue; + + struct df_link *link = DF_REF_CHAIN (src_use); + if (!link || link->next) + break; + + rtx_insn *swap_insn = DF_REF_INSN (link->ref); + if (!insn_is_swap_p (swap_insn) + || insn_is_load_p (swap_insn) + || insn_is_store_p (swap_insn)) + break; + + /* Expected stvx pattern found. Change the swap to + a copy, and propagate the AND operation into the + store. */ + to_delete[INSN_UID (swap_insn)].replace = true; + to_delete[INSN_UID (swap_insn)].replace_insn = swap_insn; + + XEXP (mem, 0) = and_operation; + SET_SRC (body) = src_reg; + INSN_CODE (insn) = -1; /* Force re-recognition. */ + df_insn_rescan (insn); + + if (dump_file) + fprintf (dump_file, "stvx opportunity found at %d\n", + INSN_UID (insn)); + } + } +} + +/* Look for patterns created from builtin lvx and stvx calls, and + canonicalize them to be properly recognized as such. */ +static void +combine_lvx_stvx_patterns (function *fun) +{ + int i; + basic_block bb; + rtx_insn *insn; + + int num_insns = get_max_uid (); + del_info *to_delete = XCNEWVEC (del_info, num_insns); + + FOR_ALL_BB_FN (bb, fun) + FOR_BB_INSNS (bb, insn) + { + if (!NONDEBUG_INSN_P (insn)) + continue; + + if (insn_is_load_p (insn) && insn_is_swap_p (insn)) + combine_lvx_pattern (insn, to_delete); + else if (insn_is_store_p (insn) && insn_is_swap_p (insn)) + combine_stvx_pattern (insn, to_delete); + } + + /* Turning swaps into copies is delayed until now, to avoid problems + with deleting instructions during the insn walk. */ + for (i = 0; i < num_insns; i++) + if (to_delete[i].replace) + { + rtx swap_body = PATTERN (to_delete[i].replace_insn); + rtx src_reg = XEXP (SET_SRC (swap_body), 0); + rtx copy = gen_rtx_SET (SET_DEST (swap_body), src_reg); + rtx_insn *new_insn = emit_insn_before (copy, + to_delete[i].replace_insn); + set_block_for_insn (new_insn, + BLOCK_FOR_INSN (to_delete[i].replace_insn)); + df_insn_rescan (new_insn); + df_insn_delete (to_delete[i].replace_insn); + remove_insn (to_delete[i].replace_insn); + to_delete[i].replace_insn->set_deleted (); + } + + free (to_delete); +} + /* Main entry point for this pass. */ unsigned int rs6000_analyze_swaps (function *fun) @@ -37833,7 +38195,7 @@ rs6000_analyze_swaps (function *fun) { swap_web_entry *insn_entry; basic_block bb; - rtx_insn *insn; + rtx_insn *insn, *curr_insn = 0; /* Dataflow analysis for use-def chains. */ df_set_flags (DF_RD_PRUNE_DEAD_DEFS); @@ -37841,12 +38203,15 @@ rs6000_analyze_swaps (function *fun) df_analyze (); df_set_flags (DF_DEFER_INSN_RESCAN); + /* Pre-pass to combine lvx and stvx patterns so we don't lose info. */ + combine_lvx_stvx_patterns (fun); + /* Allocate structure to represent webs of insns. */ insn_entry = XCNEWVEC (swap_web_entry, get_max_uid ()); /* Walk the insns to gather basic data. */ FOR_ALL_BB_FN (bb, fun) - FOR_BB_INSNS (bb, insn) + FOR_BB_INSNS_SAFE (bb, insn, curr_insn) { unsigned int uid = INSN_UID (insn); if (NONDEBUG_INSN_P (insn)) Index: gcc/config/rs6000/vector.md =================================================================== --- gcc/config/rs6000/vector.md (revision 235090) +++ gcc/config/rs6000/vector.md (working copy) @@ -167,7 +167,14 @@ if (VECTOR_MEM_VSX_P (mode)) { operands[1] = rs6000_address_for_altivec (operands[1]); - emit_insn (gen_altivec_lvx_ (operands[0], operands[1])); + rtx and_op = XEXP (operands[1], 0); + gcc_assert (GET_CODE (and_op) == AND); + rtx addr = XEXP (and_op, 0); + if (GET_CODE (addr) == PLUS) + emit_insn (gen_altivec_lvx__2op (operands[0], XEXP (addr, 0), + XEXP (addr, 1))); + else + emit_insn (gen_altivec_lvx__1op (operands[0], operands[1])); DONE; } }") @@ -183,7 +190,14 @@ if (VECTOR_MEM_VSX_P (mode)) { operands[0] = rs6000_address_for_altivec (operands[0]); - emit_insn (gen_altivec_stvx_ (operands[0], operands[1])); + rtx and_op = XEXP (operands[0], 0); + gcc_assert (GET_CODE (and_op) == AND); + rtx addr = XEXP (and_op, 0); + if (GET_CODE (addr) == PLUS) + emit_insn (gen_altivec_stvx__2op (operands[1], XEXP (addr, 0), + XEXP (addr, 1))); + else + emit_insn (gen_altivec_stvx__1op (operands[1], operands[0])); DONE; } }")