From patchwork Mon Nov 1 08:16:34 2010 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Rosen X-Patchwork-Id: 69752 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) by ozlabs.org (Postfix) with SMTP id 0F2101007D2 for ; Mon, 1 Nov 2010 19:16:51 +1100 (EST) Received: (qmail 7815 invoked by alias); 1 Nov 2010 08:16:49 -0000 Received: (qmail 7802 invoked by uid 22791); 1 Nov 2010 08:16:45 -0000 X-SWARE-Spam-Status: No, hits=-0.9 required=5.0 tests=AWL, BAYES_05, TW_TM, T_RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from mtagate2.de.ibm.com (HELO mtagate2.de.ibm.com) (195.212.17.162) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Mon, 01 Nov 2010 08:16:37 +0000 Received: from d12nrmr1707.megacenter.de.ibm.com (d12nrmr1707.megacenter.de.ibm.com [9.149.167.81]) by mtagate2.de.ibm.com (8.13.1/8.13.1) with ESMTP id oA18GYB0004611 for ; Mon, 1 Nov 2010 08:16:34 GMT Received: from d12av05.megacenter.de.ibm.com (d12av05.megacenter.de.ibm.com [9.149.165.216]) by d12nrmr1707.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id oA18GYUc3166278 for ; Mon, 1 Nov 2010 09:16:34 +0100 Received: from d12av05.megacenter.de.ibm.com (loopback [127.0.0.1]) by d12av05.megacenter.de.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id oA18GXaC015969 for ; Mon, 1 Nov 2010 09:16:33 +0100 Received: from d12mc102.megacenter.de.ibm.com (d12nrml1506.megacenter.de.ibm.com [9.149.165.56] (may be forged)) by d12av05.megacenter.de.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id oA18GXeZ015966 for ; Mon, 1 Nov 2010 09:16:33 +0100 Subject: [patch] Fix PR tree-optimization/45902 on 4.5 X-KeepSent: 76F885DF:6A88C4F5-C22577CB:006A94FF; type=4; name=$KeepSent To: gcc-patches@gcc.gnu.org Message-ID: From: Ira Rosen Date: Mon, 1 Nov 2010 10:16:34 +0200 MIME-Version: 1.0 Content-type: multipart/mixed; Boundary="0__=4DBBFD58DFF9126F8f9e8a93df938690918c4DBBFD58DFF9126F" Content-Disposition: inline X-IsSubscribed: yes Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Hi, This patch fixes pr45902 on 4.5 similar to http://gcc.gnu.org/ml/gcc-patches/2010-10/msg01820.html. Bootstrapped and tested on powerpc64-suse-linux. OK for 4.5? Thanks, Ira ChangeLog: PR tree-optimization/45902 * tree-vectorizer.h (enum stmt_vec_info_type): Add new value for shift. (vect_get_slp_defs): Add arguments. * tree-vect-stmts.c (cost_for_stmt): Handle shifts. (vect_get_vec_def_for_operand): Use operand's type to determine number of units in the created vector. (vect_get_vec_defs): Pass scalar operands to vect_get_slp_defs. (vectorizable_shift): New function. (vectorizable_operation): Move code that handles shifts to vectorizable_shift. (vectorizable_type_demotion): Pass scalar operands to vect_get_slp_defs. (vectorizable_type_promotion, vectorizable_store): Likewise. (vect_analyze_stmt): Call vectorizable_shift. (vect_transform_stmt): Likewise. * tree-vect-slp.c (vect_get_constant_vectors): Add new argument. Use it as the operand to create vectors except for stores. (vect_get_slp_defs): Add new arguments. Pass them to vect_get_constant_vectors. testsuite/ChangeLog: PR tree-optimization/45902 * gcc.dg/vect/pr45902.c: New. (See attached file: patch.txt) Index: testsuite/gcc.dg/vect/pr45902.c =================================================================== --- testsuite/gcc.dg/vect/pr45902.c (revision 0) +++ testsuite/gcc.dg/vect/pr45902.c (revision 0) @@ -0,0 +1,43 @@ +/* { dg-require-effective-target vect_int } */ + +#include +#include +#include "tree-vect.h" + +#define N 128 + +short res[N]; +short a[N]; + +int +main1 () +{ + int i; + + for (i = 0; i < N/4; i+=4) + { + res[i] = a[i] >> 8; + res[i+1] = a[i+1] >> 8; + res[i+2] = a[i+2] >> 8; + res[i+3] = a[i+3] >> 8; + } +} + +int +main () +{ + int i; + + for (i = 0; i < N; i++) + a[i] = i; + + main1 (); + + for (i = 0; i < N; i++) + if (res[i] != a[i] >> 8) + abort (); + + return 0; +} + +/* { dg-final { cleanup-tree-dump "vect" } } */ Index: tree-vectorizer.h =================================================================== --- tree-vectorizer.h (revision 166065) +++ tree-vectorizer.h (working copy) @@ -329,6 +329,7 @@ enum stmt_vec_info_type { undef_vec_info_type = 0, load_vec_info_type, store_vec_info_type, + shift_vec_info_type, op_vec_info_type, call_vec_info_type, assignment_vec_info_type, @@ -854,7 +855,7 @@ extern void vect_update_slp_costs_according_to_vf extern bool vect_analyze_slp (loop_vec_info, bb_vec_info); extern void vect_make_slp_decision (loop_vec_info); extern void vect_detect_hybrid_slp (loop_vec_info); -extern void vect_get_slp_defs (slp_tree, VEC (tree,heap) **, +extern void vect_get_slp_defs (tree, tree, slp_tree, VEC (tree,heap) **, VEC (tree,heap) **); extern LOC find_bb_location (basic_block); extern bb_vec_info vect_slp_analyze_bb (basic_block); Index: tree-vect-stmts.c =================================================================== --- tree-vect-stmts.c (revision 166065) +++ tree-vect-stmts.c (working copy) @@ -555,6 +555,7 @@ cost_for_stmt (gimple stmt) return TARG_SCALAR_LOAD_COST; case store_vec_info_type: return TARG_SCALAR_STORE_COST; + case shift_vec_info_type: case op_vec_info_type: case condition_vec_info_type: case assignment_vec_info_type: @@ -889,8 +890,7 @@ vect_get_vec_def_for_operand (tree op, gimple stmt gimple def_stmt; stmt_vec_info def_stmt_info = NULL; stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt); - tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo); - unsigned int nunits = TYPE_VECTOR_SUBPARTS (vectype); + unsigned int nunits; loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo); tree vec_inv; tree vec_cst; @@ -931,6 +931,7 @@ vect_get_vec_def_for_operand (tree op, gimple stmt { vector_type = get_vectype_for_scalar_type (TREE_TYPE (op)); gcc_assert (vector_type); + nunits = TYPE_VECTOR_SUBPARTS (vector_type); if (scalar_def) *scalar_def = op; @@ -1134,7 +1135,7 @@ vect_get_vec_defs (tree op0, tree op1, gimple stmt slp_tree slp_node) { if (slp_node) - vect_get_slp_defs (slp_node, vec_oprnds0, vec_oprnds1); + vect_get_slp_defs (op0, op1, slp_node, vec_oprnds0, vec_oprnds1); else { tree vec_oprnd; @@ -1922,6 +1923,296 @@ vectorizable_assignment (gimple stmt, gimple_stmt_ return true; } + +/* Function vectorizable_shift. + + Check if STMT performs a shift operation that can be vectorized. + If VEC_STMT is also passed, vectorize the STMT: create a vectorized + stmt to replace it, put it in VEC_STMT, and insert it at BSI. + Return FALSE if not a vectorizable STMT, TRUE otherwise. */ + +static bool +vectorizable_shift (gimple stmt, gimple_stmt_iterator *gsi, + gimple *vec_stmt, slp_tree slp_node) +{ + tree vec_dest; + tree scalar_dest; + tree op0, op1 = NULL; + tree vec_oprnd1 = NULL_TREE; + stmt_vec_info stmt_info = vinfo_for_stmt (stmt); + tree vectype = STMT_VINFO_VECTYPE (stmt_info); + loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info); + enum tree_code code; + enum machine_mode vec_mode; + tree new_temp; + int op_type; + optab optab; + int icode; + enum machine_mode optab_op2_mode; + tree def; + gimple def_stmt; + enum vect_def_type dt[2] = {vect_unknown_def_type, vect_unknown_def_type}; + gimple new_stmt = NULL; + stmt_vec_info prev_stmt_info; + int nunits_in = TYPE_VECTOR_SUBPARTS (vectype); + int nunits_out; + tree vectype_out; + int ncopies; + int j, i; + VEC (tree, heap) *vec_oprnds0 = NULL, *vec_oprnds1 = NULL; + tree vop0, vop1; + unsigned int k; + bool scalar_shift_arg = false; + bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info); + int vf; + + if (loop_vinfo) + vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo); + else + vf = 1; + + /* Multiple types in SLP are handled by creating the appropriate number of + vectorized stmts for each SLP node. Hence, NCOPIES is always 1 in + case of SLP. */ + if (slp_node) + ncopies = 1; + else + ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits_in; + + gcc_assert (ncopies >= 1); + + if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo) + return false; + + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def) + return false; + + /* Is STMT a vectorizable shift? */ + if (!is_gimple_assign (stmt)) + return false; + + if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME) + return false; + + scalar_dest = gimple_assign_lhs (stmt); + vectype_out = get_vectype_for_scalar_type (TREE_TYPE (scalar_dest)); + if (!vectype_out) + return false; + nunits_out = TYPE_VECTOR_SUBPARTS (vectype_out); + if (nunits_out != nunits_in) + return false; + + code = gimple_assign_rhs_code (stmt); + + if (!(code == LSHIFT_EXPR || code == RSHIFT_EXPR || code == LROTATE_EXPR + || code == RROTATE_EXPR)) + return false; + + op_type = TREE_CODE_LENGTH (code); + op0 = gimple_assign_rhs1 (stmt); + if (!vect_is_simple_use (op0, loop_vinfo, bb_vinfo, &def_stmt, &def, &dt[0])) + { + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "use not simple."); + return false; + } + + op1 = gimple_assign_rhs2 (stmt); + if (!vect_is_simple_use (op1, loop_vinfo, bb_vinfo, &def_stmt, &def, &dt[1])) + { + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "use not simple."); + return false; + } + + /* Determine whether the shift amount is a vector, or scalar. If the + shift/rotate amount is a vector, use the vector/vector shift optabs. */ + /* Vector shifted by vector. */ + if (dt[1] == vect_internal_def) + { + optab = optab_for_tree_code (code, vectype, optab_vector); + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "vector/vector shift/rotate found."); + } + + /* See if the machine has a vector shifted by scalar insn and if not + then see if it has a vector shifted by vector insn. */ + else if (dt[1] == vect_constant_def || dt[1] == vect_external_def) + { + optab = optab_for_tree_code (code, vectype, optab_scalar); + if (optab + && (optab_handler (optab, TYPE_MODE (vectype))->insn_code + != CODE_FOR_nothing)) + { + scalar_shift_arg = true; + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "vector/scalar shift/rotate found."); + } + else + { + optab = optab_for_tree_code (code, vectype, optab_vector); + if (optab + && (optab_handler (optab, TYPE_MODE (vectype))->insn_code + != CODE_FOR_nothing)) + { + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "vector/vector shift/rotate found."); + + /* Unlike the other binary operators, shifts/rotates have + the rhs being int, instead of the same type as the lhs, + so make sure the scalar is the right type if we are + dealing with vectors of short/char. */ + if (dt[1] == vect_constant_def) + op1 = fold_convert (TREE_TYPE (vectype), op1); + } + } + } + + else + { + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "operand mode requires invariant argument."); + return false; + } + + /* Supportable by target? */ + if (!optab) + { + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "no optab."); + return false; + } + vec_mode = TYPE_MODE (vectype); + icode = (int) optab_handler (optab, vec_mode)->insn_code; + if (icode == CODE_FOR_nothing) + { + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "op not supported by target."); + /* Check only during analysis. */ + if (GET_MODE_SIZE (vec_mode) != UNITS_PER_WORD + || (vf < vect_min_worthwhile_factor (code) + && !vec_stmt)) + return false; + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "proceeding using word mode."); + } + + /* Worthwhile without SIMD support? Check only during analysis. */ + if (!VECTOR_MODE_P (TYPE_MODE (vectype)) + && vf < vect_min_worthwhile_factor (code) + && !vec_stmt) + { + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "not worthwhile without SIMD support."); + return false; + } + + if (!vec_stmt) /* transformation not required. */ + { + STMT_VINFO_TYPE (stmt_info) = shift_vec_info_type; + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "=== vectorizable_shift ==="); + vect_model_simple_cost (stmt_info, ncopies, dt, NULL); + return true; + } + + /** Transform. **/ + + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "transform shift."); + + /* Handle def. */ + vec_dest = vect_create_destination_var (scalar_dest, vectype); + + /* Allocate VECs for vector operands. In case of SLP, vector operands are + created in the previous stages of the recursion, so no allocation is + needed, except for the case of shift with scalar shift argument. In that + case we store the scalar operand in VEC_OPRNDS1 for every vector stmt to + be created to vectorize the SLP group, i.e., SLP_NODE->VEC_STMTS_SIZE. + In case of loop-based vectorization we allocate VECs of size 1. We + allocate VEC_OPRNDS1 only in case of binary operation. */ + if (!slp_node) + { + vec_oprnds0 = VEC_alloc (tree, heap, 1); + vec_oprnds1 = VEC_alloc (tree, heap, 1); + } + else if (scalar_shift_arg) + vec_oprnds1 = VEC_alloc (tree, heap, slp_node->vec_stmts_size); + + prev_stmt_info = NULL; + for (j = 0; j < ncopies; j++) + { + /* Handle uses. */ + if (j == 0) + { + if (scalar_shift_arg) + { + /* Vector shl and shr insn patterns can be defined with scalar + operand 2 (shift operand). In this case, use constant or loop + invariant op1 directly, without extending it to vector mode + first. */ + optab_op2_mode = insn_data[icode].operand[2].mode; + if (!VECTOR_MODE_P (optab_op2_mode)) + { + if (vect_print_dump_info (REPORT_DETAILS)) + fprintf (vect_dump, "operand 1 using scalar mode."); + vec_oprnd1 = op1; + VEC_quick_push (tree, vec_oprnds1, vec_oprnd1); + if (slp_node) + { + /* Store vec_oprnd1 for every vector stmt to be created + for SLP_NODE. We check during the analysis that all + the shift arguments are the same. + TODO: Allow different constants for different vector + stmts generated for an SLP instance. */ + for (k = 0; k < slp_node->vec_stmts_size - 1; k++) + VEC_quick_push (tree, vec_oprnds1, vec_oprnd1); + } + } + } + + /* vec_oprnd1 is available if operand 1 should be of a scalar-type + (a special case for certain kind of vector shifts); otherwise, + operand 1 should be of a vector type (the usual case). */ + if (vec_oprnd1) + vect_get_vec_defs (op0, NULL_TREE, stmt, &vec_oprnds0, NULL, + slp_node); + else + vect_get_vec_defs (op0, op1, stmt, &vec_oprnds0, &vec_oprnds1, + slp_node); + } + else + vect_get_vec_defs_for_stmt_copy (dt, &vec_oprnds0, &vec_oprnds1); + + /* Arguments are ready. Create the new vector stmt. */ + for (i = 0; VEC_iterate (tree, vec_oprnds0, i, vop0); i++) + { + vop1 = VEC_index (tree, vec_oprnds1, i); + new_stmt = gimple_build_assign_with_ops (code, vec_dest, vop0, vop1); + new_temp = make_ssa_name (vec_dest, new_stmt); + gimple_assign_set_lhs (new_stmt, new_temp); + vect_finish_stmt_generation (stmt, new_stmt, gsi); + if (slp_node) + VEC_quick_push (gimple, SLP_TREE_VEC_STMTS (slp_node), new_stmt); + } + + if (slp_node) + continue; + + if (j == 0) + STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; + else + STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; + prev_stmt_info = vinfo_for_stmt (new_stmt); + } + + VEC_free (tree, heap, vec_oprnds0); + VEC_free (tree, heap, vec_oprnds1); + + return true; +} + + /* Function vectorizable_operation. Check if STMT performs a binary or unary operation that can be vectorized. @@ -1936,7 +2227,6 @@ vectorizable_operation (gimple stmt, gimple_stmt_i tree vec_dest; tree scalar_dest; tree op0, op1 = NULL; - tree vec_oprnd1 = NULL_TREE; stmt_vec_info stmt_info = vinfo_for_stmt (stmt); tree vectype = STMT_VINFO_VECTYPE (stmt_info); loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info); @@ -1946,7 +2236,6 @@ vectorizable_operation (gimple stmt, gimple_stmt_i int op_type; optab optab; int icode; - enum machine_mode optab_op2_mode; tree def; gimple def_stmt; enum vect_def_type dt[2] = {vect_unknown_def_type, vect_unknown_def_type}; @@ -1959,8 +2248,6 @@ vectorizable_operation (gimple stmt, gimple_stmt_i int j, i; VEC(tree,heap) *vec_oprnds0 = NULL, *vec_oprnds1 = NULL; tree vop0, vop1; - unsigned int k; - bool scalar_shift_arg = false; bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info); int vf; @@ -2036,63 +2323,13 @@ vectorizable_operation (gimple stmt, gimple_stmt_i } } - /* If this is a shift/rotate, determine whether the shift amount is a vector, - or scalar. If the shift/rotate amount is a vector, use the vector/vector - shift optabs. */ + /* Shifts are handled in vectorizable_shift (). */ if (code == LSHIFT_EXPR || code == RSHIFT_EXPR || code == LROTATE_EXPR || code == RROTATE_EXPR) - { - /* vector shifted by vector */ - if (dt[1] == vect_internal_def) - { - optab = optab_for_tree_code (code, vectype, optab_vector); - if (vect_print_dump_info (REPORT_DETAILS)) - fprintf (vect_dump, "vector/vector shift/rotate found."); - } + return false; - /* See if the machine has a vector shifted by scalar insn and if not - then see if it has a vector shifted by vector insn */ - else if (dt[1] == vect_constant_def || dt[1] == vect_external_def) - { - optab = optab_for_tree_code (code, vectype, optab_scalar); - if (optab - && (optab_handler (optab, TYPE_MODE (vectype))->insn_code - != CODE_FOR_nothing)) - { - scalar_shift_arg = true; - if (vect_print_dump_info (REPORT_DETAILS)) - fprintf (vect_dump, "vector/scalar shift/rotate found."); - } - else - { - optab = optab_for_tree_code (code, vectype, optab_vector); - if (optab - && (optab_handler (optab, TYPE_MODE (vectype))->insn_code - != CODE_FOR_nothing)) - { - if (vect_print_dump_info (REPORT_DETAILS)) - fprintf (vect_dump, "vector/vector shift/rotate found."); + optab = optab_for_tree_code (code, vectype, optab_default); - /* Unlike the other binary operators, shifts/rotates have - the rhs being int, instead of the same type as the lhs, - so make sure the scalar is the right type if we are - dealing with vectors of short/char. */ - if (dt[1] == vect_constant_def) - op1 = fold_convert (TREE_TYPE (vectype), op1); - } - } - } - - else - { - if (vect_print_dump_info (REPORT_DETAILS)) - fprintf (vect_dump, "operand mode requires invariant argument."); - return false; - } - } - else - optab = optab_for_tree_code (code, vectype, optab_default); - /* Supportable by target? */ if (!optab) { @@ -2155,8 +2392,6 @@ vectorizable_operation (gimple stmt, gimple_stmt_i if (op_type == binary_op) vec_oprnds1 = VEC_alloc (tree, heap, 1); } - else if (scalar_shift_arg) - vec_oprnds1 = VEC_alloc (tree, heap, slp_node->vec_stmts_size); /* In case the vectorization factor (VF) is bigger than the number of elements that we can fit in a vectype (nunits), we have to generate @@ -2217,36 +2452,7 @@ vectorizable_operation (gimple stmt, gimple_stmt_i /* Handle uses. */ if (j == 0) { - if (op_type == binary_op && scalar_shift_arg) - { - /* Vector shl and shr insn patterns can be defined with scalar - operand 2 (shift operand). In this case, use constant or loop - invariant op1 directly, without extending it to vector mode - first. */ - optab_op2_mode = insn_data[icode].operand[2].mode; - if (!VECTOR_MODE_P (optab_op2_mode)) - { - if (vect_print_dump_info (REPORT_DETAILS)) - fprintf (vect_dump, "operand 1 using scalar mode."); - vec_oprnd1 = op1; - VEC_quick_push (tree, vec_oprnds1, vec_oprnd1); - if (slp_node) - { - /* Store vec_oprnd1 for every vector stmt to be created - for SLP_NODE. We check during the analysis that all the - shift arguments are the same. - TODO: Allow different constants for different vector - stmts generated for an SLP instance. */ - for (k = 0; k < slp_node->vec_stmts_size - 1; k++) - VEC_quick_push (tree, vec_oprnds1, vec_oprnd1); - } - } - } - - /* vec_oprnd1 is available if operand 1 should be of a scalar-type - (a special case for certain kind of vector shifts); otherwise, - operand 1 should be of a vector type (the usual case). */ - if (op_type == binary_op && !vec_oprnd1) + if (op_type == binary_op) vect_get_vec_defs (op0, op1, stmt, &vec_oprnds0, &vec_oprnds1, slp_node); else @@ -2536,7 +2742,7 @@ vectorizable_type_demotion (gimple stmt, gimple_st { /* Handle uses. */ if (slp_node) - vect_get_slp_defs (slp_node, &vec_oprnds0, NULL); + vect_get_slp_defs (op0, NULL_TREE, slp_node, &vec_oprnds0, NULL); else { VEC_free (tree, heap, vec_oprnds0); @@ -2837,7 +3043,8 @@ vectorizable_type_promotion (gimple stmt, gimple_s if (j == 0) { if (slp_node) - vect_get_slp_defs (slp_node, &vec_oprnds0, &vec_oprnds1); + vect_get_slp_defs (op0, op1, slp_node, &vec_oprnds0, + &vec_oprnds1); else { vec_oprnd0 = vect_get_vec_def_for_operand (op0, stmt, NULL); @@ -3123,7 +3330,8 @@ vectorizable_store (gimple stmt, gimple_stmt_itera if (slp) { /* Get vectorized arguments for SLP_NODE. */ - vect_get_slp_defs (slp_node, &vec_oprnds, NULL); + vect_get_slp_defs (NULL_TREE, NULL_TREE, slp_node, &vec_oprnds, + NULL); vec_oprnd = VEC_index (tree, vec_oprnds, 0); } @@ -4063,6 +4271,7 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vect ok = (vectorizable_type_promotion (stmt, NULL, NULL, NULL) || vectorizable_type_demotion (stmt, NULL, NULL, NULL) || vectorizable_conversion (stmt, NULL, NULL, NULL) + || vectorizable_shift (stmt, NULL, NULL, NULL) || vectorizable_operation (stmt, NULL, NULL, NULL) || vectorizable_assignment (stmt, NULL, NULL, NULL) || vectorizable_load (stmt, NULL, NULL, NULL, NULL) @@ -4073,7 +4282,8 @@ vect_analyze_stmt (gimple stmt, bool *need_to_vect else { if (bb_vinfo) - ok = (vectorizable_operation (stmt, NULL, NULL, node) + ok = (vectorizable_shift (stmt, NULL, NULL, node) + || vectorizable_operation (stmt, NULL, NULL, node) || vectorizable_assignment (stmt, NULL, NULL, node) || vectorizable_load (stmt, NULL, NULL, node, NULL) || vectorizable_store (stmt, NULL, NULL, node)); @@ -4175,6 +4385,11 @@ vect_transform_stmt (gimple stmt, gimple_stmt_iter gcc_assert (done); break; + case shift_vec_info_type: + done = vectorizable_shift (stmt, gsi, &vec_stmt, slp_node); + gcc_assert (done); + break; + case op_vec_info_type: done = vectorizable_operation (stmt, gsi, &vec_stmt, slp_node); gcc_assert (done); Index: tree-vect-slp.c =================================================================== --- tree-vect-slp.c (revision 166065) +++ tree-vect-slp.c (working copy) @@ -1429,7 +1429,8 @@ vect_update_slp_costs_according_to_vf (loop_vec_in stmts. NUMBER_OF_VECTORS is the number of vector defs to create. */ static void -vect_get_constant_vectors (slp_tree slp_node, VEC(tree,heap) **vec_oprnds, +vect_get_constant_vectors (tree op, slp_tree slp_node, + VEC (tree, heap) **vec_oprnds, unsigned int op_num, unsigned int number_of_vectors) { VEC (gimple, heap) *stmts = SLP_TREE_SCALAR_STMTS (slp_node); @@ -1440,7 +1441,7 @@ static void tree t = NULL_TREE; int j, number_of_places_left_in_vector; tree vector_type; - tree op, vop; + tree vop; int group_size = VEC_length (gimple, stmts); unsigned int vec_num, i; int number_of_copies = 1; @@ -1453,10 +1454,7 @@ static void op = gimple_assign_rhs1 (stmt); } else - { - is_store = false; - op = gimple_op (stmt, op_num + 1); - } + is_store = false; if (CONSTANT_CLASS_P (op)) constant_p = true; @@ -1572,7 +1570,8 @@ vect_get_slp_vect_defs (slp_tree slp_node, VEC (tr the right node. This is used when the second operand must remain scalar. */ void -vect_get_slp_defs (slp_tree slp_node, VEC (tree,heap) **vec_oprnds0, +vect_get_slp_defs (tree op0, tree op1, slp_tree slp_node, + VEC (tree,heap) **vec_oprnds0, VEC (tree,heap) **vec_oprnds1) { gimple first_stmt; @@ -1610,7 +1609,7 @@ void vect_get_slp_vect_defs (SLP_TREE_LEFT (slp_node), vec_oprnds0); else /* Build vectors from scalar defs. */ - vect_get_constant_vectors (slp_node, vec_oprnds0, 0, number_of_vects); + vect_get_constant_vectors (op0, slp_node, vec_oprnds0, 0, number_of_vects); if (STMT_VINFO_DATA_REF (vinfo_for_stmt (first_stmt))) /* Since we don't call this function with loads, this is a group of @@ -1635,7 +1634,7 @@ void vect_get_slp_vect_defs (SLP_TREE_RIGHT (slp_node), vec_oprnds1); else /* Build vectors from scalar defs. */ - vect_get_constant_vectors (slp_node, vec_oprnds1, 1, number_of_vects); + vect_get_constant_vectors (op1, slp_node, vec_oprnds1, 1, number_of_vects); }