Message ID | mpttv6xfn6t.fsf@arm.com |
---|---|
State | New |
Headers | show |
Series | Move EXTRACT_LAST_REDUCTION costing to vectorizable_condition | expand |
On Thu, Nov 21, 2019 at 1:07 PM Richard Sandiford <richard.sandiford@arm.com> wrote: > > gcc.target/aarch64/sve/clastb_[57].c started failing after the increase > in the cost of vec_to_scalar (r278452). The problem is that we were > double-counting the cost of the CLASTB: once in vect_model_reduction_cost > as a vec_to_scalar and once in vectorizable_condition as a plain > vector_stmt. > > Based on the TODO above vect_model_reduction_cost, I think the > preferred long-term direction is for vectorizable_* to cost these > things itself, so that's what the patch does (for this one case only). > > Tested on aarch64-linux-gnu and x86_64-linux-gnu. OK to install? OK. > Richard > > > 2019-11-21 Richard Sandiford <richard.sandiford@arm.com> > > gcc/ > * tree-vect-stmts.c (vect_model_simple_cost): Take an optional > vect_cost_for_stmt. > (vectorizable_condition): Calculate the cost of EXTRACT_LAST_REDUCTION > here rather than... > * tree-vect-loop.c (vect_model_reduction_cost): ...here. > > Index: gcc/tree-vect-stmts.c > =================================================================== > --- gcc/tree-vect-stmts.c 2019-11-16 10:29:21.211212189 +0000 > +++ gcc/tree-vect-stmts.c 2019-11-21 12:02:44.058240392 +0000 > @@ -860,7 +860,8 @@ vect_model_simple_cost (stmt_vec_info st > enum vect_def_type *dt, > int ndts, > slp_tree node, > - stmt_vector_for_cost *cost_vec) > + stmt_vector_for_cost *cost_vec, > + vect_cost_for_stmt kind = vector_stmt) > { > int inside_cost = 0, prologue_cost = 0; > > @@ -907,7 +908,7 @@ vect_model_simple_cost (stmt_vec_info st > } > > /* Pass the inside-of-loop statements to the target-specific cost model. */ > - inside_cost += record_stmt_cost (cost_vec, ncopies, vector_stmt, > + inside_cost += record_stmt_cost (cost_vec, ncopies, kind, > stmt_info, 0, vect_body); > > if (dump_enabled_p ()) > @@ -10081,15 +10082,18 @@ vectorizable_condition (stmt_vec_info st > return false; > } > } > - if (expand_vec_cond_expr_p (vectype, comp_vectype, > - cond_code)) > - { > - STMT_VINFO_TYPE (stmt_info) = condition_vec_info_type; > - vect_model_simple_cost (stmt_info, ncopies, dts, ndts, slp_node, > - cost_vec); > - return true; > - } > - return false; > + > + vect_cost_for_stmt kind = vector_stmt; > + if (reduction_type == EXTRACT_LAST_REDUCTION) > + /* Count one reduction-like operation per vector. */ > + kind = vec_to_scalar; > + else if (!expand_vec_cond_expr_p (vectype, comp_vectype, cond_code)) > + return false; > + > + STMT_VINFO_TYPE (stmt_info) = condition_vec_info_type; > + vect_model_simple_cost (stmt_info, ncopies, dts, ndts, slp_node, > + cost_vec, kind); > + return true; > } > > /* Transform. */ > Index: gcc/tree-vect-loop.c > =================================================================== > --- gcc/tree-vect-loop.c 2019-11-19 16:25:24.882645490 +0000 > +++ gcc/tree-vect-loop.c 2019-11-21 12:02:44.050240447 +0000 > @@ -3924,8 +3924,11 @@ vect_model_reduction_cost (stmt_vec_info > > code = gimple_assign_rhs_code (orig_stmt_info->stmt); > > - if (reduction_type == EXTRACT_LAST_REDUCTION > - || reduction_type == FOLD_LEFT_REDUCTION) > + if (reduction_type == EXTRACT_LAST_REDUCTION) > + /* No extra instructions are needed in the prologue. The loop body > + operations are costed in vectorizable_condition. */ > + inside_cost = 0; > + else if (reduction_type == FOLD_LEFT_REDUCTION) > { > /* No extra instructions needed in the prologue. */ > prologue_cost = 0;
Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2019-11-16 10:29:21.211212189 +0000 +++ gcc/tree-vect-stmts.c 2019-11-21 12:02:44.058240392 +0000 @@ -860,7 +860,8 @@ vect_model_simple_cost (stmt_vec_info st enum vect_def_type *dt, int ndts, slp_tree node, - stmt_vector_for_cost *cost_vec) + stmt_vector_for_cost *cost_vec, + vect_cost_for_stmt kind = vector_stmt) { int inside_cost = 0, prologue_cost = 0; @@ -907,7 +908,7 @@ vect_model_simple_cost (stmt_vec_info st } /* Pass the inside-of-loop statements to the target-specific cost model. */ - inside_cost += record_stmt_cost (cost_vec, ncopies, vector_stmt, + inside_cost += record_stmt_cost (cost_vec, ncopies, kind, stmt_info, 0, vect_body); if (dump_enabled_p ()) @@ -10081,15 +10082,18 @@ vectorizable_condition (stmt_vec_info st return false; } } - if (expand_vec_cond_expr_p (vectype, comp_vectype, - cond_code)) - { - STMT_VINFO_TYPE (stmt_info) = condition_vec_info_type; - vect_model_simple_cost (stmt_info, ncopies, dts, ndts, slp_node, - cost_vec); - return true; - } - return false; + + vect_cost_for_stmt kind = vector_stmt; + if (reduction_type == EXTRACT_LAST_REDUCTION) + /* Count one reduction-like operation per vector. */ + kind = vec_to_scalar; + else if (!expand_vec_cond_expr_p (vectype, comp_vectype, cond_code)) + return false; + + STMT_VINFO_TYPE (stmt_info) = condition_vec_info_type; + vect_model_simple_cost (stmt_info, ncopies, dts, ndts, slp_node, + cost_vec, kind); + return true; } /* Transform. */ Index: gcc/tree-vect-loop.c =================================================================== --- gcc/tree-vect-loop.c 2019-11-19 16:25:24.882645490 +0000 +++ gcc/tree-vect-loop.c 2019-11-21 12:02:44.050240447 +0000 @@ -3924,8 +3924,11 @@ vect_model_reduction_cost (stmt_vec_info code = gimple_assign_rhs_code (orig_stmt_info->stmt); - if (reduction_type == EXTRACT_LAST_REDUCTION - || reduction_type == FOLD_LEFT_REDUCTION) + if (reduction_type == EXTRACT_LAST_REDUCTION) + /* No extra instructions are needed in the prologue. The loop body + operations are costed in vectorizable_condition. */ + inside_cost = 0; + else if (reduction_type == FOLD_LEFT_REDUCTION) { /* No extra instructions needed in the prologue. */ prologue_cost = 0;