From patchwork Thu Jul 15 10:27:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1505641 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=Z7hTpmaH; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=C+ILzuQh; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4GQVvc4tCYz9sS8 for ; Thu, 15 Jul 2021 20:28:26 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0B4943AAB016 for ; Thu, 15 Jul 2021 10:28:23 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by sourceware.org (Postfix) with ESMTPS id D47EB386185D for ; Thu, 15 Jul 2021 10:27:58 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org D47EB386185D Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id D477E1FE07; Thu, 15 Jul 2021 10:27:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1626344877; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=MYqRFLmhQiFb3L07cBdzZK2Evd6ckkGeTHBE0/FoZ84=; b=Z7hTpmaH3YwDs0pEemtEp2htyFjhWpGGapbP9H6X+VoS4uebjIecVKV5LUQcvAF8VtgGlZ YINR8PRwBerJMhTlD+60tyMBupLnHpW8du8ixr1UfMzSAWXR6MjOPUabR6uhxu6GtTtaWp st7lo/3V9lEMLGWfvgxpiseWMyphKIg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1626344877; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=MYqRFLmhQiFb3L07cBdzZK2Evd6ckkGeTHBE0/FoZ84=; b=C+ILzuQhwHW2VWpXnd9CUrKfgCoPm+juGISqO7JRMDS6e+Mh4wVS5l8GtQ7K5f3t/VOtVo +a4avF1FazB9gwBA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id BD45F13C2F; Thu, 15 Jul 2021 10:27:57 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 4vseLa0N8GAuGQAAMHmgww (envelope-from ); Thu, 15 Jul 2021 10:27:57 +0000 Date: Thu, 15 Jul 2021 12:27:57 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH 1/2] Streamline vect_gen_while Message-ID: <3q748918-s965-rs18-s3qp-27p7r45o9o@fhfr.qr> MIME-Version: 1.0 X-Spam-Status: No, score=-11.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: richard.sandiford@arm.com Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" This adjusts the vect_gen_while API to match that of vect_gen_while_not allowing further patches to generate more than one stmt for the while case. Bootstrapped and tested on x86_64-unknown-linux-gnu, tested a toy example on SVE that it still produces the same code. OK? 2021-07-15 Richard Biener * tree-vectorizer.h (vect_gen_while): Match up with vect_gen_while_not. * tree-vect-stmts.c (vect_gen_while): Adjust API to that of vect_gen_while_not. (vect_gen_while_not): Adjust. * tree-vect-loop-manip.c (vect_set_loop_controls_directly): Likewise. --- gcc/tree-vect-loop-manip.c | 14 ++++++-------- gcc/tree-vect-stmts.c | 16 ++++++++-------- gcc/tree-vectorizer.h | 3 ++- 3 files changed, 16 insertions(+), 17 deletions(-) diff --git a/gcc/tree-vect-loop-manip.c b/gcc/tree-vect-loop-manip.c index c29ffb3356c..1f3d6614e6c 100644 --- a/gcc/tree-vect-loop-manip.c +++ b/gcc/tree-vect-loop-manip.c @@ -609,11 +609,8 @@ vect_set_loop_controls_directly (class loop *loop, loop_vec_info loop_vinfo, } if (use_masks_p) - { - init_ctrl = make_temp_ssa_name (ctrl_type, NULL, "max_mask"); - gimple *tmp_stmt = vect_gen_while (init_ctrl, start, end); - gimple_seq_add_stmt (preheader_seq, tmp_stmt); - } + init_ctrl = vect_gen_while (preheader_seq, ctrl_type, + start, end, "max_mask"); else { init_ctrl = make_temp_ssa_name (compare_type, NULL, "max_len"); @@ -652,9 +649,10 @@ vect_set_loop_controls_directly (class loop *loop, loop_vec_info loop_vinfo, /* Get the control value for the next iteration of the loop. */ if (use_masks_p) { - next_ctrl = make_temp_ssa_name (ctrl_type, NULL, "next_mask"); - gcall *call = vect_gen_while (next_ctrl, test_index, this_test_limit); - gsi_insert_before (test_gsi, call, GSI_SAME_STMT); + gimple_seq stmts = NULL; + next_ctrl = vect_gen_while (&stmts, ctrl_type, test_index, + this_test_limit, "next_mask"); + gsi_insert_seq_before (test_gsi, stmts, GSI_SAME_STMT); } else { diff --git a/gcc/tree-vect-stmts.c b/gcc/tree-vect-stmts.c index d9eeda50278..6a25d661800 100644 --- a/gcc/tree-vect-stmts.c +++ b/gcc/tree-vect-stmts.c @@ -12002,19 +12002,21 @@ supportable_narrowing_operation (enum tree_code code, /* Generate and return a statement that sets vector mask MASK such that MASK[I] is true iff J + START_INDEX < END_INDEX for all J <= I. */ -gcall * -vect_gen_while (tree mask, tree start_index, tree end_index) +tree +vect_gen_while (gimple_seq *seq, tree mask_type, tree start_index, + tree end_index, const char *name) { tree cmp_type = TREE_TYPE (start_index); - tree mask_type = TREE_TYPE (mask); gcc_checking_assert (direct_internal_fn_supported_p (IFN_WHILE_ULT, cmp_type, mask_type, OPTIMIZE_FOR_SPEED)); gcall *call = gimple_build_call_internal (IFN_WHILE_ULT, 3, start_index, end_index, build_zero_cst (mask_type)); - gimple_call_set_lhs (call, mask); - return call; + tree tmp = make_temp_ssa_name (mask_type, NULL, name); + gimple_call_set_lhs (call, tmp); + gimple_seq_add_stmt (seq, call); + return tmp; } /* Generate a vector mask of type MASK_TYPE for which index I is false iff @@ -12024,9 +12026,7 @@ tree vect_gen_while_not (gimple_seq *seq, tree mask_type, tree start_index, tree end_index) { - tree tmp = make_ssa_name (mask_type); - gcall *call = vect_gen_while (tmp, start_index, end_index); - gimple_seq_add_stmt (seq, call); + tree tmp = vect_gen_while (seq, mask_type, start_index, end_index); return gimple_build (seq, BIT_NOT_EXPR, mask_type, tmp); } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 4c4bc810c35..49afdd898d0 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -1948,7 +1948,8 @@ extern bool vect_supportable_shift (vec_info *, enum tree_code, tree); extern tree vect_gen_perm_mask_any (tree, const vec_perm_indices &); extern tree vect_gen_perm_mask_checked (tree, const vec_perm_indices &); extern void optimize_mask_stores (class loop*); -extern gcall *vect_gen_while (tree, tree, tree); +extern tree vect_gen_while (gimple_seq *, tree, tree, tree, + const char * = nullptr); extern tree vect_gen_while_not (gimple_seq *, tree, tree, tree); extern opt_result vect_get_vector_types_for_stmt (vec_info *, stmt_vec_info, tree *, From patchwork Thu Jul 15 10:30:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1505643 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=a1Ks3HTI; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=ofZWRcTu; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4GQVyJ6h4Wz9sS8 for ; Thu, 15 Jul 2021 20:30:48 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0A9863AAB451 for ; Thu, 15 Jul 2021 10:30:46 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by sourceware.org (Postfix) with ESMTPS id 35D24386185D for ; Thu, 15 Jul 2021 10:30:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 35D24386185D Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 2556E1FE0C; Thu, 15 Jul 2021 10:30:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1626345021; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=PGID5wjucP7KW4uVdHktTUsb7cSkb9pjS/85eaQdMug=; b=a1Ks3HTIFjzfhOS6beCZd9hVzgzUK5aPJ+//vpDvpKnrSJhZThkzow0o1sCVt4O8UiSaOk AC/GEmfn2bTeAQVBUSRRF+esanGTGvR0Jmfz6brF+8j86SfSiDbV7KeEnfw35MgDeTuSUW 437Lr17/R3UbSbvMLOCyjHRE/beCrcg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1626345021; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=PGID5wjucP7KW4uVdHktTUsb7cSkb9pjS/85eaQdMug=; b=ofZWRcTuCiLOxa7j8kaFf0qHCAoORNuypnjopHPY9eLE4lxWzmOo4KEiDK4fnbu+Bo7fZN Js69MKEhHRNr4ECQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 0006213C2F; Thu, 15 Jul 2021 10:30:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 2wMCOjwO8GAAGgAAMHmgww (envelope-from ); Thu, 15 Jul 2021 10:30:20 +0000 Date: Thu, 15 Jul 2021 12:30:20 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH 2/2][RFC] Add loop masking support for x86 Message-ID: <73rrp0p-859r-oq2n-pss7-6744807s3qr5@fhfr.qr> MIME-Version: 1.0 X-Spam-Status: No, score=-11.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_NUMSUBJECT, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: richard.sandiford@arm.com, hongtao.liu@intel.com Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" The following extends the existing loop masking support using SVE WHILE_ULT to x86 by proving an alternate way to produce the mask using VEC_COND_EXPRs. So with --param vect-partial-vector-usage you can now enable masked vectorized epilogues (=1) or fully masked vector loops (=2). What's missing is using a scalar IV for the loop control (but in principle AVX512 can use the mask here - just the patch doesn't seem to work for AVX512 yet for some reason - likely expand_vec_cond_expr_p doesn't work there). What's also missing is providing more support for predicated operations in the case of reductions either via VEC_COND_EXPRs or via implementing some of the .COND_{ADD,SUB,MUL...} internal functions as mapping to masked AVX512 operations. For AVX2 and int foo (unsigned *a, unsigned * __restrict b, int n) { unsigned sum = 1; for (int i = 0; i < n; ++i) b[i] += a[i]; return sum; } we get .L3: vpmaskmovd (%rsi,%rax), %ymm0, %ymm3 vpmaskmovd (%rdi,%rax), %ymm0, %ymm1 addl $8, %edx vpaddd %ymm3, %ymm1, %ymm1 vpmaskmovd %ymm1, %ymm0, (%rsi,%rax) vmovd %edx, %xmm1 vpsubd %ymm15, %ymm2, %ymm0 addq $32, %rax vpbroadcastd %xmm1, %ymm1 vpaddd %ymm4, %ymm1, %ymm1 vpsubd %ymm15, %ymm1, %ymm1 vpcmpgtd %ymm1, %ymm0, %ymm0 vptest %ymm0, %ymm0 jne .L3 for the fully masked loop body and for the masked epilogue we see .L4: vmovdqu (%rsi,%rax), %ymm3 vpaddd (%rdi,%rax), %ymm3, %ymm0 vmovdqu %ymm0, (%rsi,%rax) addq $32, %rax cmpq %rax, %rcx jne .L4 movl %edx, %eax andl $-8, %eax testb $7, %dl je .L11 .L3: subl %eax, %edx vmovdqa .LC0(%rip), %ymm1 salq $2, %rax vmovd %edx, %xmm0 movl $-2147483648, %edx addq %rax, %rsi vmovd %edx, %xmm15 vpbroadcastd %xmm0, %ymm0 vpbroadcastd %xmm15, %ymm15 vpsubd %ymm15, %ymm1, %ymm1 vpsubd %ymm15, %ymm0, %ymm0 vpcmpgtd %ymm1, %ymm0, %ymm0 vpmaskmovd (%rsi), %ymm0, %ymm1 vpmaskmovd (%rdi,%rax), %ymm0, %ymm2 vpaddd %ymm2, %ymm1, %ymm1 vpmaskmovd %ymm1, %ymm0, (%rsi) .L11: vzeroupper compared to .L3: movl %edx, %r8d subl %eax, %r8d leal -1(%r8), %r9d cmpl $2, %r9d jbe .L6 leaq (%rcx,%rax,4), %r9 vmovdqu (%rdi,%rax,4), %xmm2 movl %r8d, %eax andl $-4, %eax vpaddd (%r9), %xmm2, %xmm0 addl %eax, %esi andl $3, %r8d vmovdqu %xmm0, (%r9) je .L2 .L6: movslq %esi, %r8 leaq 0(,%r8,4), %rax movl (%rdi,%r8,4), %r8d addl %r8d, (%rcx,%rax) leal 1(%rsi), %r8d cmpl %r8d, %edx jle .L2 addl $2, %esi movl 4(%rdi,%rax), %r8d addl %r8d, 4(%rcx,%rax) cmpl %esi, %edx jle .L2 movl 8(%rdi,%rax), %edx addl %edx, 8(%rcx,%rax) .L2: I'm giving this a little testing right now but will dig on why I don't get masked loops when AVX512 is enabled. Still comments are appreciated. Thanks, Richard. 2021-07-15 Richard Biener * tree-vect-stmts.c (can_produce_all_loop_masks_p): We also can produce masks with VEC_COND_EXPRs. * tree-vect-loop.c (vect_gen_while): Generate the mask with a VEC_COND_EXPR in case WHILE_ULT is not supported. --- gcc/tree-vect-loop.c | 8 ++++++- gcc/tree-vect-stmts.c | 50 ++++++++++++++++++++++++++++++++++--------- 2 files changed, 47 insertions(+), 11 deletions(-) diff --git a/gcc/tree-vect-loop.c b/gcc/tree-vect-loop.c index fc3dab0d143..2214ed11dfb 100644 --- a/gcc/tree-vect-loop.c +++ b/gcc/tree-vect-loop.c @@ -975,11 +975,17 @@ can_produce_all_loop_masks_p (loop_vec_info loop_vinfo, tree cmp_type) { rgroup_controls *rgm; unsigned int i; + tree cmp_vectype; FOR_EACH_VEC_ELT (LOOP_VINFO_MASKS (loop_vinfo), i, rgm) if (rgm->type != NULL_TREE && !direct_internal_fn_supported_p (IFN_WHILE_ULT, cmp_type, rgm->type, - OPTIMIZE_FOR_SPEED)) + OPTIMIZE_FOR_SPEED) + && ((cmp_vectype + = truth_type_for (build_vector_type + (cmp_type, TYPE_VECTOR_SUBPARTS (rgm->type)))), + true) + && !expand_vec_cond_expr_p (rgm->type, cmp_vectype, LT_EXPR)) return false; return true; } diff --git a/gcc/tree-vect-stmts.c b/gcc/tree-vect-stmts.c index 6a25d661800..216986399b1 100644 --- a/gcc/tree-vect-stmts.c +++ b/gcc/tree-vect-stmts.c @@ -12007,16 +12007,46 @@ vect_gen_while (gimple_seq *seq, tree mask_type, tree start_index, tree end_index, const char *name) { tree cmp_type = TREE_TYPE (start_index); - gcc_checking_assert (direct_internal_fn_supported_p (IFN_WHILE_ULT, - cmp_type, mask_type, - OPTIMIZE_FOR_SPEED)); - gcall *call = gimple_build_call_internal (IFN_WHILE_ULT, 3, - start_index, end_index, - build_zero_cst (mask_type)); - tree tmp = make_temp_ssa_name (mask_type, NULL, name); - gimple_call_set_lhs (call, tmp); - gimple_seq_add_stmt (seq, call); - return tmp; + if (direct_internal_fn_supported_p (IFN_WHILE_ULT, + cmp_type, mask_type, + OPTIMIZE_FOR_SPEED)) + { + gcall *call = gimple_build_call_internal (IFN_WHILE_ULT, 3, + start_index, end_index, + build_zero_cst (mask_type)); + tree tmp = make_temp_ssa_name (mask_type, NULL, name); + gimple_call_set_lhs (call, tmp); + gimple_seq_add_stmt (seq, call); + return tmp; + } + else + { + /* Generate + _1 = { start_index, start_index, ... }; + _2 = { end_index, end_index, ... }; + _3 = _1 + { 0, 1, 2 ... }; + _4 = _3 < _2; + _5 = VEC_COND_EXPR <_4, { -1, -1, ... } : { 0, 0, ... }>; */ + tree cvectype = build_vector_type (cmp_type, + TYPE_VECTOR_SUBPARTS (mask_type)); + tree si = make_ssa_name (cvectype); + gassign *ass = gimple_build_assign + (si, build_vector_from_val (cvectype, start_index)); + gimple_seq_add_stmt (seq, ass); + tree ei = make_ssa_name (cvectype); + ass = gimple_build_assign (ei, + build_vector_from_val (cvectype, end_index)); + gimple_seq_add_stmt (seq, ass); + tree incr = build_vec_series (cvectype, build_zero_cst (cmp_type), + build_one_cst (cmp_type)); + si = gimple_build (seq, PLUS_EXPR, cvectype, si, incr); + tree cmp = gimple_build (seq, LT_EXPR, truth_type_for (cvectype), + si, ei); + tree mask = gimple_build (seq, VEC_COND_EXPR, mask_type, cmp, + build_all_ones_cst (mask_type), + build_zero_cst (mask_type)); + return mask; + } } /* Generate a vector mask of type MASK_TYPE for which index I is false iff