From patchwork Tue Oct 17 11:34:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lehua Ding X-Patchwork-Id: 1849972 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8sNp2LLQz20Pd for ; Tue, 17 Oct 2023 22:35:38 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 5C4D8385773C for ; Tue, 17 Oct 2023 11:35:36 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgsg1.qq.com (smtpbgsg1.qq.com [54.254.200.92]) by sourceware.org (Postfix) with ESMTPS id ED11A3858418 for ; Tue, 17 Oct 2023 11:35:14 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org ED11A3858418 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai ARC-Filter: OpenARC Filter v1.0.0 sourceware.org ED11A3858418 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=54.254.200.92 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697542522; cv=none; b=ZT+v9AwSIbgUnpCstnDTicVwumUvVV6ZjuaQos/OPN6N9TxnldUm4C8yHDUWkQv20hJihDdzQ/3RU05uq2di370SzOhJvqwFBrpvNLy0PZ5VDhLU7LuCj4p4oGYSTLhF01SNBAJiTdytYDBZIBWlzXUswSEKUkAgdTxP9/TMpOg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697542522; c=relaxed/simple; bh=v2+tcL5wAqIw9pWosbYeUqdU+J0kAs/4DrAliLeTz/Q=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=RrStghFm7JbHmJw7zkm/gMKnBj343hHWhArB1oye+9S5AeWbhoVfYAkmQH5E7H/0W/eCoxITO0zQEmD7cWPpOmZYOw+RY7MXJIx7jc3XTRW0ZGGhOG5fBaw7u8FFCvClny9604PW6mh/3znxuuWkb31b58HqSeyYPPUkjvzw5K0= ARC-Authentication-Results: i=1; server2.sourceware.org X-QQ-mid: bizesmtp62t1697542507tlsfaoy9 Received: from rios-cad121.hadoop.rioslab.org ( [58.60.1.9]) by bizesmtp.qq.com (ESMTP) with id ; Tue, 17 Oct 2023 19:35:06 +0800 (CST) X-QQ-SSF: 01400000000000C0F000000A0000000 X-QQ-FEAT: HEHmdpKxZe2hdpm1ES5wNeI3HVog1k1hOkJhUbQ6zuSRIA0YS28Wq8c2Z9N3m uHE8C2H78EftI6i+nEPxSEoFiXd9FEx9mVW8bxm4F+msFi643s1mAX2cpkTPKb1sXE5HH7S DB0HgcQJbGM25qUenXMSdas8V2mX75EBs6Hbnzy5Zn5yXXMSVnKaGdstnjFGOSkclm4R0pA os71N9dk8/xwfWz90WsMIyBAKhZ8osan10+0hh3eTOt0D97uzT5MyUDl7yacDzi+jOFUC5D LxgGziFomFoCwxDfdCAttWGcEPCCjHNAZGMHAAHOYRse7SdNMBp+5sX0Y1vHGPRM2A7btmV VipdQyfGQiSQRS1luzFci16YSx/zhbj3FQDmSV00xif3uZD8BgafnYjJfqO6ERIqNPZlmef X-QQ-GoodBg: 2 X-BIZMAIL-ID: 9194336455871869230 From: Lehua Ding To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, rdapp.gcc@gmail.com, palmer@rivosinc.com, jeffreyalaw@gmail.com, lehua.ding@rivai.ai Subject: [PATCH V2 01/14] RISC-V: P1: Refactor avl_info/vl_vtype_info/vector_insn_info Date: Tue, 17 Oct 2023 19:34:47 +0800 Message-Id: <20231017113500.1160997-2-lehua.ding@rivai.ai> X-Mailer: git-send-email 2.36.3 In-Reply-To: <20231017113500.1160997-1-lehua.ding@rivai.ai> References: <20231017113500.1160997-1-lehua.ding@rivai.ai> MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz6a-0 X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org This sub-patch combine avl_info/vl_vtype_info/vector_insn_info to a single class vsetvl_info. gcc/ChangeLog: * config/riscv/riscv-vsetvl.cc (avl_info::avl_info): Removed. (avl_info::single_source_equal_p): Ditto. (avl_info::multiple_source_equal_p): Ditto. (avl_info::operator=): Ditto. (avl_info::operator==): Ditto. (avl_info::operator!=): Ditto. (avl_info::has_non_zero_avl): Ditto. (vl_vtype_info::vl_vtype_info): Ditto. (vl_vtype_info::operator==): Ditto. (vl_vtype_info::operator!=): Ditto. (vl_vtype_info::same_avl_p): Ditto. (vl_vtype_info::same_vtype_p): Ditto. (enum demand_flags): New enum. (vl_vtype_info::same_vlmax_p): Removed. (vector_insn_info::operator>=): Ditto. (enum class): New demand types. (vector_insn_info::operator==): Ditto. (vector_insn_info::parse_insn): Ditto. (class vsetvl_info): New class. (vector_insn_info::compatible_p): Removed. (vector_insn_info::skip_avl_compatible_p): Ditto. (vector_insn_info::compatible_avl_p): Ditto. (vector_insn_info::compatible_vtype_p): Ditto. (vector_insn_info::available_p): Ditto. (vector_insn_info::fuse_avl): Ditto. (vector_insn_info::fuse_sew_lmul): Ditto. (vector_insn_info::fuse_tail_policy): Ditto. (vector_insn_info::fuse_mask_policy): Ditto. (vector_insn_info::local_merge): Ditto. (vector_insn_info::global_merge): Ditto. (vector_insn_info::get_avl_or_vl_reg): Ditto. (vector_insn_info::update_fault_first_load_avl): Ditto. (vlmul_to_str): Ditto. (policy_to_str): Ditto. (vector_insn_info::dump): Ditto. * config/riscv/riscv-vsetvl.h (class avl_info): Ditto. (struct vl_vtype_info): Ditto. (class vector_insn_info): Ditto. --- gcc/config/riscv/riscv-vsetvl.cc | 1315 ++++++++++++------------------ gcc/config/riscv/riscv-vsetvl.h | 261 ------ 2 files changed, 515 insertions(+), 1061 deletions(-) -- 2.36.3 diff --git a/gcc/config/riscv/riscv-vsetvl.cc b/gcc/config/riscv/riscv-vsetvl.cc index 4b06d93e7f9..79ba8466556 100644 --- a/gcc/config/riscv/riscv-vsetvl.cc +++ b/gcc/config/riscv/riscv-vsetvl.cc @@ -1581,827 +1581,542 @@ vsetvl_dominated_by_p (const basic_block cfg_bb, return true; } -avl_info::avl_info (const avl_info &other) -{ - m_value = other.get_value (); - m_source = other.get_source (); -} - -avl_info::avl_info (rtx value_in, set_info *source_in) - : m_value (value_in), m_source (source_in) -{} - -bool -avl_info::single_source_equal_p (const avl_info &other) const -{ - set_info *set1 = m_source; - set_info *set2 = other.get_source (); - insn_info *insn1 = extract_single_source (set1); - insn_info *insn2 = extract_single_source (set2); - if (!insn1 || !insn2) - return false; - return source_equal_p (insn1, insn2); -} - -bool -avl_info::multiple_source_equal_p (const avl_info &other) const -{ - /* When the def info is same in RTL_SSA namespace, it's safe - to consider they are avl compatible. */ - if (m_source == other.get_source ()) - return true; - - /* We only consider handle PHI node. */ - if (!m_source->insn ()->is_phi () || !other.get_source ()->insn ()->is_phi ()) - return false; - - phi_info *phi1 = as_a (m_source); - phi_info *phi2 = as_a (other.get_source ()); - - if (phi1->is_degenerate () && phi2->is_degenerate ()) - { - /* Degenerate PHI means the PHI node only have one input. */ - - /* If both PHI nodes have the same single input in use list. - We consider they are AVL compatible. */ - if (phi1->input_value (0) == phi2->input_value (0)) - return true; - } - /* TODO: We can support more optimization cases in the future. */ - return false; -} - -avl_info & -avl_info::operator= (const avl_info &other) -{ - m_value = other.get_value (); - m_source = other.get_source (); - return *this; -} - -bool -avl_info::operator== (const avl_info &other) const -{ - if (!m_value) - return !other.get_value (); - if (!other.get_value ()) - return false; - - if (GET_CODE (m_value) != GET_CODE (other.get_value ())) - return false; - - /* Handle CONST_INT AVL. */ - if (CONST_INT_P (m_value)) - return INTVAL (m_value) == INTVAL (other.get_value ()); - - /* Handle VLMAX AVL. */ - if (vlmax_avl_p (m_value)) - return vlmax_avl_p (other.get_value ()); - if (vlmax_avl_p (other.get_value ())) - return false; - - /* If any source is undef value, we think they are not equal. */ - if (!m_source || !other.get_source ()) - return false; - - /* If both sources are single source (defined by a single real RTL) - and their definitions are same. */ - if (single_source_equal_p (other)) - return true; - - return multiple_source_equal_p (other); -} - -bool -avl_info::operator!= (const avl_info &other) const -{ - return !(*this == other); -} - -bool -avl_info::has_non_zero_avl () const -{ - if (has_avl_imm ()) - return INTVAL (get_value ()) > 0; - if (has_avl_reg ()) - return vlmax_avl_p (get_value ()); - return false; -} - -/* Initialize VL/VTYPE information. */ -vl_vtype_info::vl_vtype_info (avl_info avl_in, uint8_t sew_in, - enum vlmul_type vlmul_in, uint8_t ratio_in, - bool ta_in, bool ma_in) - : m_avl (avl_in), m_sew (sew_in), m_vlmul (vlmul_in), m_ratio (ratio_in), - m_ta (ta_in), m_ma (ma_in) -{ - gcc_assert (valid_sew_p (m_sew) && "Unexpected SEW"); -} - -bool -vl_vtype_info::operator== (const vl_vtype_info &other) const -{ - return same_avl_p (other) && m_sew == other.get_sew () - && m_vlmul == other.get_vlmul () && m_ta == other.get_ta () - && m_ma == other.get_ma () && m_ratio == other.get_ratio (); -} - -bool -vl_vtype_info::operator!= (const vl_vtype_info &other) const -{ - return !(*this == other); -} - -bool -vl_vtype_info::same_avl_p (const vl_vtype_info &other) const -{ - /* We need to compare both RTL and SET. If both AVL are CONST_INT. - For example, const_int 3 and const_int 4, we need to compare - RTL. If both AVL are REG and their REGNO are same, we need to - compare SET. */ - return get_avl () == other.get_avl () - && get_avl_source () == other.get_avl_source (); -} - -bool -vl_vtype_info::same_vtype_p (const vl_vtype_info &other) const -{ - return get_sew () == other.get_sew () && get_vlmul () == other.get_vlmul () - && get_ta () == other.get_ta () && get_ma () == other.get_ma (); -} +/* This flags indicates the minimum demand of the vl and vtype values by the + RVV instruction. For example, DEMAND_RATIO_P indicates that this RVV + instruction only needs the SEW/LMUL ratio to remain the same, and does not + require SEW and LMUL to be fixed. + Therefore, if the former RVV instruction needs DEMAND_RATIO_P and the latter + instruction needs DEMAND_SEW_LMUL_P and its SEW/LMUL is the same as that of + the former instruction, then we can make the minimu demand of the former + instruction strict to DEMAND_SEW_LMUL_P, and its required SEW and LMUL are + the SEW and LMUL of the latter instruction, and the vsetvl instruction + generated according to the new demand can also be used for the latter + instruction, so there is no need to insert a separate vsetvl instruction for + the latter instruction. */ +enum demand_flags : unsigned +{ + DEMAND_EMPTY_P = 0, + DEMAND_SEW_P = 1 << 0, + DEMAND_LMUL_P = 1 << 1, + DEMAND_RATIO_P = 1 << 2, + DEMAND_GE_SEW_P = 1 << 3, + DEMAND_TAIL_POLICY_P = 1 << 4, + DEMAND_MASK_POLICY_P = 1 << 5, + DEMAND_AVL_P = 1 << 6, + DEMAND_NON_ZERO_AVL_P = 1 << 7, +}; -bool -vl_vtype_info::same_vlmax_p (const vl_vtype_info &other) const -{ - return get_ratio () == other.get_ratio (); -} +/* We split the demand information into three parts. They are sew and lmul + related (sew_lmul_demand_type), tail and mask policy related + (policy_demand_type) and avl related (avl_demand_type). Then we define three + interfaces avaiable_with, compatible_with and merge_with. avaiable_with is + used to determine whether the two vsetvl infos prev_info and next_info are + available or not. If prev_info is available for next_info, it means that the + RVV insn corresponding to next_info on the path from prev_info to next_info + can be used without inserting a separate vsetvl instruction. compatible_with + is used to determine whether prev_info is compatible with next_info, and if + so, merge_with can be used to merge the stricter demand information from + next_info into prev_info so that prev_info becomes available to next_info. + */ -/* Compare the compatibility between Dem1 and Dem2. - If Dem1 > Dem2, Dem1 has bigger compatibility then Dem2 - meaning Dem1 is easier be compatible with others than Dem2 - or Dem2 is stricter than Dem1. - For example, Dem1 (demand SEW + LMUL) > Dem2 (demand RATIO). */ -bool -vector_insn_info::operator>= (const vector_insn_info &other) const +enum class sew_lmul_demand_type : unsigned { - if (support_relaxed_compatible_p (*this, other)) - { - unsigned array_size = sizeof (unavailable_conds) / sizeof (demands_cond); - /* Bypass AVL unavailable cases. */ - for (unsigned i = 2; i < array_size; i++) - if (unavailable_conds[i].pair.match_cond_p (this->get_demands (), - other.get_demands ()) - && unavailable_conds[i].incompatible_p (*this, other)) - return false; - return true; - } - - if (!other.compatible_p (static_cast (*this))) - return false; - if (!this->compatible_p (static_cast (other))) - return true; - - if (*this == other) - return true; - - for (const auto &cond : unavailable_conds) - if (cond.pair.match_cond_p (this->get_demands (), other.get_demands ()) - && cond.incompatible_p (*this, other)) - return false; - - return true; -} + sew_lmul = demand_flags::DEMAND_SEW_P | demand_flags::DEMAND_LMUL_P, + ratio_only = demand_flags::DEMAND_RATIO_P, + sew_only = demand_flags::DEMAND_SEW_P, + ge_sew = demand_flags::DEMAND_GE_SEW_P, + ratio_and_ge_sew + = demand_flags::DEMAND_RATIO_P | demand_flags::DEMAND_GE_SEW_P, +}; -bool -vector_insn_info::operator== (const vector_insn_info &other) const +enum class policy_demand_type : unsigned { - gcc_assert (!uninit_p () && !other.uninit_p () - && "Uninitialization should not happen"); - - /* Empty is only equal to another Empty. */ - if (empty_p ()) - return other.empty_p (); - if (other.empty_p ()) - return empty_p (); - - /* Unknown is only equal to another Unknown. */ - if (unknown_p ()) - return other.unknown_p (); - if (other.unknown_p ()) - return unknown_p (); - - for (size_t i = 0; i < NUM_DEMAND; i++) - if (m_demands[i] != other.demand_p ((enum demand_type) i)) - return false; - - /* We should consider different INSN demands as different - expression. Otherwise, we will be doing incorrect vsetvl - elimination. */ - if (m_insn != other.get_insn ()) - return false; - - if (!same_avl_p (other)) - return false; - - /* If the full VTYPE is valid, check that it is the same. */ - return same_vtype_p (other); -} + tail_mask_policy + = demand_flags::DEMAND_TAIL_POLICY_P | demand_flags::DEMAND_MASK_POLICY_P, + tail_policy_only = demand_flags::DEMAND_TAIL_POLICY_P, + mask_policy_only = demand_flags::DEMAND_MASK_POLICY_P, + ignore_policy = demand_flags::DEMAND_EMPTY_P, +}; -void -vector_insn_info::parse_insn (rtx_insn *rinsn) +enum class avl_demand_type : unsigned { - *this = vector_insn_info (); - if (!NONDEBUG_INSN_P (rinsn)) - return; - if (optimize == 0 && !has_vtype_op (rinsn)) - return; - gcc_assert (!vsetvl_discard_result_insn_p (rinsn)); - m_state = VALID; - extract_insn_cached (rinsn); - rtx avl = ::get_avl (rinsn); - m_avl = avl_info (avl, nullptr); - m_sew = ::get_sew (rinsn); - m_vlmul = ::get_vlmul (rinsn); - m_ta = tail_agnostic_p (rinsn); - m_ma = mask_agnostic_p (rinsn); -} + avl = demand_flags::DEMAND_AVL_P, + non_zero_avl = demand_flags::DEMAND_NON_ZERO_AVL_P, + ignore_avl = demand_flags::DEMAND_EMPTY_P, +}; -void -vector_insn_info::parse_insn (insn_info *insn) +class vsetvl_info { - *this = vector_insn_info (); - - /* Return if it is debug insn for the consistency with optimize == 0. */ - if (insn->is_debug_insn ()) - return; +private: + insn_info *m_insn; + bb_info *m_bb; + rtx m_avl; + rtx m_vl; + set_info *m_avl_def; + uint8_t m_sew; + uint8_t m_max_sew; + vlmul_type m_vlmul; + uint8_t m_ratio; + bool m_ta; + bool m_ma; + + sew_lmul_demand_type m_sew_lmul_demand; + policy_demand_type m_policy_demand; + avl_demand_type m_avl_demand; + + enum class state_type + { + UNINITIALIZED, + VALID, + UNKNOWN, + EMPTY, + }; + state_type m_state; + + bool m_ignore; + bool change_vtype_only; + insn_info *m_read_vl_insn; + bool use_by_non_rvv_insn; - /* We set it as unknown since we don't what will happen in CALL or ASM. */ - if (insn->is_call () || insn->is_asm ()) - { - set_unknown (); +public: + vsetvl_info () + : m_insn (nullptr), m_bb (nullptr), m_avl (NULL_RTX), m_vl (NULL_RTX), + m_avl_def (nullptr), m_sew (0), m_max_sew (0), m_vlmul (LMUL_RESERVED), + m_ratio (0), m_ta (false), m_ma (false), + m_sew_lmul_demand (sew_lmul_demand_type::sew_lmul), + m_policy_demand (policy_demand_type::tail_mask_policy), + m_avl_demand (avl_demand_type::avl), m_state (state_type::UNINITIALIZED), + m_ignore (false), change_vtype_only (false), m_read_vl_insn (nullptr), + use_by_non_rvv_insn (false) + {} + + vsetvl_info (insn_info *insn) : vsetvl_info () { parse_insn (insn); } + + vsetvl_info (rtx_insn *insn) : vsetvl_info () { parse_insn (insn); } + + void set_avl (rtx avl) { m_avl = avl; } + void set_vl (rtx vl) { m_vl = vl; } + void set_avl_def (set_info *avl_def) { m_avl_def = avl_def; } + void set_sew (uint8_t sew) { m_sew = sew; } + void set_vlmul (vlmul_type vlmul) { m_vlmul = vlmul; } + void set_ratio (uint8_t ratio) { m_ratio = ratio; } + void set_ta (bool ta) { m_ta = ta; } + void set_ma (bool ma) { m_ma = ma; } + void set_ignore () { m_ignore = true; } + void set_bb (bb_info *bb) { m_bb = bb; } + void set_max_sew (uint8_t max_sew) { m_max_sew = max_sew; } + void set_change_vtype_only () { change_vtype_only = true; } + void set_read_vl_insn (insn_info *insn) { m_read_vl_insn = insn; } + + rtx get_avl () const { return m_avl; } + rtx get_vl () const { return m_vl; } + set_info *get_avl_def () const { return m_avl_def; } + uint8_t get_sew () const { return m_sew; } + vlmul_type get_vlmul () const { return m_vlmul; } + uint8_t get_ratio () const { return m_ratio; } + bool get_ta () const { return m_ta; } + bool get_ma () const { return m_ma; } + insn_info *get_insn () const { return m_insn; } + bool ignore_p () const { return m_ignore; } + bb_info *get_bb () const { return m_bb; } + uint8_t get_max_sew () const { return m_max_sew; } + insn_info *get_read_vl_insn () const { return m_read_vl_insn; } + bool use_by_non_rvv_insn_p () const { return use_by_non_rvv_insn; } + + bool has_imm_avl () const { return m_avl && CONST_INT_P (m_avl); } + bool has_vlmax_avl () const { return vlmax_avl_p (m_avl); } + bool has_reg_avl () const + { + return m_avl && REG_P (m_avl) && !has_vlmax_avl (); + } + bool has_non_zero_avl () const + { + if (has_imm_avl ()) + return INTVAL (m_avl) > 0; + return has_vlmax_avl (); + } + bool has_reg_vl () const + { + gcc_assert (!m_vl || REG_P (m_vl)); + return m_vl && REG_P (m_vl); + } + bool has_same_ratio (const vsetvl_info &other) const + { + return get_ratio () == other.get_ratio (); + } + bool is_in_origin_bb () const { return get_insn ()->bb () == get_bb (); } + void update_avl (const vsetvl_info &other) + { + m_avl = other.get_avl (); + m_vl = other.get_vl (); + m_avl_def = other.get_avl_def (); + } + + bool uninit_p () const { return m_state == state_type::UNINITIALIZED; } + bool valid_p () const { return m_state == state_type::VALID; } + bool unknown_p () const { return m_state == state_type::UNKNOWN; } + bool empty_p () const { return m_state == state_type::EMPTY; } + bool change_vtype_only_p () const { return change_vtype_only; } + + void set_valid () { m_state = state_type::VALID; } + void set_unknown () { m_state = state_type::UNKNOWN; } + void set_empty () { m_state = state_type::EMPTY; } + + void set_sew_lmul_demand (sew_lmul_demand_type demand) + { + m_sew_lmul_demand = demand; + } + void set_policy_demand (policy_demand_type demand) + { + m_policy_demand = demand; + } + void set_avl_demand (avl_demand_type demand) { m_avl_demand = demand; } + + sew_lmul_demand_type get_sew_lmul_demand () const + { + return m_sew_lmul_demand; + } + policy_demand_type get_policy_demand () const { return m_policy_demand; } + avl_demand_type get_avl_demand () const { return m_avl_demand; } + + void normalize_demand (unsigned demand_flags) + { + switch (demand_flags + & (DEMAND_SEW_P | DEMAND_LMUL_P | DEMAND_RATIO_P | DEMAND_GE_SEW_P)) + { + case (unsigned) sew_lmul_demand_type::sew_lmul: + m_sew_lmul_demand = sew_lmul_demand_type::sew_lmul; + break; + case (unsigned) sew_lmul_demand_type::ratio_only: + m_sew_lmul_demand = sew_lmul_demand_type::ratio_only; + break; + case (unsigned) sew_lmul_demand_type::sew_only: + m_sew_lmul_demand = sew_lmul_demand_type::sew_only; + break; + case (unsigned) sew_lmul_demand_type::ge_sew: + m_sew_lmul_demand = sew_lmul_demand_type::ge_sew; + break; + case (unsigned) sew_lmul_demand_type::ratio_and_ge_sew: + m_sew_lmul_demand = sew_lmul_demand_type::ratio_and_ge_sew; + break; + default: + gcc_unreachable (); + } + + switch (demand_flags & (DEMAND_TAIL_POLICY_P | DEMAND_MASK_POLICY_P)) + { + case (unsigned) policy_demand_type::tail_mask_policy: + m_policy_demand = policy_demand_type::tail_mask_policy; + break; + case (unsigned) policy_demand_type::tail_policy_only: + m_policy_demand = policy_demand_type::tail_policy_only; + break; + case (unsigned) policy_demand_type::mask_policy_only: + m_policy_demand = policy_demand_type::mask_policy_only; + break; + case (unsigned) policy_demand_type::ignore_policy: + m_policy_demand = policy_demand_type::ignore_policy; + break; + default: + gcc_unreachable (); + } + + switch (demand_flags & (DEMAND_AVL_P | DEMAND_NON_ZERO_AVL_P)) + { + case (unsigned) avl_demand_type::avl: + m_avl_demand = avl_demand_type::avl; + break; + case (unsigned) avl_demand_type::non_zero_avl: + m_avl_demand = avl_demand_type::non_zero_avl; + break; + case (unsigned) avl_demand_type::ignore_avl: + m_avl_demand = avl_demand_type::ignore_avl; + break; + default: + gcc_unreachable (); + } + } + + void parse_insn (rtx_insn *rinsn) + { + if (!NONDEBUG_INSN_P (rinsn)) return; - } - - /* If this is something that updates VL/VTYPE that we don't know about, set - the state to unknown. */ - if (!vector_config_insn_p (insn->rtl ()) && !has_vtype_op (insn->rtl ()) - && (find_access (insn->defs (), VL_REGNUM) - || find_access (insn->defs (), VTYPE_REGNUM))) - { - set_unknown (); + if (optimize == 0 && !has_vtype_op (rinsn)) return; - } - - if (!vector_config_insn_p (insn->rtl ()) && !has_vtype_op (insn->rtl ())) - return; - - /* Warning: This function has to work on both the lowered (i.e. post - emit_local_forward_vsetvls) and pre-lowering forms. The main implication - of this is that it can't use the value of a SEW, VL, or Policy operand as - they might be stale after lowering. */ - vl_vtype_info::operator= (get_vl_vtype_info (insn)); - m_insn = insn; - m_state = VALID; - if (vector_config_insn_p (insn->rtl ())) - { - m_demands[DEMAND_AVL] = true; - m_demands[DEMAND_RATIO] = true; + gcc_assert (!vsetvl_discard_result_insn_p (rinsn)); + set_valid (); + extract_insn_cached (rinsn); + m_avl = ::get_avl (rinsn); + if (has_vlmax_avl () || vsetvl_insn_p (rinsn)) + m_vl = ::get_vl (rinsn); + m_sew = ::get_sew (rinsn); + m_vlmul = ::get_vlmul (rinsn); + m_ta = tail_agnostic_p (rinsn); + m_ma = mask_agnostic_p (rinsn); + } + + void parse_insn (insn_info *insn) + { + m_insn = insn; + m_bb = insn->bb (); + /* Return if it is debug insn for the consistency with optimize == 0. */ + if (insn->is_debug_insn ()) return; - } - - if (has_vl_op (insn->rtl ())) - m_demands[DEMAND_AVL] = true; - - if (get_attr_ratio (insn->rtl ()) != INVALID_ATTRIBUTE) - m_demands[DEMAND_RATIO] = true; - else - { - /* TODO: By default, if it doesn't demand RATIO, we set it - demand SEW && LMUL both. Some instructions may demand SEW - only and ignore LMUL, will fix it later. */ - m_demands[DEMAND_SEW] = true; - if (!ignore_vlmul_insn_p (insn->rtl ())) - m_demands[DEMAND_LMUL] = true; - } - - if (get_attr_ta (insn->rtl ()) != INVALID_ATTRIBUTE) - m_demands[DEMAND_TAIL_POLICY] = true; - if (get_attr_ma (insn->rtl ()) != INVALID_ATTRIBUTE) - m_demands[DEMAND_MASK_POLICY] = true; - - if (vector_config_insn_p (insn->rtl ())) - return; - - if (scalar_move_insn_p (insn->rtl ())) - { - if (m_avl.has_non_zero_avl ()) - m_demands[DEMAND_NONZERO_AVL] = true; - if (m_ta) - m_demands[DEMAND_GE_SEW] = true; - } - - if (!m_avl.has_avl_reg () || vlmax_avl_p (get_avl ()) || !m_avl.get_source ()) - return; - if (!m_avl.get_source ()->insn ()->is_real () - && !m_avl.get_source ()->insn ()->is_phi ()) - return; - - insn_info *def_insn = extract_single_source (m_avl.get_source ()); - if (!def_insn || !vsetvl_insn_p (def_insn->rtl ())) - return; - - vector_insn_info new_info; - new_info.parse_insn (def_insn); - if (!same_vlmax_p (new_info) && !scalar_move_insn_p (insn->rtl ())) - return; - - if (new_info.has_avl ()) - { - if (new_info.has_avl_imm ()) - set_avl_info (avl_info (new_info.get_avl (), nullptr)); - else - { - if (vlmax_avl_p (new_info.get_avl ())) - set_avl_info (avl_info (new_info.get_avl (), get_avl_source ())); - else - { - /* Conservatively propagate non-VLMAX AVL of user vsetvl: - 1. The user vsetvl should be same block with the rvv insn. - 2. The user vsetvl is the only def insn of rvv insn. - 3. The AVL is not modified between def-use chain. - 4. The VL is only used by insn within EBB. - */ - bool modified_p = false; - for (insn_info *i = def_insn->next_nondebug_insn (); - real_insn_and_same_bb_p (i, get_insn ()->bb ()); - i = i->next_nondebug_insn ()) - { - /* Consider this following sequence: - - insn 1: vsetvli a5,a3,e8,mf4,ta,mu - insn 2: vsetvli zero,a5,e32,m1,ta,ma - ... - vle32.v v1,0(a1) - vsetvli a2,zero,e32,m1,ta,ma - vadd.vv v1,v1,v1 - vsetvli zero,a5,e32,m1,ta,ma - vse32.v v1,0(a0) - ... - insn 3: sub a3,a3,a5 - ... - - We can local AVL propagate "a3" from insn 1 to insn 2 - if no insns between insn 1 and insn 2 modify "a3 even - though insn 3 modifies "a3". - Otherwise, we can't perform local AVL propagation. - - Early break if we reach the insn 2. */ - if (!before_p (i, insn)) - break; - if (find_access (i->defs (), REGNO (new_info.get_avl ()))) - { - modified_p = true; - break; - } - } - - bool has_live_out_use = false; - for (use_info *use : m_avl.get_source ()->all_uses ()) - { - if (use->is_live_out_use ()) - { - has_live_out_use = true; - break; - } - } - if (!modified_p && !has_live_out_use - && def_insn == m_avl.get_source ()->insn () - && m_insn->bb () == def_insn->bb ()) - set_avl_info (new_info.get_avl_info ()); - } - } - } - if (scalar_move_insn_p (insn->rtl ()) && m_avl.has_non_zero_avl ()) - m_demands[DEMAND_NONZERO_AVL] = true; -} - -bool -vector_insn_info::compatible_p (const vector_insn_info &other) const -{ - gcc_assert (valid_or_dirty_p () && other.valid_or_dirty_p () - && "Can't compare invalid demanded infos"); - - for (const auto &cond : incompatible_conds) - if (cond.dual_incompatible_p (*this, other)) - return false; - return true; -} - -bool -vector_insn_info::skip_avl_compatible_p (const vector_insn_info &other) const -{ - gcc_assert (valid_or_dirty_p () && other.valid_or_dirty_p () - && "Can't compare invalid demanded infos"); - unsigned array_size = sizeof (incompatible_conds) / sizeof (demands_cond); - /* Bypass AVL incompatible cases. */ - for (unsigned i = 1; i < array_size; i++) - if (incompatible_conds[i].dual_incompatible_p (*this, other)) - return false; - return true; -} - -bool -vector_insn_info::compatible_avl_p (const vl_vtype_info &other) const -{ - gcc_assert (valid_or_dirty_p () && "Can't compare invalid vl_vtype_info"); - gcc_assert (!unknown_p () && "Can't compare AVL in unknown state"); - if (!demand_p (DEMAND_AVL)) - return true; - if (demand_p (DEMAND_NONZERO_AVL) && other.has_non_zero_avl ()) - return true; - return get_avl_info () == other.get_avl_info (); -} - -bool -vector_insn_info::compatible_avl_p (const avl_info &other) const -{ - gcc_assert (valid_or_dirty_p () && "Can't compare invalid vl_vtype_info"); - gcc_assert (!unknown_p () && "Can't compare AVL in unknown state"); - gcc_assert (demand_p (DEMAND_AVL) && "Can't compare AVL undemand state"); - if (!demand_p (DEMAND_AVL)) - return true; - if (demand_p (DEMAND_NONZERO_AVL) && other.has_non_zero_avl ()) - return true; - return get_avl_info () == other; -} - -bool -vector_insn_info::compatible_vtype_p (const vl_vtype_info &other) const -{ - gcc_assert (valid_or_dirty_p () && "Can't compare invalid vl_vtype_info"); - gcc_assert (!unknown_p () && "Can't compare VTYPE in unknown state"); - if (demand_p (DEMAND_SEW)) - { - if (!demand_p (DEMAND_GE_SEW) && m_sew != other.get_sew ()) - return false; - if (demand_p (DEMAND_GE_SEW) && m_sew > other.get_sew ()) - return false; - } - if (demand_p (DEMAND_LMUL) && m_vlmul != other.get_vlmul ()) - return false; - if (demand_p (DEMAND_RATIO) && m_ratio != other.get_ratio ()) - return false; - if (demand_p (DEMAND_TAIL_POLICY) && m_ta != other.get_ta ()) - return false; - if (demand_p (DEMAND_MASK_POLICY) && m_ma != other.get_ma ()) - return false; - return true; -} - -/* Determine whether the vector instructions requirements represented by - Require are compatible with the previous vsetvli instruction represented - by this. INSN is the instruction whose requirements we're considering. */ -bool -vector_insn_info::compatible_p (const vl_vtype_info &curr_info) const -{ - gcc_assert (!uninit_p () && "Can't handle uninitialized info"); - if (empty_p ()) - return false; - - /* Nothing is compatible with Unknown. */ - if (unknown_p ()) - return false; - - /* If the instruction doesn't need an AVLReg and the SEW matches, consider - it compatible. */ - if (!demand_p (DEMAND_AVL)) - if (m_sew == curr_info.get_sew ()) - return true; - - return compatible_avl_p (curr_info) && compatible_vtype_p (curr_info); -} - -bool -vector_insn_info::available_p (const vector_insn_info &other) const -{ - return *this >= other; -} - -void -vector_insn_info::fuse_avl (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - set_insn (info1.get_insn ()); - if (info1.demand_p (DEMAND_AVL)) - { - if (info1.demand_p (DEMAND_NONZERO_AVL)) - { - if (info2.demand_p (DEMAND_AVL) - && !info2.demand_p (DEMAND_NONZERO_AVL)) - { - set_avl_info (info2.get_avl_info ()); - set_demand (DEMAND_AVL, true); - set_demand (DEMAND_NONZERO_AVL, false); - return; - } - } - set_avl_info (info1.get_avl_info ()); - set_demand (DEMAND_NONZERO_AVL, info1.demand_p (DEMAND_NONZERO_AVL)); - } - else - { - set_avl_info (info2.get_avl_info ()); - set_demand (DEMAND_NONZERO_AVL, info2.demand_p (DEMAND_NONZERO_AVL)); - } - set_demand (DEMAND_AVL, - info1.demand_p (DEMAND_AVL) || info2.demand_p (DEMAND_AVL)); -} - -void -vector_insn_info::fuse_sew_lmul (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - /* We need to fuse sew && lmul according to demand info: - - 1. GE_SEW. - 2. SEW. - 3. LMUL. - 4. RATIO. */ - if (same_sew_lmul_demand_p (info1.get_demands (), info2.get_demands ())) - { - set_demand (DEMAND_SEW, info2.demand_p (DEMAND_SEW)); - set_demand (DEMAND_LMUL, info2.demand_p (DEMAND_LMUL)); - set_demand (DEMAND_RATIO, info2.demand_p (DEMAND_RATIO)); - set_demand (DEMAND_GE_SEW, info2.demand_p (DEMAND_GE_SEW)); - set_sew (info2.get_sew ()); - set_vlmul (info2.get_vlmul ()); - set_ratio (info2.get_ratio ()); + /* We set it as unknown since we don't what will happen in CALL or ASM. */ + if (insn->is_call () || insn->is_asm ()) + { + set_unknown (); + return; + } + + /* If this is something that updates VL/VTYPE that we don't know about, set + the state to unknown. */ + if (!vector_config_insn_p (insn->rtl ()) && !has_vtype_op (insn->rtl ()) + && (find_access (insn->defs (), VL_REGNUM) + || find_access (insn->defs (), VTYPE_REGNUM))) + { + set_unknown (); + return; + } + + if (!vector_config_insn_p (insn->rtl ()) && !has_vtype_op (insn->rtl ())) + /* uninitialized */ return; - } - for (const auto &rule : fuse_rules) - { - if (rule.pair.match_cond_p (info1.get_demands (), info2.get_demands ())) - { - set_demand (DEMAND_SEW, rule.demand_sew_p); - set_demand (DEMAND_LMUL, rule.demand_lmul_p); - set_demand (DEMAND_RATIO, rule.demand_ratio_p); - set_demand (DEMAND_GE_SEW, rule.demand_ge_sew_p); - set_sew (rule.new_sew (info1, info2)); - set_vlmul (rule.new_vlmul (info1, info2)); - set_ratio (rule.new_ratio (info1, info2)); - return; - } - if (rule.pair.match_cond_p (info2.get_demands (), info1.get_demands ())) - { - set_demand (DEMAND_SEW, rule.demand_sew_p); - set_demand (DEMAND_LMUL, rule.demand_lmul_p); - set_demand (DEMAND_RATIO, rule.demand_ratio_p); - set_demand (DEMAND_GE_SEW, rule.demand_ge_sew_p); - set_sew (rule.new_sew (info2, info1)); - set_vlmul (rule.new_vlmul (info2, info1)); - set_ratio (rule.new_ratio (info2, info1)); - return; - } - } - gcc_unreachable (); -} -void -vector_insn_info::fuse_tail_policy (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - if (info1.demand_p (DEMAND_TAIL_POLICY)) - { - set_ta (info1.get_ta ()); - demand (DEMAND_TAIL_POLICY); - } - else if (info2.demand_p (DEMAND_TAIL_POLICY)) - { - set_ta (info2.get_ta ()); - demand (DEMAND_TAIL_POLICY); - } - else - set_ta (get_default_ta ()); -} - -void -vector_insn_info::fuse_mask_policy (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - if (info1.demand_p (DEMAND_MASK_POLICY)) - { - set_ma (info1.get_ma ()); - demand (DEMAND_MASK_POLICY); - } - else if (info2.demand_p (DEMAND_MASK_POLICY)) - { - set_ma (info2.get_ma ()); - demand (DEMAND_MASK_POLICY); - } - else - set_ma (get_default_ma ()); -} - -vector_insn_info -vector_insn_info::local_merge (const vector_insn_info &merge_info) const -{ - if (!vsetvl_insn_p (get_insn ()->rtl ()) && *this != merge_info) - gcc_assert (this->compatible_p (merge_info) - && "Can't merge incompatible demanded infos"); - - vector_insn_info new_info; - new_info.set_valid (); - /* For local backward data flow, we always update INSN && AVL as the - latest INSN and AVL so that we can keep track status of each INSN. */ - new_info.fuse_avl (merge_info, *this); - new_info.fuse_sew_lmul (*this, merge_info); - new_info.fuse_tail_policy (*this, merge_info); - new_info.fuse_mask_policy (*this, merge_info); - return new_info; -} - -vector_insn_info -vector_insn_info::global_merge (const vector_insn_info &merge_info, - unsigned int bb_index) const -{ - if (!vsetvl_insn_p (get_insn ()->rtl ()) && *this != merge_info) - gcc_assert (this->compatible_p (merge_info) - && "Can't merge incompatible demanded infos"); - - vector_insn_info new_info; - new_info.set_valid (); - - /* For global data flow, we should keep original INSN and AVL if they - valid since we should keep the life information of each block. - - For example: - bb 0 -> bb 1. - We should keep INSN && AVL of bb 1 since we will eventually emit - vsetvl instruction according to INSN and AVL of bb 1. */ - new_info.fuse_avl (*this, merge_info); - /* Recompute the AVL source whose block index is equal to BB_INDEX. */ - if (new_info.get_avl_source () - && new_info.get_avl_source ()->insn ()->is_phi () - && new_info.get_avl_source ()->bb ()->index () != bb_index) - { - hash_set sets - = get_all_sets (new_info.get_avl_source (), true, true, true); - new_info.set_avl_source (nullptr); - bool can_find_set_p = false; - set_info *first_set = nullptr; - for (set_info *set : sets) - { - if (!first_set) - first_set = set; - if (set->bb ()->index () == bb_index) - { - gcc_assert (!can_find_set_p); - new_info.set_avl_source (set); - can_find_set_p = true; - } - } - if (!can_find_set_p && sets.elements () == 1 - && first_set->insn ()->is_real ()) - new_info.set_avl_source (first_set); - } - - /* Make sure VLMAX AVL always has a set_info the get VL. */ - if (vlmax_avl_p (new_info.get_avl ())) - { - if (this->get_avl_source ()) - new_info.set_avl_source (this->get_avl_source ()); - else - { - gcc_assert (merge_info.get_avl_source ()); - new_info.set_avl_source (merge_info.get_avl_source ()); - } - } - - new_info.fuse_sew_lmul (*this, merge_info); - new_info.fuse_tail_policy (*this, merge_info); - new_info.fuse_mask_policy (*this, merge_info); - return new_info; -} - -/* Wrapper helps to return the AVL or VL operand for the - vector_insn_info. Return AVL if the AVL is not VLMAX. - Otherwise, return the VL operand. */ -rtx -vector_insn_info::get_avl_or_vl_reg (void) const -{ - gcc_assert (has_avl_reg ()); - if (!vlmax_avl_p (get_avl ())) - return get_avl (); - - rtx_insn *rinsn = get_insn ()->rtl (); - if (has_vl_op (rinsn) || vsetvl_insn_p (rinsn)) - { - rtx vl = ::get_vl (rinsn); - /* For VLMAX, we should make sure we get the - REG to emit 'vsetvl VL,zero' since the 'VL' - should be the REG according to RVV ISA. */ - if (REG_P (vl)) - return vl; - } - - /* We always has avl_source if it is VLMAX AVL. */ - gcc_assert (get_avl_source ()); - return get_avl_reg_rtx (); -} + set_valid (); + + m_avl = ::get_avl (insn->rtl ()); + if (m_avl) + { + if (vsetvl_insn_p (insn->rtl ()) || has_vlmax_avl ()) + m_vl = ::get_vl (insn->rtl ()); + + if (has_reg_avl ()) + m_avl_def = find_access (insn->uses (), REGNO (m_avl))->def (); + } + + m_sew = ::get_sew (insn->rtl ()); + m_vlmul = ::get_vlmul (insn->rtl ()); + m_ratio = get_attr_ratio (insn->rtl ()); + /* when get_attr_ratio is invalid, this kind of instructions + doesn't care about ratio. However, we still need this value + in demand info backward analysis. */ + if (m_ratio == INVALID_ATTRIBUTE) + m_ratio = calculate_ratio (m_sew, m_vlmul); + m_ta = tail_agnostic_p (insn->rtl ()); + m_ma = mask_agnostic_p (insn->rtl ()); + + /* If merge operand is undef value, we prefer agnostic. */ + int merge_op_idx = get_attr_merge_op_idx (insn->rtl ()); + if (merge_op_idx != INVALID_ATTRIBUTE + && satisfies_constraint_vu (recog_data.operand[merge_op_idx])) + { + m_ta = true; + m_ma = true; + } + + /* Determine the demand info of the RVV insn. */ + m_max_sew = get_max_int_sew (); + unsigned demand_flags = 0; + if (vector_config_insn_p (insn->rtl ())) + { + demand_flags |= demand_flags::DEMAND_AVL_P; + demand_flags |= demand_flags::DEMAND_RATIO_P; + } + else + { + if (has_vl_op (insn->rtl ())) + { + if (scalar_move_insn_p (insn->rtl ())) + { + /* If the avl for vmv.s.x comes from the vsetvl instruction, we + don't know if the avl is non-zero, so it is set to + DEMAND_AVL_P for now. it may be corrected to + DEMAND_NON_ZERO_AVL_P later when more information is + available. + */ + if (has_non_zero_avl ()) + demand_flags |= demand_flags::DEMAND_NON_ZERO_AVL_P; + else + demand_flags |= demand_flags::DEMAND_AVL_P; + } + else + demand_flags |= demand_flags::DEMAND_AVL_P; + } -bool -vector_insn_info::update_fault_first_load_avl (insn_info *insn) -{ - // Update AVL to vl-output of the fault first load. - const insn_info *read_vl = get_forward_read_vl_insn (insn); - if (read_vl) - { - rtx vl = SET_DEST (PATTERN (read_vl->rtl ())); - def_info *def = find_access (read_vl->defs (), REGNO (vl)); - set_info *set = safe_dyn_cast (def); - set_avl_info (avl_info (vl, set)); - set_insn (insn); - return true; - } - return false; -} + if (get_attr_ratio (insn->rtl ()) != INVALID_ATTRIBUTE) + demand_flags |= demand_flags::DEMAND_RATIO_P; + else + { + if (scalar_move_insn_p (insn->rtl ()) && m_ta) + { + demand_flags |= demand_flags::DEMAND_GE_SEW_P; + m_max_sew = get_attr_type (insn->rtl ()) == TYPE_VFMOVFV + ? get_max_float_sew () + : get_max_int_sew (); + } + else + demand_flags |= demand_flags::DEMAND_SEW_P; + + if (!ignore_vlmul_insn_p (insn->rtl ())) + demand_flags |= demand_flags::DEMAND_LMUL_P; + } -static const char * -vlmul_to_str (vlmul_type vlmul) -{ - switch (vlmul) - { - case LMUL_1: - return "m1"; - case LMUL_2: - return "m2"; - case LMUL_4: - return "m4"; - case LMUL_8: - return "m8"; - case LMUL_RESERVED: - return "INVALID LMUL"; - case LMUL_F8: - return "mf8"; - case LMUL_F4: - return "mf4"; - case LMUL_F2: - return "mf2"; + if (!m_ta) + demand_flags |= demand_flags::DEMAND_TAIL_POLICY_P; + if (!m_ma) + demand_flags |= demand_flags::DEMAND_MASK_POLICY_P; + } + + normalize_demand (demand_flags); + + /* Optimize AVL from the vsetvl instruction. */ + insn_info *def_insn = extract_single_source (get_avl_def ()); + if (def_insn && vsetvl_insn_p (def_insn->rtl ())) + { + vsetvl_info def_info = vsetvl_info (def_insn); + if ((scalar_move_insn_p (insn->rtl ()) + || def_info.get_ratio () == get_ratio ()) + && (def_info.has_vlmax_avl () || def_info.has_imm_avl ())) + { + update_avl (def_info); + if (scalar_move_insn_p (insn->rtl ()) && has_non_zero_avl ()) + m_avl_demand = avl_demand_type::non_zero_avl; + } + } + + /* Determine if dest operand(vl) has been used by non-RVV instructions. */ + if (has_reg_vl ()) + { + const hash_set vl_uses + = get_all_real_uses (get_insn (), REGNO (get_vl ())); + for (use_info *use : vl_uses) + { + gcc_assert (use->insn ()->is_real ()); + rtx_insn *rinsn = use->insn ()->rtl (); + if (!has_vl_op (rinsn) + || count_regno_occurrences (rinsn, REGNO (get_vl ())) != 1) + { + use_by_non_rvv_insn = true; + break; + } + rtx avl = ::get_avl (rinsn); + if (!avl || REGNO (get_vl ()) != REGNO (avl)) + { + use_by_non_rvv_insn = true; + break; + } + } + } - default: + /* Collect the read vl insn for the fault-only-first rvv loads. */ + if (fault_first_load_p (insn->rtl ())) + { + for (insn_info *i = insn->next_nondebug_insn (); + i->bb () == insn->bb (); i = i->next_nondebug_insn ()) + { + if (find_access (i->defs (), VL_REGNUM)) + break; + if (i->rtl () && read_vl_insn_p (i->rtl ())) + { + m_read_vl_insn = i; + break; + } + } + } + } + + bool operator== (const vsetvl_info &other) const + { + gcc_assert (!uninit_p () && !other.uninit_p () + && "Uninitialization should not happen"); + + if (empty_p ()) + return other.empty_p (); + if (unknown_p ()) + return other.unknown_p (); + + return get_insn () == other.get_insn () && get_bb () == other.get_bb () + && get_avl () == other.get_avl () && get_vl () == other.get_vl () + && get_avl_def () == other.get_avl_def () + && get_sew () == other.get_sew () + && get_vlmul () == other.get_vlmul () && get_ta () == other.get_ta () + && get_ma () == other.get_ma () + && get_avl_demand () == other.get_avl_demand () + && get_sew_lmul_demand () == other.get_sew_lmul_demand () + && get_policy_demand () == other.get_policy_demand (); + } + + void dump (FILE *file, const char *indent = "") const + { + if (uninit_p ()) + { + fprintf (file, "UNINITIALIZED.\n"); + return; + } + else if (unknown_p ()) + { + fprintf (file, "UNKNOWN.\n"); + return; + } + else if (empty_p ()) + { + fprintf (file, "EMPTY.\n"); + return; + } + else if (valid_p ()) + fprintf (file, "VALID (insn %u, bb %u)%s\n", get_insn ()->uid (), + get_bb ()->index (), ignore_p () ? " (ignore)" : ""); + else gcc_unreachable (); - } -} - -static const char * -policy_to_str (bool agnostic_p) -{ - return agnostic_p ? "agnostic" : "undisturbed"; -} -void -vector_insn_info::dump (FILE *file) const -{ - fprintf (file, "["); - if (uninit_p ()) - fprintf (file, "UNINITIALIZED,"); - else if (valid_p ()) - fprintf (file, "VALID,"); - else if (unknown_p ()) - fprintf (file, "UNKNOWN,"); - else if (empty_p ()) - fprintf (file, "EMPTY,"); - else - fprintf (file, "DIRTY,"); - - fprintf (file, "Demand field={%d(VL),", demand_p (DEMAND_AVL)); - fprintf (file, "%d(DEMAND_NONZERO_AVL),", demand_p (DEMAND_NONZERO_AVL)); - fprintf (file, "%d(SEW),", demand_p (DEMAND_SEW)); - fprintf (file, "%d(DEMAND_GE_SEW),", demand_p (DEMAND_GE_SEW)); - fprintf (file, "%d(LMUL),", demand_p (DEMAND_LMUL)); - fprintf (file, "%d(RATIO),", demand_p (DEMAND_RATIO)); - fprintf (file, "%d(TAIL_POLICY),", demand_p (DEMAND_TAIL_POLICY)); - fprintf (file, "%d(MASK_POLICY)}\n", demand_p (DEMAND_MASK_POLICY)); - - fprintf (file, "AVL="); - print_rtl_single (file, get_avl ()); - fprintf (file, "SEW=%d,", get_sew ()); - fprintf (file, "VLMUL=%s,", vlmul_to_str (get_vlmul ())); - fprintf (file, "RATIO=%d,", get_ratio ()); - fprintf (file, "TAIL_POLICY=%s,", policy_to_str (get_ta ())); - fprintf (file, "MASK_POLICY=%s", policy_to_str (get_ma ())); - fprintf (file, "]\n"); - - if (valid_p ()) - { - if (get_insn ()) - { - fprintf (file, "The real INSN="); - print_rtl_single (file, get_insn ()->rtl ()); - } - } -} + fprintf (file, "%sDemand fields:", indent); + if (m_sew_lmul_demand == sew_lmul_demand_type::sew_lmul) + fprintf (file, " demand_sew_lmul"); + else if (m_sew_lmul_demand == sew_lmul_demand_type::ratio_only) + fprintf (file, " demand_ratio_only"); + else if (m_sew_lmul_demand == sew_lmul_demand_type::sew_only) + fprintf (file, " demand_sew_only"); + else if (m_sew_lmul_demand == sew_lmul_demand_type::ge_sew) + fprintf (file, " demand_ge_sew"); + else if (m_sew_lmul_demand == sew_lmul_demand_type::ratio_and_ge_sew) + fprintf (file, " demand_ratio_and_ge_sew"); + + if (m_policy_demand == policy_demand_type::tail_mask_policy) + fprintf (file, " demand_tail_mask_policy"); + else if (m_policy_demand == policy_demand_type::tail_policy_only) + fprintf (file, " demand_tail_policy_only"); + else if (m_policy_demand == policy_demand_type::mask_policy_only) + fprintf (file, " demand_mask_policy_only"); + + if (m_avl_demand == avl_demand_type::avl) + fprintf (file, " demand_avl"); + else if (m_avl_demand == avl_demand_type::non_zero_avl) + fprintf (file, " demand_non_zero_avl"); + fprintf (file, "\n"); + + fprintf (file, "%sSEW=%d, ", indent, get_sew ()); + fprintf (file, "VLMUL=%s, ", vlmul_to_str (get_vlmul ())); + fprintf (file, "RATIO=%d, ", get_ratio ()); + fprintf (file, "MAX_SEW=%d\n", get_max_sew ()); + + fprintf (file, "%sTAIL_POLICY=%s, ", indent, policy_to_str (get_ta ())); + fprintf (file, "MASK_POLICY=%s\n", policy_to_str (get_ma ())); + + fprintf (file, "%sAVL=", indent); + print_rtl_single (file, get_avl ()); + fprintf (file, "%sVL=", indent); + print_rtl_single (file, get_vl ()); + if (change_vtype_only_p ()) + fprintf (file, "%schange vtype only\n", indent); + if (get_read_vl_insn ()) + fprintf (file, "%sread_vl_insn: insn %u\n", indent, + get_read_vl_insn ()->uid ()); + if (use_by_non_rvv_insn_p ()) + fprintf (file, "%suse_by_non_rvv_insn=true\n", indent); + } +}; vector_infos_manager::vector_infos_manager () { diff --git a/gcc/config/riscv/riscv-vsetvl.h b/gcc/config/riscv/riscv-vsetvl.h index 53549abfac5..9eab276190e 100644 --- a/gcc/config/riscv/riscv-vsetvl.h +++ b/gcc/config/riscv/riscv-vsetvl.h @@ -82,267 +82,6 @@ enum def_type CLOBBER_DEF = 1 << 4 }; -/* AVL info for RVV instruction. Most RVV instructions have AVL operand in - implicit dependency. The AVL comparison between 2 RVV instructions is - very important since it affects our decision whether we should insert - a vsetvl instruction in this situation. AVL operand of all RVV instructions - can only be either a const_int value with < 32 or a reg value which can be - define by either a real RTL instruction or a PHI instruction. So we need a - standalone method to define AVL comparison and we can not simpily use - operator "==" to compare 2 RTX value since it's to strict which will make - use miss a lot of optimization opportunities. This method handle these - following cases: - - - Background: - Insert-vsetvl PASS is working after RA. - - - Terminology: - - pr: Pseudo-register. - - hr: Hardware-register. - - - Case 1: - - Before RA: - li pr138,13 - insn1 (implicit depend on pr138). - li pr138,14 - insn2 (implicit depend on pr139). - - After RA: - li hr5,13 - insn1 (implicit depend on hr5). - li hr5,14 - insn2 (implicit depend on hr5). - - Correct IR after vsetvl PASS: - li hr5,13 - vsetvl1 zero,hr5.... - insn1 (implicit depend on hr5). - li hr5,14 - vsetvl2 zero,hr5.... - insn2 (implicit depend on hr5). - - In this case, both insn1 and insn2 are using hr5 as the same AVL. - If we use "rtx_equal_p" or "REGNO (AVL1) == REGNO (AVL)", we will end - up with missing the vsetvl2 instruction which creates wrong result. - - Note: Using "==" operator to compare 2 AVL RTX strictly can fix this - issue. However, it is a too strict comparison method since not all member - variables in RTX data structure are not neccessary to be the same. It will - make us miss a lot of optimization opportunities. - - - Case 2: - - After RA: - bb 0: - li hr5,13 - bb 1: - li hr5,14 - bb2: - insn1 (implicit depend on hr5). - insn2 (implicit depend on hr5). - - In this case, we may end up with different AVL RTX and produce redundant - vsetvl instruction. - - VALUE is the implicit dependency in each RVV instruction. - SOURCE is the source definition information of AVL operand. */ -class avl_info -{ -private: - rtx m_value; - rtl_ssa::set_info *m_source; - -public: - avl_info () : m_value (NULL_RTX), m_source (nullptr) {} - avl_info (const avl_info &); - avl_info (rtx, rtl_ssa::set_info *); - rtx get_value () const { return m_value; } - rtl_ssa::set_info *get_source () const { return m_source; } - void set_source (rtl_ssa::set_info *set) { m_source = set; } - bool single_source_equal_p (const avl_info &) const; - bool multiple_source_equal_p (const avl_info &) const; - avl_info &operator= (const avl_info &); - bool operator== (const avl_info &) const; - bool operator!= (const avl_info &) const; - - bool has_avl_imm () const - { - return get_value () && CONST_INT_P (get_value ()); - } - bool has_avl_reg () const { return get_value () && REG_P (get_value ()); } - bool has_avl_no_reg () const { return !get_value (); } - bool has_non_zero_avl () const; - bool has_avl () const { return get_value (); } -}; - -/* Basic structure to save VL/VTYPE information. */ -struct vl_vtype_info -{ -protected: - /* AVL can be either register or const_int. */ - avl_info m_avl; - /* Fields from VTYPE. The VTYPE checking depend on the flag - dem_* before. */ - uint8_t m_sew; - riscv_vector::vlmul_type m_vlmul; - uint8_t m_ratio; - bool m_ta; - bool m_ma; - -public: - void set_sew (uint8_t sew) { m_sew = sew; } - void set_vlmul (riscv_vector::vlmul_type vlmul) { m_vlmul = vlmul; } - void set_ratio (uint8_t ratio) { m_ratio = ratio; } - void set_ta (bool ta) { m_ta = ta; } - void set_ma (bool ma) { m_ma = ma; } - - vl_vtype_info () - : m_avl (avl_info ()), m_sew (0), m_vlmul (riscv_vector::LMUL_RESERVED), - m_ratio (0), m_ta (0), m_ma (0) - {} - vl_vtype_info (const vl_vtype_info &) = default; - vl_vtype_info &operator= (const vl_vtype_info &) = default; - vl_vtype_info (avl_info, uint8_t, riscv_vector::vlmul_type, uint8_t, bool, - bool); - - bool operator== (const vl_vtype_info &) const; - bool operator!= (const vl_vtype_info &) const; - - bool has_avl_imm () const { return m_avl.has_avl_imm (); } - bool has_avl_reg () const { return m_avl.has_avl_reg (); } - bool has_avl_no_reg () const { return m_avl.has_avl_no_reg (); } - bool has_non_zero_avl () const { return m_avl.has_non_zero_avl (); }; - bool has_avl () const { return m_avl.has_avl (); } - - rtx get_avl () const { return m_avl.get_value (); } - const avl_info &get_avl_info () const { return m_avl; } - rtl_ssa::set_info *get_avl_source () const { return m_avl.get_source (); } - void set_avl_source (rtl_ssa::set_info *set) { m_avl.set_source (set); } - void set_avl_info (const avl_info &avl) { m_avl = avl; } - uint8_t get_sew () const { return m_sew; } - riscv_vector::vlmul_type get_vlmul () const { return m_vlmul; } - uint8_t get_ratio () const { return m_ratio; } - bool get_ta () const { return m_ta; } - bool get_ma () const { return m_ma; } - - bool same_avl_p (const vl_vtype_info &) const; - bool same_vtype_p (const vl_vtype_info &) const; - bool same_vlmax_p (const vl_vtype_info &) const; -}; - -class vector_insn_info : public vl_vtype_info -{ -private: - enum state_type - { - UNINITIALIZED, - VALID, - UNKNOWN, - EMPTY, - - /* The block is polluted as containing VSETVL instruction during dem - backward propagation to gain better LCM optimization even though - such VSETVL instruction is not really emit yet during this time. */ - DIRTY, - }; - - enum state_type m_state; - - bool m_demands[NUM_DEMAND]; - - /* TODO: Assume INSN1 = INSN holding of definition of AVL. - INSN2 = INSN that is inserted a vsetvl insn before. - We may need to add a new member to save INSN of holding AVL. - m_insn is holding the INSN that is inserted a vsetvl insn before in - Phase 2. Ideally, most of the time INSN1 == INSN2. However, considering - such case: - - vmv.x.s (INSN2) - vle8.v (INSN1) - - If these 2 instructions are compatible, we should only issue a vsetvl INSN - (with AVL included) before vmv.x.s, but vmv.x.s is not the INSN holding the - definition of AVL. */ - rtl_ssa::insn_info *m_insn; - - friend class vector_infos_manager; - -public: - vector_insn_info () - : vl_vtype_info (), m_state (UNINITIALIZED), m_demands{false}, - m_insn (nullptr) - {} - - /* Parse the instruction to get VL/VTYPE information and demanding - * information. */ - /* This is only called by simple_vsetvl subroutine when optimize == 0. - Since RTL_SSA can not be enabled when optimize == 0, we don't initialize - the m_insn. */ - void parse_insn (rtx_insn *); - /* This is only called by lazy_vsetvl subroutine when optimize > 0. - We use RTL_SSA framework to initialize the insn_info. */ - void parse_insn (rtl_ssa::insn_info *); - - bool operator>= (const vector_insn_info &) const; - bool operator== (const vector_insn_info &) const; - - bool uninit_p () const { return m_state == UNINITIALIZED; } - bool valid_p () const { return m_state == VALID; } - bool unknown_p () const { return m_state == UNKNOWN; } - bool empty_p () const { return m_state == EMPTY; } - bool dirty_p () const { return m_state == DIRTY; } - bool valid_or_dirty_p () const - { - return m_state == VALID || m_state == DIRTY; - } - bool available_p (const vector_insn_info &) const; - - static vector_insn_info get_unknown () - { - vector_insn_info info; - info.set_unknown (); - return info; - } - - void set_valid () { m_state = VALID; } - void set_unknown () { m_state = UNKNOWN; } - void set_empty () { m_state = EMPTY; } - void set_dirty () { m_state = DIRTY; } - void set_insn (rtl_ssa::insn_info *insn) { m_insn = insn; } - - bool demand_p (enum demand_type type) const { return m_demands[type]; } - void demand (enum demand_type type) { m_demands[type] = true; } - void set_demand (enum demand_type type, bool value) - { - m_demands[type] = value; - } - void fuse_avl (const vector_insn_info &, const vector_insn_info &); - void fuse_sew_lmul (const vector_insn_info &, const vector_insn_info &); - void fuse_tail_policy (const vector_insn_info &, const vector_insn_info &); - void fuse_mask_policy (const vector_insn_info &, const vector_insn_info &); - - bool compatible_p (const vector_insn_info &) const; - bool skip_avl_compatible_p (const vector_insn_info &) const; - bool compatible_avl_p (const vl_vtype_info &) const; - bool compatible_avl_p (const avl_info &) const; - bool compatible_vtype_p (const vl_vtype_info &) const; - bool compatible_p (const vl_vtype_info &) const; - vector_insn_info local_merge (const vector_insn_info &) const; - vector_insn_info global_merge (const vector_insn_info &, unsigned int) const; - - rtl_ssa::insn_info *get_insn () const { return m_insn; } - const bool *get_demands (void) const { return m_demands; } - rtx get_avl_or_vl_reg (void) const; - rtx get_avl_reg_rtx (void) const - { - return gen_rtx_REG (Pmode, get_avl_source ()->regno ()); - } - bool update_fault_first_load_avl (rtl_ssa::insn_info *); - - void dump (FILE *) const; -}; - struct vector_block_info { /* The local_dem vector insn_info of the block. */ From patchwork Tue Oct 17 11:34:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lehua Ding X-Patchwork-Id: 1849975 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8sQR09QNz20Pd for ; Tue, 17 Oct 2023 22:37:02 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7E5443853533 for ; Tue, 17 Oct 2023 11:36:58 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgbr1.qq.com (smtpbgbr1.qq.com [54.207.19.206]) by sourceware.org (Postfix) with ESMTPS id 55F073858020 for ; Tue, 17 Oct 2023 11:35:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 55F073858020 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 55F073858020 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=54.207.19.206 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697542526; cv=none; b=nF6jZXlF7IqiWvYUELAjJSeFHiCUOKbHingceGvw9zaemTB+8HC49tqyN8xMyXAFXYWtDa4MwabJOabu1PBiYPqsTYL9P+tpm2MNIerTxcg5jyPUk00JGcqFDoBPN4Pql/u1rbxcy3qMZ1cC5eZIAfsv4L0icuniX0wgHX/XTXE= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697542526; c=relaxed/simple; bh=Cpen3EVH7accaNWTN4EJGCxo0SM1SgYB84NnhGyta7k=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=i+AmjvoD0KXEiaVtBdKwtXBPDnin5dNcjPLoE6K9DNzb781NsNcRYWuLhvJYXxiGpA80Yj3jgl4h5KNwegfEmdWZ+41ZNbwwX9+icHdPT6NLDTcz6Iz/zGP2GslVUx44NIgJ81JnYLfuoePfdNLSCek9HnhB3A/DVZjx5me6D4U= ARC-Authentication-Results: i=1; server2.sourceware.org X-QQ-mid: bizesmtp62t1697542511trkmy8u9 Received: from rios-cad121.hadoop.rioslab.org ( [58.60.1.9]) by bizesmtp.qq.com (ESMTP) with id ; Tue, 17 Oct 2023 19:35:10 +0800 (CST) X-QQ-SSF: 01400000000000C0F000000A0000000 X-QQ-FEAT: JfUuUofuJFp9CBQw3XswvIKgSUqJfAnUB9u9o51X+cA6V9CBvvwqxEyAPUy+I opf2GNy1UIoTRJrLROnGoOFzbzJGuoIW4w3ioDzHRMwCiNbKPRNleIk3X+5tesI4XDFKwfT tvC/RurzFMIByW0mXGXF7z3fQhvbXn+GZjAA6arIg0C/Volj225uyknN1E6HMSQ60av4Wbv mZpTTndL6zvr4Rhsjuxzipr9AJrPa1bQbuJDgYqELxrQ67U8VShla5MN1MbOoZLpSSZV8Jd Mb62pC9S6J8MmUWO9e5ZOszrGrwl3o+MoiMCIjJaogizSZDg4ieGGOVcMopuaSesPkfByZf elCnMeksKZOAHBRktava6t4FbzBQUtkt/r96FfDoTO3q8cfHX9AnXW2s/uBUT4nMHPd9DPB X-QQ-GoodBg: 2 X-BIZMAIL-ID: 945746313763580012 From: Lehua Ding To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, rdapp.gcc@gmail.com, palmer@rivosinc.com, jeffreyalaw@gmail.com, lehua.ding@rivai.ai Subject: [PATCH V2 02/14] RISC-V: P2: Refactor and cleanup demand system Date: Tue, 17 Oct 2023 19:34:48 +0800 Message-Id: <20231017113500.1160997-3-lehua.ding@rivai.ai> X-Mailer: git-send-email 2.36.3 In-Reply-To: <20231017113500.1160997-1-lehua.ding@rivai.ai> References: <20231017113500.1160997-1-lehua.ding@rivai.ai> MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz6a-0 X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, KAM_SHORT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org This sub-patch refactor the demand system. I split the demand information into three parts. They are sew and lmul related (sew_lmul_demand_type), tail and mask policy related (policy_demand_type) and avl related (avl_demand_type). Then we define three interfaces avaiable_with, compatible_with and merge_with. avaiable_with is used to determine whether the two vsetvl infos prev_info and next_info are available or not. If prev_info is available for next_info, it means that the RVV insn corresponding to next_info on the path from prev_info to next_info can be used without inserting a separate vsetvl instruction. compatible_with is used to determine whether prev_info is compatible with next_info, and if so, merge_with can be used to merge the stricter demand information from next_info into prev_info so that prev_info becomes available to next_info. gcc/ChangeLog: * config/riscv/riscv-vsetvl.cc (incompatible_avl_p): Removed. (different_sew_p): Removed. (different_lmul_p): Removed. (different_ratio_p): Removed. (different_tail_policy_p): Removed. (different_mask_policy_p): Removed. (possible_zero_avl_p): Removed. (second_ratio_invalid_for_first_sew_p): Removed. (second_ratio_invalid_for_first_lmul_p): Removed. (float_insn_valid_sew_p): Removed. (second_sew_less_than_first_sew_p): Removed. (first_sew_less_than_second_sew_p): Removed. (compare_lmul): Removed. (second_lmul_less_than_first_lmul_p): Removed. (second_ratio_less_than_first_ratio_p): Removed. (DEF_INCOMPATIBLE_COND): Removed. (greatest_sew): Removed. (first_sew): Removed. (second_sew): Removed. (first_vlmul): Removed. (second_vlmul): Removed. (first_ratio): Removed. (second_ratio): Removed. (vlmul_for_first_sew_second_ratio): Removed. (vlmul_for_greatest_sew_second_ratio): Removed. (ratio_for_second_sew_first_vlmul): Removed. (DEF_SEW_LMUL_FUSE_RULE): Removed. (always_unavailable): Removed. (avl_unavailable_p): Removed. (sew_unavailable_p): Removed. (lmul_unavailable_p): Removed. (ge_sew_unavailable_p): Removed. (ge_sew_lmul_unavailable_p): Removed. (ge_sew_ratio_unavailable_p): Removed. (DEF_UNAVAILABLE_COND): Removed. (same_sew_lmul_demand_p): Removed. (propagate_avl_across_demands_p): Removed. (reg_available_p): Removed. (support_relaxed_compatible_p): Removed. (class demand_system): New class. (DEF_SEW_LMUL_RULE): New Marco. (DEF_POLICY_RULE): New macro. (DEF_AVL_RULE): New macro. * config/riscv/riscv-vsetvl.def (DEF_INCOMPATIBLE_COND): Removed. (DEF_SEW_LMUL_RULE): New macro. (DEF_SEW_LMUL_FUSE_RULE): Removed. (DEF_POLICY_RULE): New macro. (DEF_UNAVAILABLE_COND): Removed. (DEF_AVL_RULE): New macro. (sew_lmul): New demand type. (ratio_only): New demand type. (sew_only): New demand type. (ge_sew): New demand type. (ratio_and_ge_sew): New demand type. (tail_mask_policy): New demand type. (tail_policy_only): New demand type. (mask_policy_only): New demand type. (ignore_policy): New demand type. (avl): New demand type. (non_zero_avl): New demand type. (ignore_avl): New demand type. * config/riscv/riscv-vsetvl.h (enum demand_type): Removed. (enum demand_status): Removed. (enum fusion_type): Removed. (struct demands_pair): Removed. (struct demands_cond): Removed. (struct demands_fuse_rule): Removed. --- gcc/config/riscv/riscv-vsetvl.cc | 1062 ++++++++++++++++++----------- gcc/config/riscv/riscv-vsetvl.def | 634 ++++------------- gcc/config/riscv/riscv-vsetvl.h | 79 --- 3 files changed, 814 insertions(+), 961 deletions(-) -- 2.36.3 diff --git a/gcc/config/riscv/riscv-vsetvl.cc b/gcc/config/riscv/riscv-vsetvl.cc index 79ba8466556..be40b6fdf4c 100644 --- a/gcc/config/riscv/riscv-vsetvl.cc +++ b/gcc/config/riscv/riscv-vsetvl.cc @@ -1091,401 +1091,6 @@ calculate_vlmul (unsigned int sew, unsigned int ratio) return LMUL_RESERVED; } -static bool -incompatible_avl_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return !info1.compatible_avl_p (info2) && !info2.compatible_avl_p (info1); -} - -static bool -different_sew_p (const vector_insn_info &info1, const vector_insn_info &info2) -{ - return info1.get_sew () != info2.get_sew (); -} - -static bool -different_lmul_p (const vector_insn_info &info1, const vector_insn_info &info2) -{ - return info1.get_vlmul () != info2.get_vlmul (); -} - -static bool -different_ratio_p (const vector_insn_info &info1, const vector_insn_info &info2) -{ - return info1.get_ratio () != info2.get_ratio (); -} - -static bool -different_tail_policy_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return info1.get_ta () != info2.get_ta (); -} - -static bool -different_mask_policy_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return info1.get_ma () != info2.get_ma (); -} - -static bool -possible_zero_avl_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return !info1.has_non_zero_avl () || !info2.has_non_zero_avl (); -} - -static bool -second_ratio_invalid_for_first_sew_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return calculate_vlmul (info1.get_sew (), info2.get_ratio ()) - == LMUL_RESERVED; -} - -static bool -second_ratio_invalid_for_first_lmul_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return calculate_sew (info1.get_vlmul (), info2.get_ratio ()) == 0; -} - -static bool -float_insn_valid_sew_p (const vector_insn_info &info, unsigned int sew) -{ - if (info.get_insn () && info.get_insn ()->is_real () - && get_attr_type (info.get_insn ()->rtl ()) == TYPE_VFMOVFV) - { - if (sew == 16) - return TARGET_VECTOR_ELEN_FP_16; - else if (sew == 32) - return TARGET_VECTOR_ELEN_FP_32; - else if (sew == 64) - return TARGET_VECTOR_ELEN_FP_64; - } - return true; -} - -static bool -second_sew_less_than_first_sew_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return info2.get_sew () < info1.get_sew () - || !float_insn_valid_sew_p (info1, info2.get_sew ()); -} - -static bool -first_sew_less_than_second_sew_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return info1.get_sew () < info2.get_sew () - || !float_insn_valid_sew_p (info2, info1.get_sew ()); -} - -/* return 0 if LMUL1 == LMUL2. - return -1 if LMUL1 < LMUL2. - return 1 if LMUL1 > LMUL2. */ -static int -compare_lmul (vlmul_type vlmul1, vlmul_type vlmul2) -{ - if (vlmul1 == vlmul2) - return 0; - - switch (vlmul1) - { - case LMUL_1: - if (vlmul2 == LMUL_2 || vlmul2 == LMUL_4 || vlmul2 == LMUL_8) - return 1; - else - return -1; - case LMUL_2: - if (vlmul2 == LMUL_4 || vlmul2 == LMUL_8) - return 1; - else - return -1; - case LMUL_4: - if (vlmul2 == LMUL_8) - return 1; - else - return -1; - case LMUL_8: - return -1; - case LMUL_F2: - if (vlmul2 == LMUL_1 || vlmul2 == LMUL_2 || vlmul2 == LMUL_4 - || vlmul2 == LMUL_8) - return 1; - else - return -1; - case LMUL_F4: - if (vlmul2 == LMUL_F2 || vlmul2 == LMUL_1 || vlmul2 == LMUL_2 - || vlmul2 == LMUL_4 || vlmul2 == LMUL_8) - return 1; - else - return -1; - case LMUL_F8: - return 0; - default: - gcc_unreachable (); - } -} - -static bool -second_lmul_less_than_first_lmul_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return compare_lmul (info2.get_vlmul (), info1.get_vlmul ()) == -1; -} - -static bool -second_ratio_less_than_first_ratio_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return info2.get_ratio () < info1.get_ratio (); -} - -static CONSTEXPR const demands_cond incompatible_conds[] = { -#define DEF_INCOMPATIBLE_COND(AVL1, SEW1, LMUL1, RATIO1, NONZERO_AVL1, \ - GE_SEW1, TAIL_POLICTY1, MASK_POLICY1, AVL2, \ - SEW2, LMUL2, RATIO2, NONZERO_AVL2, GE_SEW2, \ - TAIL_POLICTY2, MASK_POLICY2, COND) \ - {{{AVL1, SEW1, LMUL1, RATIO1, NONZERO_AVL1, GE_SEW1, TAIL_POLICTY1, \ - MASK_POLICY1}, \ - {AVL2, SEW2, LMUL2, RATIO2, NONZERO_AVL2, GE_SEW2, TAIL_POLICTY2, \ - MASK_POLICY2}}, \ - COND}, -#include "riscv-vsetvl.def" -}; - -static unsigned -greatest_sew (const vector_insn_info &info1, const vector_insn_info &info2) -{ - return std::max (info1.get_sew (), info2.get_sew ()); -} - -static unsigned -first_sew (const vector_insn_info &info1, const vector_insn_info &) -{ - return info1.get_sew (); -} - -static unsigned -second_sew (const vector_insn_info &, const vector_insn_info &info2) -{ - return info2.get_sew (); -} - -static vlmul_type -first_vlmul (const vector_insn_info &info1, const vector_insn_info &) -{ - return info1.get_vlmul (); -} - -static vlmul_type -second_vlmul (const vector_insn_info &, const vector_insn_info &info2) -{ - return info2.get_vlmul (); -} - -static unsigned -first_ratio (const vector_insn_info &info1, const vector_insn_info &) -{ - return info1.get_ratio (); -} - -static unsigned -second_ratio (const vector_insn_info &, const vector_insn_info &info2) -{ - return info2.get_ratio (); -} - -static vlmul_type -vlmul_for_first_sew_second_ratio (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return calculate_vlmul (info1.get_sew (), info2.get_ratio ()); -} - -static vlmul_type -vlmul_for_greatest_sew_second_ratio (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return calculate_vlmul (MAX (info1.get_sew (), info2.get_sew ()), - info2.get_ratio ()); -} - -static unsigned -ratio_for_second_sew_first_vlmul (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - return calculate_ratio (info2.get_sew (), info1.get_vlmul ()); -} - -static CONSTEXPR const demands_fuse_rule fuse_rules[] = { -#define DEF_SEW_LMUL_FUSE_RULE(DEMAND_SEW1, DEMAND_LMUL1, DEMAND_RATIO1, \ - DEMAND_GE_SEW1, DEMAND_SEW2, DEMAND_LMUL2, \ - DEMAND_RATIO2, DEMAND_GE_SEW2, NEW_DEMAND_SEW, \ - NEW_DEMAND_LMUL, NEW_DEMAND_RATIO, \ - NEW_DEMAND_GE_SEW, NEW_SEW, NEW_VLMUL, \ - NEW_RATIO) \ - {{{DEMAND_ANY, DEMAND_SEW1, DEMAND_LMUL1, DEMAND_RATIO1, DEMAND_ANY, \ - DEMAND_GE_SEW1, DEMAND_ANY, DEMAND_ANY}, \ - {DEMAND_ANY, DEMAND_SEW2, DEMAND_LMUL2, DEMAND_RATIO2, DEMAND_ANY, \ - DEMAND_GE_SEW2, DEMAND_ANY, DEMAND_ANY}}, \ - NEW_DEMAND_SEW, \ - NEW_DEMAND_LMUL, \ - NEW_DEMAND_RATIO, \ - NEW_DEMAND_GE_SEW, \ - NEW_SEW, \ - NEW_VLMUL, \ - NEW_RATIO}, -#include "riscv-vsetvl.def" -}; - -static bool -always_unavailable (const vector_insn_info &, const vector_insn_info &) -{ - return true; -} - -static bool -avl_unavailable_p (const vector_insn_info &info1, const vector_insn_info &info2) -{ - return !info2.compatible_avl_p (info1.get_avl_info ()); -} - -static bool -sew_unavailable_p (const vector_insn_info &info1, const vector_insn_info &info2) -{ - if (!info2.demand_p (DEMAND_LMUL) && !info2.demand_p (DEMAND_RATIO)) - { - if (info2.demand_p (DEMAND_GE_SEW)) - return info1.get_sew () < info2.get_sew (); - return info1.get_sew () != info2.get_sew (); - } - return true; -} - -static bool -lmul_unavailable_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - if (info1.get_vlmul () == info2.get_vlmul () && !info2.demand_p (DEMAND_SEW) - && !info2.demand_p (DEMAND_RATIO)) - return false; - return true; -} - -static bool -ge_sew_unavailable_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - if (!info2.demand_p (DEMAND_LMUL) && !info2.demand_p (DEMAND_RATIO) - && info2.demand_p (DEMAND_GE_SEW)) - return info1.get_sew () < info2.get_sew (); - return true; -} - -static bool -ge_sew_lmul_unavailable_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - if (!info2.demand_p (DEMAND_RATIO) && info2.demand_p (DEMAND_GE_SEW)) - return info1.get_sew () < info2.get_sew (); - return true; -} - -static bool -ge_sew_ratio_unavailable_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - if (!info2.demand_p (DEMAND_LMUL)) - { - if (info2.demand_p (DEMAND_GE_SEW)) - return info1.get_sew () < info2.get_sew (); - /* Demand GE_SEW should be available for non-demand SEW. */ - else if (!info2.demand_p (DEMAND_SEW)) - return false; - } - return true; -} - -static CONSTEXPR const demands_cond unavailable_conds[] = { -#define DEF_UNAVAILABLE_COND(AVL1, SEW1, LMUL1, RATIO1, NONZERO_AVL1, GE_SEW1, \ - TAIL_POLICTY1, MASK_POLICY1, AVL2, SEW2, LMUL2, \ - RATIO2, NONZERO_AVL2, GE_SEW2, TAIL_POLICTY2, \ - MASK_POLICY2, COND) \ - {{{AVL1, SEW1, LMUL1, RATIO1, NONZERO_AVL1, GE_SEW1, TAIL_POLICTY1, \ - MASK_POLICY1}, \ - {AVL2, SEW2, LMUL2, RATIO2, NONZERO_AVL2, GE_SEW2, TAIL_POLICTY2, \ - MASK_POLICY2}}, \ - COND}, -#include "riscv-vsetvl.def" -}; - -static bool -same_sew_lmul_demand_p (const bool *dems1, const bool *dems2) -{ - return dems1[DEMAND_SEW] == dems2[DEMAND_SEW] - && dems1[DEMAND_LMUL] == dems2[DEMAND_LMUL] - && dems1[DEMAND_RATIO] == dems2[DEMAND_RATIO] && !dems1[DEMAND_GE_SEW] - && !dems2[DEMAND_GE_SEW]; -} - -static bool -propagate_avl_across_demands_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - if (info2.demand_p (DEMAND_AVL)) - { - if (info2.demand_p (DEMAND_NONZERO_AVL)) - return info1.demand_p (DEMAND_AVL) - && !info1.demand_p (DEMAND_NONZERO_AVL) && info1.has_avl_reg (); - } - else - return info1.demand_p (DEMAND_AVL) && info1.has_avl_reg (); - return false; -} - -static bool -reg_available_p (const insn_info *insn, const vector_insn_info &info) -{ - if (info.has_avl_reg () && !info.get_avl_source ()) - return false; - insn_info *def_insn = info.get_avl_source ()->insn (); - if (def_insn->bb () == insn->bb ()) - return before_p (def_insn, insn); - else - return dominated_by_p (CDI_DOMINATORS, insn->bb ()->cfg_bb (), - def_insn->bb ()->cfg_bb ()); -} - -/* Return true if the instruction support relaxed compatible check. */ -static bool -support_relaxed_compatible_p (const vector_insn_info &info1, - const vector_insn_info &info2) -{ - if (fault_first_load_p (info1.get_insn ()->rtl ()) - && info2.demand_p (DEMAND_AVL) && info2.has_avl_reg () - && info2.get_avl_source () && info2.get_avl_source ()->insn ()->is_phi ()) - { - hash_set sets - = get_all_sets (info2.get_avl_source (), true, false, false); - for (set_info *set : sets) - { - if (read_vl_insn_p (set->insn ()->rtl ())) - { - const insn_info *insn - = get_backward_fault_first_load_insn (set->insn ()); - if (insn == info1.get_insn ()) - return info2.compatible_vtype_p (info1); - } - } - } - return false; -} - /* Count the number of REGNO in RINSN. */ static int count_regno_occurrences (rtx_insn *rinsn, unsigned int regno) @@ -2118,6 +1723,673 @@ public: } }; +class demand_system +{ +private: + sbitmap *m_avl_def_in; + sbitmap *m_avl_def_out; + + /* predictors. */ + + inline bool always_true (const vsetvl_info &prev ATTRIBUTE_UNUSED, + const vsetvl_info &next ATTRIBUTE_UNUSED) + { + return true; + } + inline bool always_false (const vsetvl_info &prev ATTRIBUTE_UNUSED, + const vsetvl_info &next ATTRIBUTE_UNUSED) + { + return false; + } + + /* predictors for sew and lmul */ + + inline bool eq_lmul_p (const vsetvl_info &prev, const vsetvl_info &next) + { + return prev.get_vlmul () == next.get_vlmul (); + } + inline bool eq_sew_p (const vsetvl_info &prev, const vsetvl_info &next) + { + return prev.get_sew () == next.get_sew (); + } + inline bool eq_sew_lmul_p (const vsetvl_info &prev, const vsetvl_info &next) + { + return eq_lmul_p (prev, next) && eq_sew_p (prev, next); + } + inline bool ge_next_sew_p (const vsetvl_info &prev, const vsetvl_info &next) + { + return prev.get_sew () == next.get_sew () + || (next.get_ta () && prev.get_sew () > next.get_sew ()); + } + inline bool ge_prev_sew_p (const vsetvl_info &prev, const vsetvl_info &next) + { + return prev.get_sew () == next.get_sew () + || (prev.get_ta () && prev.get_sew () < next.get_sew ()); + } + inline bool le_next_max_sew_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return prev.get_sew () <= next.get_max_sew (); + } + inline bool le_prev_max_sew_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return next.get_sew () <= prev.get_max_sew (); + } + inline bool max_sew_overlap_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return !(prev.get_sew () > next.get_max_sew () + || next.get_sew () > prev.get_max_sew ()); + } + inline bool eq_ratio_p (const vsetvl_info &prev, const vsetvl_info &next) + { + return prev.has_same_ratio (next); + } + inline bool has_prev_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return prev.get_ratio () >= (next.get_sew () / 8); + } + inline bool has_next_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return next.get_ratio () >= (prev.get_sew () / 8); + } + + inline bool ge_next_sew_and_eq_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return ge_next_sew_p (prev, next) && eq_ratio_p (prev, next); + } + inline bool ge_next_sew_and_le_next_max_sew_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return ge_next_sew_p (prev, next) && le_next_max_sew_p (prev, next); + } + inline bool + ge_next_sew_and_le_next_max_sew_and_has_next_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return ge_next_sew_p (prev, next) && le_next_max_sew_p (prev, next) + && has_next_ratio_p (prev, next); + } + inline bool ge_prev_sew_and_le_prev_max_sew_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return ge_prev_sew_p (prev, next) && le_prev_max_sew_p (prev, next); + } + inline bool max_sew_overlap_and_has_next_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return has_next_ratio_p (prev, next) && max_sew_overlap_p (prev, next); + } + inline bool + ge_prev_sew_and_le_prev_max_sew_and_eq_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return ge_prev_sew_p (prev, next) && eq_ratio_p (prev, next) + && le_prev_max_sew_p (prev, next); + } + inline bool max_sew_overlap_and_has_prev_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return has_prev_ratio_p (prev, next) && max_sew_overlap_p (prev, next); + } + inline bool + ge_prev_sew_and_le_prev_max_sew_and_has_prev_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return ge_prev_sew_p (prev, next) && has_prev_ratio_p (prev, next) + && le_prev_max_sew_p (prev, next); + } + inline bool max_sew_overlap_and_eq_ratio_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return eq_ratio_p (prev, next) && max_sew_overlap_p (prev, next); + } + + /* predictors for tail and mask policy */ + + inline bool eq_tail_policy_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return prev.get_ta () == next.get_ta (); + } + inline bool eq_mask_policy_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return prev.get_ma () == next.get_ma (); + } + inline bool eq_tail_mask_policy_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return eq_tail_policy_p (prev, next) && eq_mask_policy_p (prev, next); + } + + inline bool comp_tail_policy_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return prev.get_ta () || next.get_ta () || eq_tail_policy_p (prev, next); + } + + inline bool comp_mask_policy_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return prev.get_ma () || next.get_ma () || eq_mask_policy_p (prev, next); + } + + inline bool comp_tail_mask_policy_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return comp_tail_policy_p (prev, next) && comp_mask_policy_p (prev, next); + } + + /* predictors for avl */ + + inline bool def_or_use_vl_p (insn_info *i, const vsetvl_info &info) + { + return info.has_reg_vl () + && (find_access (i->uses (), REGNO (info.get_vl ())) + || find_access (i->defs (), REGNO (info.get_vl ()))); + } + inline bool def_avl_p (insn_info *i, const vsetvl_info &info) + { + return info.has_reg_avl () + && find_access (i->defs (), REGNO (info.get_avl ())); + } + + inline bool def_reg_between (insn_info *prev_insn, insn_info *curr_insn, + unsigned regno) + { + gcc_assert (prev_insn->compare_with (curr_insn) < 0); + for (insn_info *i = curr_insn->prev_nondebug_insn (); i != prev_insn; + i = i->prev_nondebug_insn ()) + { + // no def of regno + if (find_access (i->defs (), regno)) + return true; + } + return false; + } + + inline bool same_reg_avl_p (const vsetvl_info &prev, const vsetvl_info &next) + { + if (!prev.has_reg_avl () || !next.has_reg_avl ()) + return false; + + if (same_equiv_note_p (prev.get_avl_def (), next.get_avl_def ())) + return true; + + if (REGNO (prev.get_avl ()) != REGNO (next.get_avl ())) + return false; + + insn_info *prev_insn = prev.get_insn (); + if (prev.get_bb () != prev_insn->bb ()) + prev_insn = prev.get_bb ()->end_insn (); + + insn_info *next_insn = next.get_insn (); + if (next.get_bb () != next_insn->bb ()) + next_insn = next.get_bb ()->end_insn (); + + return safe_move_avl_vl_p (prev_insn, next_insn, next, false); + } + + inline bool equal_avl_p (const vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + + if (prev.get_ratio () != next.get_ratio ()) + return false; + + if (next.has_reg_vl () && next.use_by_non_rvv_insn_p ()) + return false; + + if (vector_config_insn_p (prev.get_insn ()->rtl ()) && next.get_avl_def () + && next.get_avl_def ()->insn () == prev.get_insn ()) + return true; + + if (prev.get_read_vl_insn ()) + { + if (!next.has_reg_avl () || !next.get_avl_def ()) + return false; + insn_info *avl_def_insn = extract_single_source (next.get_avl_def ()); + return avl_def_insn == prev.get_read_vl_insn (); + } + + if (prev == next && prev.has_reg_avl ()) + { + insn_info *insn = prev.get_insn (); + bb_info *bb = insn->bb (); + for (insn_info *i = insn; real_insn_and_same_bb_p (i, bb); + i = i->next_nondebug_insn ()) + if (find_access (i->defs (), REGNO (prev.get_avl ()))) + return false; + } + + if (prev.has_vlmax_avl () && next.has_vlmax_avl ()) + return true; + else if (prev.has_imm_avl () && next.has_imm_avl ()) + return INTVAL (prev.get_avl ()) == INTVAL (next.get_avl ()); + else if (prev.has_reg_vl () && next.has_reg_avl () + && REGNO (prev.get_vl ()) == REGNO (next.get_avl ())) + { + insn_info *prev_insn = prev.get_insn (); + if (prev.get_bb () != prev_insn->bb ()) + prev_insn = prev.get_bb ()->end_insn (); + + insn_info *next_insn = next.get_insn (); + if (next.get_bb () != next_insn->bb ()) + next_insn = next.get_bb ()->end_insn (); + + return safe_move_avl_vl_p (prev_insn, next_insn, next, false); + } + else if (prev.has_reg_avl () && next.has_reg_avl ()) + return same_reg_avl_p (prev, next); + + return false; + } + inline bool equal_avl_or_prev_non_zero_avl_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + return equal_avl_p (prev, next) || prev.has_non_zero_avl (); + } + + inline bool can_use_next_avl_p (const vsetvl_info &prev, + const vsetvl_info &next) + { + if (!next.has_reg_avl () && !next.has_reg_vl ()) + return true; + + insn_info *prev_insn = prev.get_insn (); + if (prev.get_bb () != prev_insn->bb ()) + prev_insn = prev.get_bb ()->end_insn (); + + insn_info *next_insn = next.get_insn (); + if (next.get_bb () != next_insn->bb ()) + next_insn = next.get_bb ()->end_insn (); + + return safe_move_avl_vl_p (prev_insn, next_insn, next); + } + + inline bool equal_avl_or_next_non_zero_avl_and_can_use_next_avl_p ( + const vsetvl_info &prev, const vsetvl_info &next) + { + return equal_avl_p (prev, next) + || (next.has_non_zero_avl () && can_use_next_avl_p (prev, next)); + } + + /* modifiers */ + + inline void nop (const vsetvl_info &prev ATTRIBUTE_UNUSED, + const vsetvl_info &next ATTRIBUTE_UNUSED) + {} + + /* modifiers for sew and lmul */ + + inline void use_min_max_sew (vsetvl_info &prev, const vsetvl_info &next) + { + prev.set_max_sew (MIN (prev.get_max_sew (), next.get_max_sew ())); + } + inline void use_next_sew (vsetvl_info &prev, const vsetvl_info &next) + { + prev.set_sew (next.get_sew ()); + use_min_max_sew (prev, next); + } + inline void use_max_sew (vsetvl_info &prev, const vsetvl_info &next) + { + auto max_sew = std::max (prev.get_sew (), next.get_sew ()); + prev.set_sew (max_sew); + use_min_max_sew (prev, next); + } + inline void use_next_sew_lmul (vsetvl_info &prev, const vsetvl_info &next) + { + use_next_sew (prev, next); + prev.set_vlmul (next.get_vlmul ()); + prev.set_ratio (next.get_ratio ()); + } + inline void use_next_sew_with_prev_ratio (vsetvl_info &prev, + const vsetvl_info &next) + { + use_next_sew (prev, next); + prev.set_vlmul (calculate_vlmul (next.get_sew (), prev.get_ratio ())); + } + inline void modify_lmul_with_next_ratio (vsetvl_info &prev, + const vsetvl_info &next) + { + prev.set_vlmul (calculate_vlmul (prev.get_sew (), next.get_ratio ())); + prev.set_ratio (next.get_ratio ()); + } + + inline void use_max_sew_and_lmul_with_next_ratio (vsetvl_info &prev, + const vsetvl_info &next) + { + prev.set_vlmul (calculate_vlmul (prev.get_sew (), next.get_ratio ())); + use_max_sew (prev, next); + prev.set_ratio (next.get_ratio ()); + } + + inline void use_max_sew_and_lmul_with_prev_ratio (vsetvl_info &prev, + const vsetvl_info &next) + { + auto max_sew = std::max (prev.get_sew (), next.get_sew ()); + prev.set_vlmul (calculate_vlmul (max_sew, prev.get_ratio ())); + prev.set_sew (max_sew); + } + + /* modifiers for tail and mask policy */ + + inline void use_tail_policy (vsetvl_info &prev, const vsetvl_info &next) + { + if (!next.get_ta ()) + prev.set_ta (next.get_ta ()); + } + inline void use_mask_policy (vsetvl_info &prev, const vsetvl_info &next) + { + if (!next.get_ma ()) + prev.set_ma (next.get_ma ()); + } + inline void use_tail_mask_policy (vsetvl_info &prev, const vsetvl_info &next) + { + use_tail_policy (prev, next); + use_mask_policy (prev, next); + } + + /* modifiers for avl */ + + inline void use_next_avl (vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (can_use_next_avl_p (prev, next)); + prev.update_avl (next); + } + + inline void use_next_avl_when_not_equal (vsetvl_info &prev, + const vsetvl_info &next) + { + if (equal_avl_p (prev, next)) + return; + gcc_assert (next.has_non_zero_avl ()); + use_next_avl (prev, next); + } + +public: + demand_system () : m_avl_def_in (nullptr), m_avl_def_out (nullptr) {} + + void set_avl_in_out_data (sbitmap *avl_def_in, sbitmap *avl_def_out) + { + m_avl_def_in = avl_def_in; + m_avl_def_out = avl_def_out; + } + + /* Can we move vsetvl info between prev_insn and next_insn safe? */ + bool safe_move_avl_vl_p (insn_info *prev_insn, insn_info *next_insn, + const vsetvl_info &info, bool ignore_vl = false) + { + gcc_assert ((ignore_vl && info.has_reg_avl ()) + || (info.has_reg_avl () || info.has_reg_vl ())); + + gcc_assert (!prev_insn->is_debug_insn () && !next_insn->is_debug_insn ()); + if (prev_insn->bb () == next_insn->bb () + && prev_insn->compare_with (next_insn) < 0) + { + for (insn_info *i = next_insn->prev_nondebug_insn (); i != prev_insn; + i = i->prev_nondebug_insn ()) + { + // no def amd use of vl + if (!ignore_vl && def_or_use_vl_p (i, info)) + return false; + + // no def of avl + if (def_avl_p (i, info)) + return false; + } + return true; + } + else + { + if (!ignore_vl && info.has_reg_vl ()) + { + bitmap live_out = df_get_live_out (prev_insn->bb ()->cfg_bb ()); + if (bitmap_bit_p (live_out, REGNO (info.get_vl ()))) + return false; + } + + if (info.has_reg_avl () && m_avl_def_in && m_avl_def_out) + { + bool has_avl_out = false; + unsigned regno = REGNO (info.get_avl ()); + unsigned expr_id; + sbitmap_iterator sbi; + EXECUTE_IF_SET_IN_BITMAP (m_avl_def_out[prev_insn->bb ()->index ()], + 0, expr_id, sbi) + { + if (get_regno (expr_id, last_basic_block_for_fn (cfun)) + != regno) + continue; + has_avl_out = true; + if (!bitmap_bit_p (m_avl_def_in[next_insn->bb ()->index ()], + expr_id)) + return false; + } + if (!has_avl_out) + return false; + } + + for (insn_info *i = next_insn; i != next_insn->bb ()->head_insn (); + i = i->prev_nondebug_insn ()) + { + // no def amd use of vl + if (!ignore_vl && def_or_use_vl_p (i, info)) + return false; + + // no def of avl + if (def_avl_p (i, info)) + return false; + } + + for (insn_info *i = prev_insn->bb ()->end_insn (); i != prev_insn; + i = i->prev_nondebug_insn ()) + { + // no def amd use of vl + if (!ignore_vl && def_or_use_vl_p (i, info)) + return false; + + // no def of avl + if (def_avl_p (i, info)) + return false; + } + } + return true; + } + + bool compatible_sew_lmul_with (const vsetvl_info &prev, + const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + sew_lmul_demand_type prev_flags = prev.get_sew_lmul_demand (); + sew_lmul_demand_type next_flags = next.get_sew_lmul_demand (); +#define DEF_SEW_LMUL_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == sew_lmul_demand_type::PREV_FLAGS \ + && next_flags == sew_lmul_demand_type::NEXT_FLAGS) \ + return COMPATIBLE_P (prev, next); + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + bool available_sew_lmul_with (const vsetvl_info &prev, + const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + sew_lmul_demand_type prev_flags = prev.get_sew_lmul_demand (); + sew_lmul_demand_type next_flags = next.get_sew_lmul_demand (); +#define DEF_SEW_LMUL_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == sew_lmul_demand_type::PREV_FLAGS \ + && next_flags == sew_lmul_demand_type::NEXT_FLAGS) \ + return AVAILABLE_P (prev, next); + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + void merge_sew_lmul_with (vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + sew_lmul_demand_type prev_flags = prev.get_sew_lmul_demand (); + sew_lmul_demand_type next_flags = next.get_sew_lmul_demand (); +#define DEF_SEW_LMUL_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == sew_lmul_demand_type::PREV_FLAGS \ + && next_flags == sew_lmul_demand_type::NEXT_FLAGS) \ + { \ + gcc_assert (COMPATIBLE_P (prev, next)); \ + FUSE (prev, next); \ + prev.set_sew_lmul_demand (sew_lmul_demand_type::NEW_FLAGS); \ + return; \ + } + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + bool compatible_policy_with (const vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + policy_demand_type prev_flags = prev.get_policy_demand (); + policy_demand_type next_flags = next.get_policy_demand (); +#define DEF_POLICY_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == policy_demand_type::PREV_FLAGS \ + && next_flags == policy_demand_type::NEXT_FLAGS) \ + return COMPATIBLE_P (prev, next); + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + bool available_policy_with (const vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + policy_demand_type prev_flags = prev.get_policy_demand (); + policy_demand_type next_flags = next.get_policy_demand (); +#define DEF_POLICY_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == policy_demand_type::PREV_FLAGS \ + && next_flags == policy_demand_type::NEXT_FLAGS) \ + return AVAILABLE_P (prev, next); + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + void merge_policy_with (vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + policy_demand_type prev_flags = prev.get_policy_demand (); + policy_demand_type next_flags = next.get_policy_demand (); +#define DEF_POLICY_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == policy_demand_type::PREV_FLAGS \ + && next_flags == policy_demand_type::NEXT_FLAGS) \ + { \ + gcc_assert (COMPATIBLE_P (prev, next)); \ + FUSE (prev, next); \ + prev.set_policy_demand (policy_demand_type::NEW_FLAGS); \ + return; \ + } + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + bool compatible_avl_with (const vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + avl_demand_type prev_flags = prev.get_avl_demand (); + avl_demand_type next_flags = next.get_avl_demand (); +#define DEF_AVL_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == avl_demand_type::PREV_FLAGS \ + && next_flags == avl_demand_type::NEXT_FLAGS) \ + return COMPATIBLE_P (prev, next); + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + bool available_avl_with (const vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + avl_demand_type prev_flags = prev.get_avl_demand (); + avl_demand_type next_flags = next.get_avl_demand (); +#define DEF_AVL_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == avl_demand_type::PREV_FLAGS \ + && next_flags == avl_demand_type::NEXT_FLAGS) \ + return AVAILABLE_P (prev, next); + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + void merge_avl_with (vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (prev.valid_p () && next.valid_p ()); + avl_demand_type prev_flags = prev.get_avl_demand (); + avl_demand_type next_flags = next.get_avl_demand (); +#define DEF_AVL_RULE(PREV_FLAGS, NEXT_FLAGS, NEW_FLAGS, COMPATIBLE_P, \ + AVAILABLE_P, FUSE) \ + if (prev_flags == avl_demand_type::PREV_FLAGS \ + && next_flags == avl_demand_type::NEXT_FLAGS) \ + { \ + gcc_assert (COMPATIBLE_P (prev, next)); \ + FUSE (prev, next); \ + prev.set_avl_demand (avl_demand_type::NEW_FLAGS); \ + return; \ + } + +#include "riscv-vsetvl.def" + + gcc_unreachable (); + } + + bool compatible_with (const vsetvl_info &prev, const vsetvl_info &next) + { + bool compatible_p = compatible_sew_lmul_with (prev, next) + && compatible_policy_with (prev, next) + && compatible_avl_with (prev, next); + return compatible_p; + } + + bool available_with (const vsetvl_info &prev, const vsetvl_info &next) + { + bool available_p = available_sew_lmul_with (prev, next) + && available_policy_with (prev, next) + && available_avl_with (prev, next); + gcc_assert (!available_p || compatible_with (prev, next)); + return available_p; + } + + void merge_with (vsetvl_info &prev, const vsetvl_info &next) + { + gcc_assert (compatible_with (prev, next)); + merge_sew_lmul_with (prev, next); + merge_policy_with (prev, next); + merge_avl_with (prev, next); + gcc_assert (available_with (prev, next)); + } +}; + vector_infos_manager::vector_infos_manager () { vector_edge_list = nullptr; diff --git a/gcc/config/riscv/riscv-vsetvl.def b/gcc/config/riscv/riscv-vsetvl.def index 709cc4ee0df..9bf804cc56b 100644 --- a/gcc/config/riscv/riscv-vsetvl.def +++ b/gcc/config/riscv/riscv-vsetvl.def @@ -18,496 +18,156 @@ You should have received a copy of the GNU General Public License along with GCC; see the file COPYING3. If not see . */ -#ifndef DEF_INCOMPATIBLE_COND -#define DEF_INCOMPATIBLE_COND(AVL1, SEW1, LMUL1, RATIO1, NONZERO_AVL1, \ - GE_SEW1, TAIL_POLICTY1, MASK_POLICY1, AVL2, \ - SEW2, LMUL2, RATIO2, NONZERO_AVL2, GE_SEW2, \ - TAIL_POLICTY2, MASK_POLICY2, COND) +/* DEF_XXX_RULE (prev_demand, next_demand, fused_demand, compatible_p, + available_p, fuse) + prev_demand: the prev vector insn's sew_lmul_type + next_demand: the next vector insn's sew_lmul_type + fused_demand: if them are compatible, change prev_info demand to the + fused_demand after fuse prev_info and next_info + compatible_p: check if prev_demand and next_demand are compatible + available_p: check if prev_demand is available for next_demand + fuse: if them are compatible, how to modify prev_info */ + +#ifndef DEF_SEW_LMUL_RULE +#define DEF_SEW_LMUL_RULE(prev_demand, next_demand, fused_demand, \ + compatible_p, available_p, fuse) #endif -#ifndef DEF_SEW_LMUL_FUSE_RULE -#define DEF_SEW_LMUL_FUSE_RULE(DEMAND_SEW1, DEMAND_LMUL1, DEMAND_RATIO1, \ - DEMAND_GE_SEW1, DEMAND_SEW2, DEMAND_LMUL2, \ - DEMAND_RATIO2, DEMAND_GE_SEW2, NEW_DEMAND_SEW, \ - NEW_DEMAND_LMUL, NEW_DEMAND_RATIO, \ - NEW_DEMAND_GE_SEW, NEW_SEW, NEW_VLMUL, \ - NEW_RATIO) +#ifndef DEF_POLICY_RULE +#define DEF_POLICY_RULE(prev_demand, next_demand, fused_demand, compatible_p, \ + available_p, fuse) #endif -#ifndef DEF_UNAVAILABLE_COND -#define DEF_UNAVAILABLE_COND(AVL1, SEW1, LMUL1, RATIO1, NONZERO_AVL1, GE_SEW1, \ - TAIL_POLICTY1, MASK_POLICY1, AVL2, SEW2, LMUL2, \ - RATIO2, NONZERO_AVL2, GE_SEW2, TAIL_POLICTY2, \ - MASK_POLICY2, COND) +#ifndef DEF_AVL_RULE +#define DEF_AVL_RULE(prev_demand, next_demand, fused_demand, compatible_p, \ + available_p, fuse) #endif -/* Case 1: Demand compatible AVL. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_TRUE, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_TRUE, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ incompatible_avl_p) - -/* Case 2: Demand same SEW. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_sew_p) - -/* Case 3: Demand same LMUL. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_lmul_p) - -/* Case 4: Demand same RATIO. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_ratio_p) - -/* Case 5: Demand same TAIL_POLICY. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_TRUE, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_TRUE, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_tail_policy_p) - -/* Case 6: Demand same MASK_POLICY. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_TRUE, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_TRUE, - /*COND*/ different_mask_policy_p) - -/* Case 7: Demand non zero AVL. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_TRUE, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_TRUE, /*GE_SEW*/ DEMAND_ANY, - DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_TRUE, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_ANY, - DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ possible_zero_avl_p) - -/* Case 8: First SEW/LMUL/GE_SEW <-> Second RATIO/SEW. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ second_ratio_invalid_for_first_sew_p) -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ second_ratio_invalid_for_first_lmul_p) -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ second_sew_less_than_first_sew_p) - -/* Case 9: First (GE_SEW + LMUL) <-> Second RATIO. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ second_ratio_less_than_first_ratio_p) -/* Case 11: First (SEW + LMUL) <-> Second RATIO. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_ratio_p) -/* Case 13: First (GE_SEW/SEW + RATIO) <-> Second LMUL. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_lmul_p) -/* Case 14: First (LMUL + RATIO) <-> Second SEW. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_sew_p) -/* Case 15: First (LMUL + RATIO) <-> Second GE_SEW. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ first_sew_less_than_second_sew_p) - -/* Case 16: First SEW + Second LMUL <-> First RATIO. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_lmul_p) -/* Case 17: First SEW + Second LMUL <-> Second RATIO. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_sew_p) - -/* Case 18: First SEW + Second RATIO <-> First LMUL. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_ratio_p) - -/* Case 19: First GE_SEW + Second LMUL <-> First RATIO. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ second_lmul_less_than_first_lmul_p) -/* Case 20: First GE_SEW + Second LMUL <-> Second RATIO. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ second_sew_less_than_first_sew_p) - -/* Case 21: First GE_SEW + Second RATIO <-> First LMUL. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ second_ratio_less_than_first_ratio_p) - -/* Case 22: First GE_SEW + Second SEW + First LMUL + Second ratio. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_lmul_p) - -/* Case 23: First GE_SEW + Second SEW + Second LMUL + First ratio. */ -DEF_INCOMPATIBLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ different_ratio_p) - -/* Merge rules. */ -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_TRUE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ false, - /*NEW_DEMAND_RATIO*/ false, - /*NEW_DEMAND_GE_SEW*/ true, greatest_sew, first_vlmul, - first_ratio) - -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_ANY, - /*RATIO*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*SEW*/ DEMAND_ANY, /*LMUL*/ DEMAND_ANY, - /*RATIO*/ DEMAND_TRUE, /*GE_SEW*/ DEMAND_ANY, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ true, - /*NEW_DEMAND_RATIO*/ false, - /*NEW_DEMAND_GE_SEW*/ false, first_sew, - vlmul_for_first_sew_second_ratio, second_ratio) -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_ANY, /*LMUL*/ DEMAND_TRUE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_ANY, - /*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_ANY, - /*RATIO*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ true, - /*NEW_DEMAND_RATIO*/ false, - /*NEW_DEMAND_GE_SEW*/ false, second_sew, first_vlmul, - ratio_for_second_sew_first_vlmul) -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_FALSE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_TRUE, /*GE_SEW*/ DEMAND_FALSE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ false, - /*NEW_DEMAND_RATIO*/ true, - /*NEW_DEMAND_GE_SEW*/ true, first_sew, - vlmul_for_first_sew_second_ratio, second_ratio) -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_TRUE, /*GE_SEW*/ DEMAND_TRUE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ false, - /*NEW_DEMAND_RATIO*/ true, - /*NEW_DEMAND_GE_SEW*/ true, greatest_sew, - vlmul_for_greatest_sew_second_ratio, second_ratio) -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_FALSE, /*LMUL*/ DEMAND_TRUE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_FALSE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ true, - /*NEW_DEMAND_RATIO*/ false, - /*NEW_DEMAND_GE_SEW*/ true, first_sew, second_vlmul, - second_ratio) -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_TRUE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_FALSE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ true, - /*NEW_DEMAND_RATIO*/ false, - /*NEW_DEMAND_GE_SEW*/ false, second_sew, second_vlmul, - second_ratio) -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_TRUE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_TRUE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ true, - /*NEW_DEMAND_RATIO*/ false, - /*NEW_DEMAND_GE_SEW*/ false, greatest_sew, second_vlmul, - second_ratio) - -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_FALSE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ false, - /*NEW_DEMAND_RATIO*/ false, - /*NEW_DEMAND_GE_SEW*/ false, second_sew, second_vlmul, - second_ratio) -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_TRUE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_FALSE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ true, - /*NEW_DEMAND_RATIO*/ false, - /*NEW_DEMAND_GE_SEW*/ false, second_sew, first_vlmul, - second_ratio) -DEF_SEW_LMUL_FUSE_RULE (/*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_TRUE, /*GE_SEW*/ DEMAND_TRUE, - /*SEW*/ DEMAND_TRUE, /*LMUL*/ DEMAND_FALSE, - /*RATIO*/ DEMAND_FALSE, /*GE_SEW*/ DEMAND_FALSE, - /*NEW_DEMAND_SEW*/ true, - /*NEW_DEMAND_LMUL*/ false, - /*NEW_DEMAND_RATIO*/ true, - /*NEW_DEMAND_GE_SEW*/ false, second_sew, first_vlmul, - first_ratio) - -/* Define the unavailable cases for LCM. */ - -/* Case 1: Dem1 (Not demand AVL) is unavailable to Dem2 (Demand AVL). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_FALSE, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_TRUE, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ always_unavailable) -/* Case 2: Dem1 (Demand AVL) is unavailable to Dem2 (Demand normal AVL). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_TRUE, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_TRUE, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ avl_unavailable_p) - -/* Case 3: Dem1 (Not demand TAIL) is unavailable to Dem2 (Demand TAIL). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_FALSE, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_TRUE, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ always_unavailable) - -/* Case 4: Dem1 (Not demand MASK) is unavailable to Dem2 (Demand MASK). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_FALSE, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_TRUE, - /*COND*/ always_unavailable) - -/* Case 5: Dem1 (Demand RATIO) is unavailable to Dem2 (Demand SEW/GE_SEW/LMUL). - */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_FALSE, - /*LMUL*/ DEMAND_FALSE, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ always_unavailable) -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_FALSE, - /*LMUL*/ DEMAND_FALSE, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ always_unavailable) - -/* Case 6: Dem1 (Demand SEW). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_FALSE, /*RATIO*/ DEMAND_FALSE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ sew_unavailable_p) - -/* Case 7: Dem1 (Demand LMUL). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_FALSE, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_FALSE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_FALSE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ lmul_unavailable_p) - -/* Case 8: Dem1 (Demand GE_SEW). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_FALSE, /*RATIO*/ DEMAND_FALSE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ ge_sew_unavailable_p) - -/* Case 9: Dem1 (Demand GE_SEW + LMUL). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_TRUE, /*RATIO*/ DEMAND_FALSE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ ge_sew_lmul_unavailable_p) - -/* Case 10: Dem1 (Demand GE_SEW + RATIO). */ -DEF_UNAVAILABLE_COND (/*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_TRUE, - /*LMUL*/ DEMAND_FALSE, /*RATIO*/ DEMAND_TRUE, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_TRUE, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*AVL*/ DEMAND_ANY, /*SEW*/ DEMAND_ANY, - /*LMUL*/ DEMAND_ANY, /*RATIO*/ DEMAND_ANY, - /*NONZERO_AVL*/ DEMAND_ANY, /*GE_SEW*/ DEMAND_ANY, - /*TAIL_POLICTY*/ DEMAND_ANY, /*MASK_POLICY*/ DEMAND_ANY, - /*COND*/ ge_sew_ratio_unavailable_p) - -#undef DEF_INCOMPATIBLE_COND -#undef DEF_SEW_LMUL_FUSE_RULE -#undef DEF_UNAVAILABLE_COND +/* Define SEW and LMUL rules. */ +DEF_SEW_LMUL_RULE (sew_lmul, sew_lmul, sew_lmul, eq_sew_lmul_p, eq_sew_lmul_p, + nop) +DEF_SEW_LMUL_RULE (sew_lmul, ratio_only, sew_lmul, eq_ratio_p, eq_ratio_p, nop) +DEF_SEW_LMUL_RULE (sew_lmul, sew_only, sew_lmul, eq_sew_p, eq_sew_p, nop) +DEF_SEW_LMUL_RULE (sew_lmul, ge_sew, sew_lmul, + ge_next_sew_and_le_next_max_sew_p, + ge_next_sew_and_le_next_max_sew_p, nop) +DEF_SEW_LMUL_RULE (sew_lmul, ratio_and_ge_sew, sew_lmul, + ge_next_sew_and_le_next_max_sew_and_has_next_ratio_p, + ge_next_sew_and_le_next_max_sew_and_has_next_ratio_p, nop) + +DEF_SEW_LMUL_RULE (ratio_only, sew_lmul, sew_lmul, eq_ratio_p, always_false, + use_next_sew_lmul) +/* use_next_sew_lmul for testcase no change. */ +DEF_SEW_LMUL_RULE (ratio_only, ratio_only, ratio_only, eq_ratio_p, eq_ratio_p, + use_next_sew_lmul) +DEF_SEW_LMUL_RULE (ratio_only, sew_only, sew_lmul, has_prev_ratio_p, + always_false, use_next_sew_with_prev_ratio) +DEF_SEW_LMUL_RULE (ratio_only, ge_sew, ratio_and_ge_sew, has_prev_ratio_p, + always_false, use_next_sew_with_prev_ratio) +DEF_SEW_LMUL_RULE (ratio_only, ratio_and_ge_sew, ratio_and_ge_sew, eq_ratio_p, + always_false, use_next_sew_lmul) + +DEF_SEW_LMUL_RULE (sew_only, sew_lmul, sew_lmul, eq_sew_p, always_false, + use_next_sew_lmul) +DEF_SEW_LMUL_RULE (sew_only, ratio_only, sew_lmul, has_next_ratio_p, + always_false, modify_lmul_with_next_ratio) +DEF_SEW_LMUL_RULE (sew_only, sew_only, sew_only, eq_sew_p, eq_sew_p, nop) +DEF_SEW_LMUL_RULE (sew_only, ge_sew, sew_only, + ge_next_sew_and_le_next_max_sew_p, ge_next_sew_p, nop) +DEF_SEW_LMUL_RULE (sew_only, ratio_and_ge_sew, sew_lmul, + ge_next_sew_and_le_next_max_sew_and_has_next_ratio_p, + always_false, modify_lmul_with_next_ratio) + +DEF_SEW_LMUL_RULE (ge_sew, sew_lmul, sew_lmul, + ge_prev_sew_and_le_prev_max_sew_p, always_false, + use_next_sew_lmul) +DEF_SEW_LMUL_RULE (ge_sew, ratio_only, ratio_and_ge_sew, has_next_ratio_p, + always_false, modify_lmul_with_next_ratio) +DEF_SEW_LMUL_RULE (ge_sew, sew_only, sew_only, + ge_prev_sew_and_le_prev_max_sew_p, always_false, + use_next_sew) +DEF_SEW_LMUL_RULE (ge_sew, ge_sew, ge_sew, max_sew_overlap_p, ge_next_sew_p, + use_max_sew) +DEF_SEW_LMUL_RULE (ge_sew, ratio_and_ge_sew, ratio_and_ge_sew, + max_sew_overlap_and_has_next_ratio_p, always_false, + use_max_sew_and_lmul_with_next_ratio) + +DEF_SEW_LMUL_RULE (ratio_and_ge_sew, sew_lmul, sew_lmul, + ge_prev_sew_and_le_prev_max_sew_and_eq_ratio_p, always_false, + use_next_sew_lmul) +DEF_SEW_LMUL_RULE (ratio_and_ge_sew, ratio_only, ratio_and_ge_sew, eq_ratio_p, + eq_ratio_p, use_max_sew_and_lmul_with_prev_ratio) +DEF_SEW_LMUL_RULE (ratio_and_ge_sew, sew_only, sew_only, + ge_prev_sew_and_le_prev_max_sew_and_has_prev_ratio_p, + always_false, use_next_sew_with_prev_ratio) +DEF_SEW_LMUL_RULE (ratio_and_ge_sew, ge_sew, ratio_and_ge_sew, + max_sew_overlap_and_has_prev_ratio_p, ge_next_sew_p, + use_max_sew_and_lmul_with_prev_ratio) +DEF_SEW_LMUL_RULE (ratio_and_ge_sew, ratio_and_ge_sew, ratio_and_ge_sew, + max_sew_overlap_and_eq_ratio_p, ge_next_sew_and_eq_ratio_p, + use_max_sew_and_lmul_with_prev_ratio) + +/* Define TAIL and MASK compatible and merge rules. */ + +DEF_POLICY_RULE (tail_mask_policy, tail_mask_policy, tail_mask_policy, + comp_tail_mask_policy_p, eq_tail_mask_policy_p, + use_tail_mask_policy) +DEF_POLICY_RULE (tail_mask_policy, tail_policy_only, tail_mask_policy, + comp_tail_policy_p, eq_tail_policy_p, use_tail_policy) +DEF_POLICY_RULE (tail_mask_policy, mask_policy_only, tail_mask_policy, + comp_mask_policy_p, eq_mask_policy_p, use_mask_policy) +DEF_POLICY_RULE (tail_mask_policy, ignore_policy, tail_mask_policy, always_true, + always_true, nop) + +DEF_POLICY_RULE (tail_policy_only, tail_mask_policy, tail_mask_policy, + comp_tail_policy_p, always_false, use_mask_policy) +DEF_POLICY_RULE (tail_policy_only, tail_policy_only, tail_policy_only, + comp_tail_policy_p, eq_tail_policy_p, use_tail_policy) +DEF_POLICY_RULE (tail_policy_only, mask_policy_only, tail_mask_policy, + always_true, always_false, use_mask_policy) +DEF_POLICY_RULE (tail_policy_only, ignore_policy, tail_policy_only, always_true, + always_true, nop) + +DEF_POLICY_RULE (mask_policy_only, tail_mask_policy, tail_mask_policy, + comp_mask_policy_p, always_false, use_tail_policy) +DEF_POLICY_RULE (mask_policy_only, tail_policy_only, tail_mask_policy, + always_true, always_false, use_tail_policy) +DEF_POLICY_RULE (mask_policy_only, mask_policy_only, mask_policy_only, + comp_mask_policy_p, eq_mask_policy_p, use_mask_policy) +DEF_POLICY_RULE (mask_policy_only, ignore_policy, mask_policy_only, always_true, + always_true, nop) + +DEF_POLICY_RULE (ignore_policy, tail_mask_policy, tail_mask_policy, always_true, + always_false, use_tail_mask_policy) +DEF_POLICY_RULE (ignore_policy, tail_policy_only, tail_policy_only, always_true, + always_false, use_tail_policy) +DEF_POLICY_RULE (ignore_policy, mask_policy_only, mask_policy_only, always_true, + always_false, use_mask_policy) +DEF_POLICY_RULE (ignore_policy, ignore_policy, ignore_policy, always_true, + always_true, nop) + +/* Define AVL compatible and merge rules. */ + +DEF_AVL_RULE (avl, avl, avl, equal_avl_p, equal_avl_p, nop) +DEF_AVL_RULE (avl, non_zero_avl, avl, equal_avl_or_prev_non_zero_avl_p, + equal_avl_or_prev_non_zero_avl_p, nop) +DEF_AVL_RULE (avl, ignore_avl, avl, always_true, always_true, nop) + +DEF_AVL_RULE (non_zero_avl, avl, avl, + equal_avl_or_next_non_zero_avl_and_can_use_next_avl_p, + always_false, use_next_avl_when_not_equal) + +DEF_AVL_RULE (non_zero_avl, non_zero_avl, non_zero_avl, always_true, + always_true, nop) +DEF_AVL_RULE (non_zero_avl, ignore_avl, non_zero_avl, always_true, always_true, + nop) + +DEF_AVL_RULE (ignore_avl, avl, avl, can_use_next_avl_p, always_false, + use_next_avl) +DEF_AVL_RULE (ignore_avl, non_zero_avl, non_zero_avl, can_use_next_avl_p, + always_false, use_next_avl) +DEF_AVL_RULE (ignore_avl, ignore_avl, ignore_avl, always_true, always_true, nop) + +#undef DEF_SEW_LMUL_RULE +#undef DEF_POLICY_RULE +#undef DEF_AVL_RULE diff --git a/gcc/config/riscv/riscv-vsetvl.h b/gcc/config/riscv/riscv-vsetvl.h index 9eab276190e..542b2aea625 100644 --- a/gcc/config/riscv/riscv-vsetvl.h +++ b/gcc/config/riscv/riscv-vsetvl.h @@ -40,33 +40,6 @@ enum emit_type EMIT_AFTER, }; -enum demand_type -{ - DEMAND_AVL, - DEMAND_SEW, - DEMAND_LMUL, - DEMAND_RATIO, - DEMAND_NONZERO_AVL, - DEMAND_GE_SEW, - DEMAND_TAIL_POLICY, - DEMAND_MASK_POLICY, - NUM_DEMAND -}; - -enum demand_status -{ - DEMAND_FALSE, - DEMAND_TRUE, - DEMAND_ANY, -}; - -enum fusion_type -{ - INVALID_FUSION, - VALID_AVL_FUSION, - KILLED_AVL_FUSION -}; - enum def_type { REAL_SET = 1 << 0, @@ -171,57 +144,5 @@ public: void dump (FILE *) const; }; -struct demands_pair -{ - demand_status first[NUM_DEMAND]; - demand_status second[NUM_DEMAND]; - bool match_cond_p (const bool *dems1, const bool *dems2) const - { - for (unsigned i = 0; i < NUM_DEMAND; i++) - { - if (first[i] != DEMAND_ANY && first[i] != dems1[i]) - return false; - if (second[i] != DEMAND_ANY && second[i] != dems2[i]) - return false; - } - return true; - } -}; - -struct demands_cond -{ - demands_pair pair; - using CONDITION_TYPE - = bool (*) (const vector_insn_info &, const vector_insn_info &); - CONDITION_TYPE incompatible_p; - bool dual_incompatible_p (const vector_insn_info &info1, - const vector_insn_info &info2) const - { - return ((pair.match_cond_p (info1.get_demands (), info2.get_demands ()) - && incompatible_p (info1, info2)) - || (pair.match_cond_p (info2.get_demands (), info1.get_demands ()) - && incompatible_p (info2, info1))); - } -}; - -struct demands_fuse_rule -{ - demands_pair pair; - bool demand_sew_p; - bool demand_lmul_p; - bool demand_ratio_p; - bool demand_ge_sew_p; - - using NEW_SEW - = unsigned (*) (const vector_insn_info &, const vector_insn_info &); - using NEW_VLMUL - = vlmul_type (*) (const vector_insn_info &, const vector_insn_info &); - using NEW_RATIO - = unsigned (*) (const vector_insn_info &, const vector_insn_info &); - NEW_SEW new_sew; - NEW_VLMUL new_vlmul; - NEW_RATIO new_ratio; -}; - } // namespace riscv_vector #endif From patchwork Tue Oct 17 11:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lehua Ding X-Patchwork-Id: 1849973 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8sPZ28xQz20Pd for ; Tue, 17 Oct 2023 22:36:18 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 458863856965 for ; Tue, 17 Oct 2023 11:36:16 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtpbgjp3.qq.com (smtpbgjp3.qq.com [54.92.39.34]) by sourceware.org (Postfix) with ESMTPS id 8216F3858416 for ; Tue, 17 Oct 2023 11:35:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 8216F3858416 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivai.ai Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivai.ai ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 8216F3858416 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=54.92.39.34 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697542525; cv=none; b=HrpQzdqppSIEbGNyUw+hMPcpuHE7TLecC7s6c5G00zVJPOtYZY0uz35CCPo9477bEBVBI7UEuX3Eb0ft9YU1RUZCLggvmwyh1ja42XqQoCrDMrfHwEcFj8AD7gVd3zKQqxQhPrMk8FvjSyaUzs55MslmWwthNBlSgj6eSDLgTxw= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1697542525; c=relaxed/simple; bh=gBgjp2Xwp3F5jpr3/8r39HUq6N5g8bzIzRsavXK4WOI=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=YAb4vvXYZRqjCOLC/xhZ96ARmX7a13f4E7O9iKzElds2hSGp9LJ64qMm/09s3LReibmHaJcvFSgeVRvfTjJS6AV2LRHbMspgB7jo5NwG5xdBQiH98cyioEFRBpzwv40ptlIXcMt8rRf9or2MbT0GQHxs8avr0K1WHbVvW9pmj4A= ARC-Authentication-Results: i=1; server2.sourceware.org X-QQ-mid: bizesmtp62t1697542515tg7rgp5b Received: from rios-cad121.hadoop.rioslab.org ( [58.60.1.9]) by bizesmtp.qq.com (ESMTP) with id ; Tue, 17 Oct 2023 19:35:14 +0800 (CST) X-QQ-SSF: 01400000000000C0F000000A0000000 X-QQ-FEAT: lkL5M32tl2DK5d8v0mpBMdwePrBq9um1JFPHpdSMExMqNonFMwmfynud2nLsv p1czGJhZUud08AlbmSERS1ep4KSM8bguKlMEM3V7LtRpi6Gd5Rjd+l8RtSKcM7hw18o0hgI ZuTjQt7/2/8nEvTdl81oLLwRdJbBJ/q/N88vjUY1GCFbkM9PMY7rK/i6ip4gBcBDxQigD1P SSUxGUowbcobWNGvFm2wEvfVyLrIvTLKm5vZTzFNCHlbxWyotjbCe7NtDKmSGz3NS/st3ui JmibYAoYynMaymQDbt1+z3k466ZZRzMqUTkdlKOFlF+2MJ2YwU/Ff36MM6EzZaoRXU6RFRt rxanPa/NTXXweDkVJ8eTehSDgoP3++OxMMamNknldpMQ3IaOUhzv99IhePJbTmNyPjjeNkV YZNVdasDi4c= X-QQ-GoodBg: 2 X-BIZMAIL-ID: 10832152116283760800 From: Lehua Ding To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, rdapp.gcc@gmail.com, palmer@rivosinc.com, jeffreyalaw@gmail.com, lehua.ding@rivai.ai Subject: [PATCH V2 03/14] RISC-V: P3: Refactor vector_infos_manager Date: Tue, 17 Oct 2023 19:34:49 +0800 Message-Id: <20231017113500.1160997-4-lehua.ding@rivai.ai> X-Mailer: git-send-email 2.36.3 In-Reply-To: <20231017113500.1160997-1-lehua.ding@rivai.ai> References: <20231017113500.1160997-1-lehua.ding@rivai.ai> MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:rivai.ai:qybglogicsvrgz:qybglogicsvrgz6a-0 X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org This sub-patch refactor vector_infos_manager to a pre_vsetvl class which is responsible for the entire lazy vsetvl jobs. There is no need to introduce a separate vsetvl infos manager, because vsetvl infos are modified by the optimization code. gcc/ChangeLog: * config/riscv/riscv-vsetvl.cc (vector_infos_manager::vector_infos_manager): Removed. (class pre_vsetvl): New class. (vector_infos_manager::create_expr): Removed. (vector_infos_manager::get_expr_id): Removed. (vector_infos_manager::all_same_ratio_p): Removed. (vector_infos_manager::all_avail_in_compatible_p): Removed. (vector_infos_manager::all_same_avl_p): Removed. (vector_infos_manager::expr_set_num): Removed. (vector_infos_manager::release): Removed. (vector_infos_manager::create_bitmap_vectors): Removed. (vector_infos_manager::free_bitmap_vectors): Removed. (vector_infos_manager::dump): Removed. * config/riscv/riscv-vsetvl.h (class vector_infos_manager): Removed. --- gcc/config/riscv/riscv-vsetvl.cc | 632 +++++++++++++------------------ gcc/config/riscv/riscv-vsetvl.h | 75 ---- 2 files changed, 257 insertions(+), 450 deletions(-) -- 2.36.3 diff --git a/gcc/config/riscv/riscv-vsetvl.cc b/gcc/config/riscv/riscv-vsetvl.cc index be40b6fdf4c..c219ad178bb 100644 --- a/gcc/config/riscv/riscv-vsetvl.cc +++ b/gcc/config/riscv/riscv-vsetvl.cc @@ -2390,402 +2390,284 @@ public: } }; -vector_infos_manager::vector_infos_manager () +class pre_vsetvl { - vector_edge_list = nullptr; - vector_kill = nullptr; - vector_del = nullptr; - vector_insert = nullptr; - vector_antic = nullptr; - vector_transp = nullptr; - vector_comp = nullptr; - vector_avin = nullptr; - vector_avout = nullptr; - vector_antin = nullptr; - vector_antout = nullptr; - vector_earliest = nullptr; - vector_insn_infos.safe_grow_cleared (get_max_uid ()); - vector_block_infos.safe_grow_cleared (last_basic_block_for_fn (cfun)); - if (!optimize) - { - basic_block cfg_bb; - rtx_insn *rinsn; - FOR_ALL_BB_FN (cfg_bb, cfun) - { - vector_block_infos[cfg_bb->index].local_dem = vector_insn_info (); - vector_block_infos[cfg_bb->index].reaching_out = vector_insn_info (); - FOR_BB_INSNS (cfg_bb, rinsn) - vector_insn_infos[INSN_UID (rinsn)].parse_insn (rinsn); - } - } - else - { - for (const bb_info *bb : crtl->ssa->bbs ()) - { - vector_block_infos[bb->index ()].local_dem = vector_insn_info (); - vector_block_infos[bb->index ()].reaching_out = vector_insn_info (); - for (insn_info *insn : bb->real_insns ()) - vector_insn_infos[insn->uid ()].parse_insn (insn); - vector_block_infos[bb->index ()].probability = profile_probability (); - } - } -} - -void -vector_infos_manager::create_expr (vector_insn_info &info) -{ - for (size_t i = 0; i < vector_exprs.length (); i++) - if (*vector_exprs[i] == info) - return; - vector_exprs.safe_push (&info); -} - -size_t -vector_infos_manager::get_expr_id (const vector_insn_info &info) const -{ - for (size_t i = 0; i < vector_exprs.length (); i++) - if (*vector_exprs[i] == info) - return i; - gcc_unreachable (); -} - -auto_vec -vector_infos_manager::get_all_available_exprs ( - const vector_insn_info &info) const -{ - auto_vec available_list; - for (size_t i = 0; i < vector_exprs.length (); i++) - if (info.available_p (*vector_exprs[i])) - available_list.safe_push (i); - return available_list; -} - -bool -vector_infos_manager::all_same_ratio_p (sbitmap bitdata) const -{ - if (bitmap_empty_p (bitdata)) - return false; - - int ratio = -1; - unsigned int bb_index; - sbitmap_iterator sbi; - - EXECUTE_IF_SET_IN_BITMAP (bitdata, 0, bb_index, sbi) - { - if (ratio == -1) - ratio = vector_exprs[bb_index]->get_ratio (); - else if (vector_exprs[bb_index]->get_ratio () != ratio) - return false; - } - return true; -} - -/* Return TRUE if the incoming vector configuration state - to CFG_BB is compatible with the vector configuration - state in CFG_BB, FALSE otherwise. */ -bool -vector_infos_manager::all_avail_in_compatible_p (const basic_block cfg_bb) const -{ - const auto &info = vector_block_infos[cfg_bb->index].local_dem; - sbitmap avin = vector_avin[cfg_bb->index]; - unsigned int bb_index; - sbitmap_iterator sbi; - EXECUTE_IF_SET_IN_BITMAP (avin, 0, bb_index, sbi) - { - const auto &avin_info - = static_cast (*vector_exprs[bb_index]); - if (!info.compatible_p (avin_info)) - return false; - } - return true; -} - -bool -vector_infos_manager::all_same_avl_p (const basic_block cfg_bb, - sbitmap bitdata) const -{ - if (bitmap_empty_p (bitdata)) - return false; - - const auto &block_info = vector_block_infos[cfg_bb->index]; - if (!block_info.local_dem.demand_p (DEMAND_AVL)) - return true; - - avl_info avl = block_info.local_dem.get_avl_info (); - unsigned int bb_index; - sbitmap_iterator sbi; - - EXECUTE_IF_SET_IN_BITMAP (bitdata, 0, bb_index, sbi) - { - if (vector_exprs[bb_index]->get_avl_info () != avl) - return false; - } - return true; -} +private: + demand_system dem; + auto_vec vector_block_infos; + + /* data for avl reaching defintion. */ + sbitmap avl_regs; + sbitmap *avl_def_in; + sbitmap *avl_def_out; + sbitmap *reg_def_loc; + + /* data for vsetvl info reaching defintion. */ + vsetvl_info unknow_info; + auto_vec vsetvl_def_exprs; + sbitmap *vsetvl_def_in; + sbitmap *vsetvl_def_out; + + /* data for lcm */ + auto_vec exprs; + sbitmap *avloc; + sbitmap *avin; + sbitmap *avout; + sbitmap *kill; + sbitmap *antloc; + sbitmap *transp; + sbitmap *insert; + sbitmap *del; + struct edge_list *edges; + + auto_vec delete_list; + + vsetvl_block_info &get_block_info (const bb_info *bb) + { + return vector_block_infos[bb->index ()]; + } + const vsetvl_block_info &get_block_info (const basic_block bb) const + { + return vector_block_infos[bb->index]; + } -bool -vector_infos_manager::earliest_fusion_worthwhile_p ( - const basic_block cfg_bb) const -{ - edge e; - edge_iterator ei; - profile_probability prob = profile_probability::uninitialized (); - FOR_EACH_EDGE (e, ei, cfg_bb->succs) - { - if (prob == profile_probability::uninitialized ()) - prob = vector_block_infos[e->dest->index].probability; - else if (prob == vector_block_infos[e->dest->index].probability) - continue; - else - /* We pick the highest probability among those incompatible VSETVL - infos. When all incompatible VSTEVL infos have same probability, we - don't pick any of them. */ - return true; - } - return false; -} + vsetvl_block_info &get_block_info (const basic_block bb) + { + return vector_block_infos[bb->index]; + } -bool -vector_infos_manager::vsetvl_dominated_by_all_preds_p ( - const basic_block cfg_bb, const vector_insn_info &info) const -{ - edge e; - edge_iterator ei; - FOR_EACH_EDGE (e, ei, cfg_bb->preds) - { - const auto &reaching_out = vector_block_infos[e->src->index].reaching_out; - if (e->src->index == cfg_bb->index && reaching_out.compatible_p (info)) - continue; - if (!vsetvl_dominated_by_p (e->src, info, reaching_out, false)) - return false; - } - return true; -} + void add_expr (auto_vec &exprs, vsetvl_info &info) + { + for (vsetvl_info *item : exprs) + { + if (*item == info) + return; + } + exprs.safe_push (&info); + } -size_t -vector_infos_manager::expr_set_num (sbitmap bitdata) const -{ - size_t count = 0; - for (size_t i = 0; i < vector_exprs.length (); i++) - if (bitmap_bit_p (bitdata, i)) - count++; - return count; -} + unsigned get_expr_index (auto_vec &exprs, + const vsetvl_info &info) + { + for (size_t i = 0; i < exprs.length (); i += 1) + { + if (*exprs[i] == info) + return i; + } + gcc_unreachable (); + } -void -vector_infos_manager::release (void) -{ - if (!vector_insn_infos.is_empty ()) - vector_insn_infos.release (); - if (!vector_block_infos.is_empty ()) - vector_block_infos.release (); - if (!vector_exprs.is_empty ()) - vector_exprs.release (); - - gcc_assert (to_refine_vsetvls.is_empty ()); - gcc_assert (to_delete_vsetvls.is_empty ()); - if (optimize > 0) - free_bitmap_vectors (); -} + bool anticpatable_exp_p (const vsetvl_info &header_info) + { + if (!header_info.has_reg_avl () && !header_info.has_reg_vl ()) + return true; -void -vector_infos_manager::create_bitmap_vectors (void) -{ - /* Create the bitmap vectors. */ - vector_antic = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), - vector_exprs.length ()); - vector_transp = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), - vector_exprs.length ()); - vector_comp = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), - vector_exprs.length ()); - vector_avin = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), - vector_exprs.length ()); - vector_avout = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), - vector_exprs.length ()); - vector_kill = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), - vector_exprs.length ()); - vector_antin = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), - vector_exprs.length ()); - vector_antout = sbitmap_vector_alloc (last_basic_block_for_fn (cfun), - vector_exprs.length ()); - - bitmap_vector_ones (vector_transp, last_basic_block_for_fn (cfun)); - bitmap_vector_clear (vector_antic, last_basic_block_for_fn (cfun)); - bitmap_vector_clear (vector_comp, last_basic_block_for_fn (cfun)); - vector_edge_list = create_edge_list (); - vector_earliest = sbitmap_vector_alloc (NUM_EDGES (vector_edge_list), - vector_exprs.length ()); -} + bb_info *bb = header_info.get_bb (); + insn_info *prev_insn = bb->head_insn (); + insn_info *next_insn = header_info.get_insn (); + if (bb != next_insn->bb ()) + next_insn = bb->end_insn (); -void -vector_infos_manager::free_bitmap_vectors (void) -{ - /* Finished. Free up all the things we've allocated. */ - free_edge_list (vector_edge_list); - if (vector_del) - sbitmap_vector_free (vector_del); - if (vector_insert) - sbitmap_vector_free (vector_insert); - if (vector_kill) - sbitmap_vector_free (vector_kill); - if (vector_antic) - sbitmap_vector_free (vector_antic); - if (vector_transp) - sbitmap_vector_free (vector_transp); - if (vector_comp) - sbitmap_vector_free (vector_comp); - if (vector_avin) - sbitmap_vector_free (vector_avin); - if (vector_avout) - sbitmap_vector_free (vector_avout); - if (vector_antin) - sbitmap_vector_free (vector_antin); - if (vector_antout) - sbitmap_vector_free (vector_antout); - if (vector_earliest) - sbitmap_vector_free (vector_earliest); - - vector_edge_list = nullptr; - vector_kill = nullptr; - vector_del = nullptr; - vector_insert = nullptr; - vector_antic = nullptr; - vector_transp = nullptr; - vector_comp = nullptr; - vector_avin = nullptr; - vector_avout = nullptr; - vector_antin = nullptr; - vector_antout = nullptr; - vector_earliest = nullptr; -} + return dem.safe_move_avl_vl_p (prev_insn, next_insn, header_info); + } -void -vector_infos_manager::dump (FILE *file) const -{ - basic_block cfg_bb; - rtx_insn *rinsn; + bool available_exp_p (const vsetvl_info &prev_info, + const vsetvl_info &next_info) + { + return dem.available_with (prev_info, next_info); + } - fprintf (file, "\n"); - FOR_ALL_BB_FN (cfg_bb, cfun) - { - fprintf (file, "Local vector info of :\n", cfg_bb->index); - fprintf (file, "
="); - vector_block_infos[cfg_bb->index].local_dem.dump (file); - FOR_BB_INSNS (cfg_bb, rinsn) - { - if (!NONDEBUG_INSN_P (rinsn) || !has_vtype_op (rinsn)) - continue; - fprintf (file, "=", INSN_UID (rinsn)); - const auto &info = vector_insn_infos[INSN_UID (rinsn)]; - info.dump (file); - } - fprintf (file, "