New .md construct: define_insn_and_rewrite

Several SVE patterns need define_insn_and_splits that generate the
same insn_code, but with different operands.  That's probably a
niche requirement, but it's cropping up often enough on the ACLE
branch that I think it would be good to have a syntactic sugar for it.

This patch therefore adds a new construct called define_insn_and_rewrite.
It's basically a define_insn_and_split with an implicit split pattern,
obtained by copying the insn pattern and replacing match_operands with
match_dups and match_operators with match_op_dups.

Any comments?  Suggestions for better names?

Richard


2019-05-14  Richard Sandiford  <richard.sandiford@arm.com>

gcc/
	* doc/md.texi: Document define_insn_and_rewrite.
	* rtl.def (DEFINE_INSN_AND_REWRITE): New rtx code.
	* gensupport.c (queue_elem): Update comment.
	(replace_operands_with_dups): New function.
	(gen_rewrite_sequence): Likewise.
	(process_rtx): Handle DEFINE_INSN_AND_REWRITE.
	* read-rtl.c (apply_subst_iterator): Likewise.
	(add_condition_to_rtx, named_rtx_p): Likewise.
	(rtx_reader::read_rtx_operand): Likewise.
	* config/aarch64/aarch64-sve.md
	(while_ult<GPI:mode><PRED_ALL:mode>_cc): Rename to...
	(*while_ult<GPI:mode><PRED_ALL:mode>_cc): ...this and use
	define_insn_and_rewrite.
	(*cond_<optab><mode>_any): Turn into define_insn_and_rewrites.
	Remove separate define_split.

> This patch therefore adds a new construct called define_insn_and_rewrite.
> It's basically a define_insn_and_split with an implicit split pattern,
> obtained by copying the insn pattern and replacing match_operands with
> match_dups and match_operators with match_op_dups.

Isn't that what define_subst does in a different context?

> Any comments?  Suggestions for better names?

define_insn_and_subst?

Eric Botcazou <ebotcazou@adacore.com> writes:
>> This patch therefore adds a new construct called define_insn_and_rewrite.
>> It's basically a define_insn_and_split with an implicit split pattern,
>> obtained by copying the insn pattern and replacing match_operands with
>> match_dups and match_operators with match_op_dups.
>
> Isn't that what define_subst does in a different context?
>
>> Any comments?  Suggestions for better names?
>
> define_insn_and_subst?

define_subst is a static substitution though, creating multiple insn codes,
whereas the substitution here is dynamic and can be controlled by C++ code.
It might be confusing to link the two.

Thanks,
Richard

On Tue, 14 May 2019, Richard Sandiford wrote:
> Several SVE patterns need define_insn_and_splits that generate the
> same insn_code, but with different operands.  That's probably a
> niche requirement, but it's cropping up often enough on the ACLE
> branch that I think it would be good to have a syntactic sugar for it.
>
> This patch therefore adds a new construct called define_insn_and_rewrite.

> It's basically a define_insn_and_split with an implicit split pattern,
> obtained by copying the insn pattern and replacing match_operands with
> match_dups and match_operators with match_op_dups.

I think that sentence should be in the documentation, in one of
the introductory sentences.

> Any comments?

>  Suggestions for better names?

I'll pass on this, for now. ;-)
But, IMHO it lacks an allusion to define_insn_and_split.

> Index: gcc/doc/md.texi

> +@end smallexample

Can you please add the equivalent define_insn_and_split
"expansion" to the example?  It'd help understanding the concept
graphically at a glance.

brgds, H-P

On 5/14/19 8:14 AM, Richard Sandiford wrote:
> Several SVE patterns need define_insn_and_splits that generate the
> same insn_code, but with different operands.  That's probably a
> niche requirement, but it's cropping up often enough on the ACLE
> branch that I think it would be good to have a syntactic sugar for it.
> 
> This patch therefore adds a new construct called define_insn_and_rewrite.
> It's basically a define_insn_and_split with an implicit split pattern,
> obtained by copying the insn pattern and replacing match_operands with
> match_dups and match_operators with match_op_dups.
> 
> Any comments?  Suggestions for better names?
No strong opinions here other than to mention that I think there's other
places we could use this.

Jeff

Hans-Peter Nilsson <hp@bitrange.com> writes:
> On Tue, 14 May 2019, Richard Sandiford wrote:
>> Several SVE patterns need define_insn_and_splits that generate the
>> same insn_code, but with different operands.  That's probably a
>> niche requirement, but it's cropping up often enough on the ACLE
>> branch that I think it would be good to have a syntactic sugar for it.
>>
>> This patch therefore adds a new construct called define_insn_and_rewrite.
>
>> It's basically a define_insn_and_split with an implicit split pattern,
>> obtained by copying the insn pattern and replacing match_operands with
>> match_dups and match_operators with match_op_dups.
>
> I think that sentence should be in the documentation, in one of
> the introductory sentences.
>
>> Any comments?
>
>>  Suggestions for better names?
>
> I'll pass on this, for now. ;-)
> But, IMHO it lacks an allusion to define_insn_and_split.
>
>> Index: gcc/doc/md.texi
>
>> +@end smallexample
>
> Can you please add the equivalent define_insn_and_split
> "expansion" to the example?  It'd help understanding the concept
> graphically at a glance.

OK, here's what I installed.

Richard


2019-05-31  Richard Sandiford  <richard.sandiford@arm.com>

gcc/
	* doc/md.texi: Document define_insn_and_rewrite.
	* rtl.def (DEFINE_INSN_AND_REWRITE): New rtx code.
	* gensupport.c (queue_elem): Update comment.
	(replace_operands_with_dups): New function.
	(gen_rewrite_sequence): Likewise.
	(process_rtx): Handle DEFINE_INSN_AND_REWRITE.
	* read-rtl.c (apply_subst_iterator): Likewise.
	(add_condition_to_rtx, named_rtx_p): Likewise.
	(rtx_reader::read_rtx_operand): Likewise.
	* config/aarch64/aarch64-sve.md
	(while_ult<GPI:mode><PRED_ALL:mode>_cc): Rename to...
	(*while_ult<GPI:mode><PRED_ALL:mode>_cc): ...this and use
	define_insn_and_rewrite.
	(*cond_<optab><mode>_any): Turn into define_insn_and_rewrites.
	Remove separate define_split.

Index: gcc/doc/md.texi
===================================================================
--- gcc/doc/md.texi	2019-05-31 17:26:54.367202007 +0100
+++ gcc/doc/md.texi	2019-05-31 17:27:03.711176173 +0100
@@ -8498,6 +8498,119 @@ functionality as two separate @code{defi
 patterns.  It exists for compactness, and as a maintenance tool to prevent
 having to ensure the two patterns' templates match.
 
+@findex define_insn_and_rewrite
+It is sometimes useful to have a @code{define_insn_and_split}
+that replaces specific operands of an instruction but leaves the
+rest of the instruction pattern unchanged.  You can do this directly
+with a @code{define_insn_and_split}, but it requires a
+@var{new-insn-pattern-1} that repeats most of the original @var{insn-pattern}.
+There is also the complication that an implicit @code{parallel} in
+@var{insn-pattern} must become an explicit @code{parallel} in
+@var{new-insn-pattern-1}, which is easy to overlook.
+A simpler alternative is to use @code{define_insn_and_rewrite}, which
+is a form of @code{define_insn_and_split} that automatically generates
+@var{new-insn-pattern-1} by replacing each @code{match_operand}
+in @var{insn-pattern} with a corresponding @code{match_dup}, and each
+@code{match_operator} in the pattern with a corresponding @code{match_op_dup}.
+The arguments are otherwise identical to @code{define_insn_and_split}:
+
+@smallexample
+(define_insn_and_rewrite
+  [@var{insn-pattern}]
+  "@var{condition}"
+  "@var{output-template}"
+  "@var{split-condition}"
+  "@var{preparation-statements}"
+  [@var{insn-attributes}])
+@end smallexample
+
+The @code{match_dup}s and @code{match_op_dup}s in the new
+instruction pattern use any new operand values that the
+@var{preparation-statements} store in the @code{operands} array,
+as for a normal @code{define_insn_and_split}.  @var{preparation-statements}
+can also emit additional instructions before the new instruction.
+They can even emit an entirely different sequence of instructions and
+use @code{DONE} to avoid emitting a new form of the original
+instruction.
+
+The split in a @code{define_insn_and_rewrite} is only intended
+to apply to existing instructions that match @var{insn-pattern}.
+@var{split-condition} must therefore start with @code{&&},
+so that the split condition applies on top of @var{condition}.
+
+Here is an example from the AArch64 SVE port, in which operand 1 is
+known to be equivalent to an all-true constant and isn't used by the
+output template:
+
+@smallexample
+(define_insn_and_rewrite "*while_ult<GPI:mode><PRED_ALL:mode>_cc"
+  [(set (reg:CC CC_REGNUM)
+        (compare:CC
+          (unspec:SI [(match_operand:PRED_ALL 1)
+                      (unspec:PRED_ALL
+                        [(match_operand:GPI 2 "aarch64_reg_or_zero" "rZ")
+                         (match_operand:GPI 3 "aarch64_reg_or_zero" "rZ")]
+                        UNSPEC_WHILE_LO)]
+                     UNSPEC_PTEST_PTRUE)
+          (const_int 0)))
+   (set (match_operand:PRED_ALL 0 "register_operand" "=Upa")
+        (unspec:PRED_ALL [(match_dup 2)
+                          (match_dup 3)]
+                         UNSPEC_WHILE_LO))]
+  "TARGET_SVE"
+  "whilelo\t%0.<PRED_ALL:Vetype>, %<w>2, %<w>3"
+  ;; Force the compiler to drop the unused predicate operand, so that we
+  ;; don't have an unnecessary PTRUE.
+  "&& !CONSTANT_P (operands[1])"
+  @{
+    operands[1] = CONSTM1_RTX (<MODE>mode);
+  @}
+)
+@end smallexample
+
+The splitter in this case simply replaces operand 1 with the constant
+value that it is known to have.  The equivalent @code{define_insn_and_split}
+would be:
+
+@smallexample
+(define_insn_and_split "*while_ult<GPI:mode><PRED_ALL:mode>_cc"
+  [(set (reg:CC CC_REGNUM)
+        (compare:CC
+          (unspec:SI [(match_operand:PRED_ALL 1)
+                      (unspec:PRED_ALL
+                        [(match_operand:GPI 2 "aarch64_reg_or_zero" "rZ")
+                         (match_operand:GPI 3 "aarch64_reg_or_zero" "rZ")]
+                        UNSPEC_WHILE_LO)]
+                     UNSPEC_PTEST_PTRUE)
+          (const_int 0)))
+   (set (match_operand:PRED_ALL 0 "register_operand" "=Upa")
+        (unspec:PRED_ALL [(match_dup 2)
+                          (match_dup 3)]
+                         UNSPEC_WHILE_LO))]
+  "TARGET_SVE"
+  "whilelo\t%0.<PRED_ALL:Vetype>, %<w>2, %<w>3"
+  ;; Force the compiler to drop the unused predicate operand, so that we
+  ;; don't have an unnecessary PTRUE.
+  "&& !CONSTANT_P (operands[1])"
+  [(parallel
+     [(set (reg:CC CC_REGNUM)
+           (compare:CC
+             (unspec:SI [(match_dup 1)
+                         (unspec:PRED_ALL [(match_dup 2)
+                                           (match_dup 3)]
+                                          UNSPEC_WHILE_LO)]
+                        UNSPEC_PTEST_PTRUE)
+             (const_int 0)))
+      (set (match_dup 0)
+           (unspec:PRED_ALL [(match_dup 2)
+                             (match_dup 3)]
+                            UNSPEC_WHILE_LO))])]
+  @{
+    operands[1] = CONSTM1_RTX (<MODE>mode);
+  @}
+)
+@end smallexample
+
 @end ifset
 @ifset INTERNALS
 @node Including Patterns
Index: gcc/rtl.def
===================================================================
--- gcc/rtl.def	2019-05-30 18:34:41.914468735 +0100
+++ gcc/rtl.def	2019-05-31 17:27:03.715176163 +0100
@@ -936,6 +936,12 @@ DEF_RTL_EXPR(DEFINE_SPLIT, "define_split
    7: optionally, a vector of attributes for this insn.  */
 DEF_RTL_EXPR(DEFINE_INSN_AND_SPLIT, "define_insn_and_split", "sEsTsESV", RTX_EXTRA)
 
+/* A form of define_insn_and_split in which the split insn pattern (operand 5)
+   is determined automatically by replacing match_operands with match_dups
+   and match_operators with match_op_dups.  The operands are the same as
+   define_insn_and_split but with operand 5 removed.  */
+DEF_RTL_EXPR(DEFINE_INSN_AND_REWRITE, "define_insn_and_rewrite", "sEsTsSV", RTX_EXTRA)
+
 /* Definition of an RTL peephole operation.
    Follows the same arguments as define_split.  */
 DEF_RTL_EXPR(DEFINE_PEEPHOLE2, "define_peephole2", "EsES", RTX_EXTRA)
Index: gcc/gensupport.c
===================================================================
--- gcc/gensupport.c	2019-05-30 18:34:41.422470115 +0100
+++ gcc/gensupport.c	2019-05-31 17:27:03.711176173 +0100
@@ -70,8 +70,8 @@ struct queue_elem
   rtx data;
   file_location loc;
   struct queue_elem *next;
-  /* In a DEFINE_INSN that came from a DEFINE_INSN_AND_SPLIT, SPLIT
-     points to the generated DEFINE_SPLIT.  */
+  /* In a DEFINE_INSN that came from a DEFINE_INSN_AND_SPLIT or
+     DEFINE_INSN_AND_REWRITE, SPLIT points to the generated DEFINE_SPLIT.  */
   struct queue_elem *split;
 };
 
@@ -485,6 +485,63 @@ remove_constraints (rtx part)
       }
 }
 
+/* Recursively replace MATCH_OPERANDs with MATCH_DUPs and MATCH_OPERATORs
+   with MATCH_OP_DUPs in X.  */
+
+static rtx
+replace_operands_with_dups (rtx x)
+{
+  if (x == 0)
+    return x;
+
+  rtx newx;
+  if (GET_CODE (x) == MATCH_OPERAND)
+    {
+      newx = rtx_alloc (MATCH_DUP);
+      XINT (newx, 0) = XINT (x, 0);
+    }
+  else if (GET_CODE (x) == MATCH_OPERATOR)
+    {
+      newx = rtx_alloc (MATCH_OP_DUP);
+      XINT (newx, 0) = XINT (x, 0);
+      XVEC (newx, 1) = XVEC (x, 2);
+    }
+  else
+    newx = shallow_copy_rtx (x);
+
+  const char *format_ptr = GET_RTX_FORMAT (GET_CODE (x));
+  for (int i = 0; i < GET_RTX_LENGTH (GET_CODE (x)); i++)
+    switch (*format_ptr++)
+      {
+      case 'e':
+      case 'u':
+	XEXP (newx, i) = replace_operands_with_dups (XEXP (x, i));
+	break;
+      case 'E':
+	if (XVEC (x, i) != NULL)
+	  {
+	    XVEC (newx, i) = rtvec_alloc (XVECLEN (x, i));
+	    for (int j = 0; j < XVECLEN (x, i); j++)
+	      XVECEXP (newx, i, j)
+		= replace_operands_with_dups (XVECEXP (x, i, j));
+	  }
+	break;
+      }
+  return newx;
+}
+
+/* Convert matching pattern VEC from a DEFINE_INSN_AND_REWRITE into
+   a sequence that should be generated by the splitter.  */
+
+static rtvec
+gen_rewrite_sequence (rtvec vec)
+{
+  rtvec new_vec = rtvec_alloc (1);
+  rtx x = add_implicit_parallel (vec);
+  RTVEC_ELT (new_vec, 0) = replace_operands_with_dups (x);
+  return new_vec;
+}
+
 /* Process a top level rtx in some way, queuing as appropriate.  */
 
 static void
@@ -527,6 +584,7 @@ process_rtx (rtx desc, file_location loc
       break;
 
     case DEFINE_INSN_AND_SPLIT:
+    case DEFINE_INSN_AND_REWRITE:
       {
 	const char *split_cond;
 	rtx split;
@@ -534,6 +592,7 @@ process_rtx (rtx desc, file_location loc
 	int i;
 	struct queue_elem *insn_elem;
 	struct queue_elem *split_elem;
+	int split_code = (GET_CODE (desc) == DEFINE_INSN_AND_REWRITE ? 5 : 6);
 
 	/* Create a split with values from the insn_and_split.  */
 	split = rtx_alloc (DEFINE_SPLIT);
@@ -555,12 +614,17 @@ process_rtx (rtx desc, file_location loc
 	    split_cond = rtx_reader_ptr->join_c_conditions (XSTR (desc, 2),
 							    split_cond + 2);
 	  }
+	else if (GET_CODE (desc) == DEFINE_INSN_AND_REWRITE)
+	  error_at (loc, "the rewrite condition must start with `&&'");
 	XSTR (split, 1) = split_cond;
-	XVEC (split, 2) = XVEC (desc, 5);
-	XSTR (split, 3) = XSTR (desc, 6);
+	if (GET_CODE (desc) == DEFINE_INSN_AND_REWRITE)
+	  XVEC (split, 2) = gen_rewrite_sequence (XVEC (desc, 1));
+	else
+	  XVEC (split, 2) = XVEC (desc, 5);
+	XSTR (split, 3) = XSTR (desc, split_code);
 
 	/* Fix up the DEFINE_INSN.  */
-	attr = XVEC (desc, 7);
+	attr = XVEC (desc, split_code + 1);
 	PUT_CODE (desc, DEFINE_INSN);
 	XVEC (desc, 4) = attr;
 
Index: gcc/read-rtl.c
===================================================================
--- gcc/read-rtl.c	2019-05-30 18:34:40.478472767 +0100
+++ gcc/read-rtl.c	2019-05-31 17:27:03.715176163 +0100
@@ -286,9 +286,11 @@ apply_subst_iterator (rtx rt, unsigned i
     return;
   gcc_assert (GET_CODE (rt) == DEFINE_INSN
 	      || GET_CODE (rt) == DEFINE_INSN_AND_SPLIT
+	      || GET_CODE (rt) == DEFINE_INSN_AND_REWRITE
 	      || GET_CODE (rt) == DEFINE_EXPAND);
 
-  int attrs = GET_CODE (rt) == DEFINE_INSN_AND_SPLIT ? 7 : 4;
+  int attrs = (GET_CODE (rt) == DEFINE_INSN_AND_SPLIT ? 7
+	       : GET_CODE (rt) == DEFINE_INSN_AND_REWRITE ? 6 : 4);
   attrs_vec = XVEC (rt, attrs);
 
   /* If we've already added attribute 'current_iterator_name', then we
@@ -549,6 +551,7 @@ add_condition_to_rtx (rtx x, const char
       break;
 
     case DEFINE_INSN_AND_SPLIT:
+    case DEFINE_INSN_AND_REWRITE:
       XSTR (x, 2) = add_condition_to_string (XSTR (x, 2), extra);
       XSTR (x, 4) = add_condition_to_string (XSTR (x, 4), extra);
       break;
@@ -632,6 +635,7 @@ named_rtx_p (rtx x)
     case DEFINE_EXPAND:
     case DEFINE_INSN:
     case DEFINE_INSN_AND_SPLIT:
+    case DEFINE_INSN_AND_REWRITE:
       return true;
 
     default:
@@ -1837,8 +1841,8 @@ rtx_reader::read_rtx_operand (rtx return
 	    break;
 	  }
 
-	/* The output template slot of a DEFINE_INSN,
-	   DEFINE_INSN_AND_SPLIT, or DEFINE_PEEPHOLE automatically
+	/* The output template slot of a DEFINE_INSN, DEFINE_INSN_AND_SPLIT,
+	   DEFINE_INSN_AND_REWRITE or DEFINE_PEEPHOLE automatically
 	   gets a star inserted as its first character, if it is
 	   written with a brace block instead of a string constant.  */
 	star_if_braced = (format_ptr[idx] == 'T');
@@ -1855,7 +1859,8 @@ rtx_reader::read_rtx_operand (rtx return
 	if (*stringbuf == '\0'
 	    && idx == 0
 	    && (GET_CODE (return_rtx) == DEFINE_INSN
-		|| GET_CODE (return_rtx) == DEFINE_INSN_AND_SPLIT))
+		|| GET_CODE (return_rtx) == DEFINE_INSN_AND_SPLIT
+		|| GET_CODE (return_rtx) == DEFINE_INSN_AND_REWRITE))
 	  {
 	    struct obstack *string_obstack = get_string_obstack ();
 	    char line_name[20];
Index: gcc/config/aarch64/aarch64-sve.md
===================================================================
--- gcc/config/aarch64/aarch64-sve.md	2019-05-30 18:36:23.538183631 +0100
+++ gcc/config/aarch64/aarch64-sve.md	2019-05-31 17:27:03.711176173 +0100
@@ -1345,7 +1345,7 @@ (define_insn "while_ult<GPI:mode><PRED_A
 ;; WHILELO sets the flags in the same way as a PTEST with a PTRUE GP.
 ;; Handle the case in which both results are useful.  The GP operand
 ;; to the PTEST isn't needed, so we allow it to be anything.
-(define_insn_and_split "while_ult<GPI:mode><PRED_ALL:mode>_cc"
+(define_insn_and_rewrite "*while_ult<GPI:mode><PRED_ALL:mode>_cc"
   [(set (reg:CC CC_REGNUM)
 	(compare:CC
 	  (unspec:SI [(match_operand:PRED_ALL 1)
@@ -1364,12 +1364,8 @@ (define_insn_and_split "while_ult<GPI:mo
   ;; Force the compiler to drop the unused predicate operand, so that we
   ;; don't have an unnecessary PTRUE.
   "&& !CONSTANT_P (operands[1])"
-  [(const_int 0)]
   {
-    emit_insn (gen_while_ult<GPI:mode><PRED_ALL:mode>_cc
-	       (operands[0], CONSTM1_RTX (<MODE>mode),
-		operands[2], operands[3]));
-    DONE;
+    operands[1] = CONSTM1_RTX (<MODE>mode);
   }
 )
 
@@ -2003,7 +1999,7 @@ (define_insn "*cond_<optab><mode>_z"
 )
 
 ;; Synthetic predications with select unmatched.
-(define_insn "*cond_<optab><mode>_any"
+(define_insn_and_rewrite "*cond_<optab><mode>_any"
   [(set (match_operand:SVE_I 0 "register_operand" "=&w")
 	(unspec:SVE_I
 	  [(match_operand:<VPRED> 1 "register_operand" "Upl")
@@ -2012,11 +2008,20 @@ (define_insn "*cond_<optab><mode>_any"
 	     (match_operand:SVE_I 3 "register_operand" "w"))
 	   (match_operand:SVE_I 4 "register_operand"   "w")]
 	  UNSPEC_SEL))]
-  "TARGET_SVE"
+  "TARGET_SVE
+   && !(rtx_equal_p (operands[0], operands[4])
+        || rtx_equal_p (operands[2], operands[4])
+        || rtx_equal_p (operands[3], operands[4]))"
   "#"
+  "&& reload_completed"
+  {
+    emit_insn (gen_vcond_mask_<mode><vpred> (operands[0], operands[2],
+					     operands[4], operands[1]));
+    operands[4] = operands[2] = operands[0];
+  }
 )
 
-(define_insn "*cond_<optab><mode>_any"
+(define_insn_and_rewrite "*cond_<optab><mode>_any"
   [(set (match_operand:SVE_SDI 0 "register_operand" "=&w")
 	(unspec:SVE_SDI
 	  [(match_operand:<VPRED> 1 "register_operand" "Upl")
@@ -2025,33 +2030,17 @@ (define_insn "*cond_<optab><mode>_any"
 	     (match_operand:SVE_SDI 3 "register_operand" "w"))
 	   (match_operand:SVE_SDI 4 "register_operand"   "w")]
 	  UNSPEC_SEL))]
-  "TARGET_SVE"
-  "#"
-)
-
-(define_split
-  [(set (match_operand:SVE_I 0 "register_operand")
-	(unspec:SVE_I
-	  [(match_operand:<VPRED> 1 "register_operand")
-	   (match_operator:SVE_I 5 "aarch64_sve_any_binary_operator"
-	     [(match_operand:SVE_I 2 "register_operand")
-	      (match_operand:SVE_I 3 "register_operand")])
-	   (match_operand:SVE_I 4 "register_operand")]
-	  UNSPEC_SEL))]
-  "TARGET_SVE && reload_completed
+  "TARGET_SVE
    && !(rtx_equal_p (operands[0], operands[4])
         || rtx_equal_p (operands[2], operands[4])
         || rtx_equal_p (operands[3], operands[4]))"
-  ; Not matchable by any one insn or movprfx insn.  We need a separate select.
-  [(set (match_dup 0)
-	(unspec:SVE_I [(match_dup 1) (match_dup 2) (match_dup 4)]
-                      UNSPEC_SEL))
-   (set (match_dup 0)
-	(unspec:SVE_I
-	  [(match_dup 1)
-	   (match_op_dup 5 [(match_dup 0) (match_dup 3)])
-           (match_dup 0)]
-	  UNSPEC_SEL))]
+  "#"
+  "&& reload_completed"
+  {
+    emit_insn (gen_vcond_mask_<mode><vpred> (operands[0], operands[2],
+					     operands[4], operands[1]));
+    operands[4] = operands[2] = operands[0];
+  }
 )
 
 ;; Set operand 0 to the last active element in operand 3, or to tied
@@ -2957,7 +2946,7 @@ (define_insn "*cond_<optab><mode>_z"
 )
 
 ;; Synthetic predication of floating-point operations with select unmatched.
-(define_insn_and_split "*cond_<optab><mode>_any"
+(define_insn_and_rewrite "*cond_<optab><mode>_any"
   [(set (match_operand:SVE_F 0 "register_operand" "=&w")
 	(unspec:SVE_F
 	  [(match_operand:<VPRED> 1 "register_operand" "Upl")
@@ -2974,14 +2963,11 @@ (define_insn_and_split "*cond_<optab><mo
         || rtx_equal_p (operands[2], operands[4])
         || rtx_equal_p (operands[3], operands[4]))"
   ; Not matchable by any one insn or movprfx insn.  We need a separate select.
-  [(set (match_dup 0)
-	(unspec:SVE_F [(match_dup 1) (match_dup 2) (match_dup 4)] UNSPEC_SEL))
-   (set (match_dup 0)
-	(unspec:SVE_F
-	  [(match_dup 1)
-	   (unspec:SVE_F [(match_dup 0) (match_dup 3)] SVE_COND_FP_BINARY)
-           (match_dup 0)]
-	  UNSPEC_SEL))]
+  {
+    emit_insn (gen_vcond_mask_<mode><vpred> (operands[0], operands[2],
+					     operands[4], operands[1]));
+    operands[4] = operands[2] = operands[0];
+  }
 )
 
 ;; Predicated floating-point ternary operations with select.
@@ -3044,7 +3030,7 @@ (define_insn "*cond_<optab><mode>_4"
 
 ;; Predicated floating-point ternary operations in which the value for
 ;; inactive lanes is distinct from the other inputs.
-(define_insn_and_split "*cond_<optab><mode>_any"
+(define_insn_and_rewrite "*cond_<optab><mode>_any"
   [(set (match_operand:SVE_F 0 "register_operand" "=&w, &w, ?&w")
 	(unspec:SVE_F
 	  [(match_operand:<VPRED> 1 "register_operand" "Upl, Upl, Upl")
@@ -3066,16 +3052,11 @@ (define_insn_and_split "*cond_<optab><mo
   "&& reload_completed
    && !CONSTANT_P (operands[5])
    && !rtx_equal_p (operands[0], operands[5])"
-  [(set (match_dup 0)
-	(unspec:SVE_F [(match_dup 1) (match_dup 4) (match_dup 5)] UNSPEC_SEL))
-   (set (match_dup 0)
-	(unspec:SVE_F
-	  [(match_dup 1)
-	   (unspec:SVE_F [(match_dup 2) (match_dup 3) (match_dup 0)]
-			 SVE_COND_FP_TERNARY)
-           (match_dup 0)]
-	  UNSPEC_SEL))]
-  ""
+  {
+    emit_insn (gen_vcond_mask_<mode><vpred> (operands[0], operands[4],
+					     operands[5], operands[1]));
+    operands[5] = operands[4] = operands[0];
+  }
   [(set_attr "movprfx" "yes")]
 )