diff mbox

PR target/68991: Add vector_memory_operand and "Bm" constraint

Message ID CAMe9rOpOhuBk46DHFkdJymGcD89yDH+GGeongQSkK4txN_e-Uw@mail.gmail.com
State New
Headers show

Commit Message

H.J. Lu Jan. 4, 2016, 11:25 p.m. UTC
On Mon, Jan 4, 2016 at 1:11 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
> On Mon, Jan 4, 2016 at 12:19 PM, Uros Bizjak <ubizjak@gmail.com> wrote:
>> On Mon, Jan 4, 2016 at 4:11 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>> On Sat, Jan 2, 2016 at 10:26 AM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>>> On Sat, Jan 2, 2016 at 3:58 AM, Richard Biener
>>>> <richard.guenther@gmail.com> wrote:
>>>>> On January 2, 2016 11:32:33 AM GMT+01:00, Uros Bizjak <ubizjak@gmail.com> wrote:
>>>>>>On Thu, Dec 31, 2015 at 4:29 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>>>>>>> On Thu, Dec 31, 2015 at 1:14 AM, Uros Bizjak <ubizjak@gmail.com>
>>>>>>wrote:
>>>>>>>> On Wed, Dec 30, 2015 at 9:53 PM, H.J. Lu <hjl.tools@gmail.com>
>>>>>>wrote:
>>>>>>>>> SSE vector arithmetic and logic instructions only accept aligned
>>>>>>memory
>>>>>>>>> operand.  This patch adds vector_memory_operand and "Bm" constraint
>>>>>>for
>>>>>>>>> aligned SSE memory operand.  They are applied to SSE any_logic
>>>>>>patterns.
>>>>>>>>>
>>>>>>>>> OK for trunk and release branches if there are regressions?
>>>>>>>>
>>>>>>>> This patch is just papering over deeper problem, as Jakub said in
>>>>>>the PR [1]:
>>>>>>>>
>>>>>>>> --q--
>>>>>>>> GCC uses the ix86_legitimate_combined_insn target hook to disallow
>>>>>>>> misaligned memory into certain SSE instructions.
>>>>>>>> (subreg:V4SI (reg:TI 245 [ MEM[(const struct bitset
>>>>>>&)FeatureEntry_21 + 8] ]) 0)
>>>>>>>> is not misaligned memory, it is a subreg of a pseudo register, so it
>>>>>>is fine.
>>>>>>>> If the replacement of the pseudo register with memory happens in
>>>>>>some
>>>>>>>> other pass, then it probably either should use the
>>>>>>>> legitimate_combined_insn target hook or some other one.  I think we
>>>>>>>> have already a PR where that happens during live range shrinking.
>>>>>>>> --/q--
>>>>>>>>
>>>>>>>> Please figure out where memory replacement happens. There are
>>>>>>several
>>>>>>>> other SSE insns (please grep the .md for "ssememalign" attribute)
>>>>>>that
>>>>>>>> are affected by this problem, so fixing a couple of patterns won't
>>>>>>>> solve the problem completely.
>>>>>>>
>>>>>>> LRA turns
>>>>>>>
>>>>>>> insn 64 63 108 6 (set (reg:V4SI 148 [ vect__28.85 ])
>>>>>>>         (xor:V4SI (reg:V4SI 149)
>>>>>>>             (subreg:V4SI (reg:TI 147 [ MEM[(const struct bitset
>>>>>>> &)FeatureEntry_2(D)] ]) 0))) foo.ii:26 3454 {*xorv4si3}
>>>>>>>      (expr_list:REG_DEAD (reg:V4SI 149)
>>>>>>>         (expr_list:REG_DEAD (reg:TI 147 [ MEM[(const struct bitset
>>>>>>> &)FeatureEntry_2(D)] ])
>>>>>>>             (expr_list:REG_EQUIV (mem/c:V4SI (plus:DI (reg/f:DI 20
>>>>>>frame)
>>>>>>>                         (const_int -16 [0xfffffffffffffff0])) [3
>>>>>>> MEM[(unsigned int *)&D.2851]+0 S16 A128])
>>>>>>>                 (nil)))))
>>>>>>>
>>>>>>> into
>>>>>>>
>>>>>>> (insn 64 63 108 6 (set (reg:V4SI 21 xmm0 [orig:148 vect__28.85 ]
>>>>>>[148])
>>>>>>>         (xor:V4SI (reg:V4SI 21 xmm0 [149])
>>>>>>>             (mem:V4SI (reg/v/f:DI 4 si [orig:117 FeatureEntry ]
>>>>>>[117])
>>>>>>> [6 MEM[(const struct bitset &)FeatureEntry_2(D)]+0 S16 A32])))
>>>>>>> foo.ii:26 3454 {*xorv4si3}
>>>>>>>      (expr_list:REG_EQUIV (mem/c:V4SI (plus:DI (reg/f:DI 20 frame)
>>>>>>>                 (const_int -16 [0xfffffffffffffff0])) [3
>>>>>>MEM[(unsigned
>>>>>>> int *)&D.2851]+0 S16 A128])
>>>>>>>         (nil)))
>>>>>>>
>>>>>>> since
>>>>>>>
>>>>>>> (mem:V4SI (reg/v/f:DI 4 si [orig:117 FeatureEntry ] [117]) [6
>>>>>>> MEM[(const struct bitset &)FeatureEntry_2(D)]+0 S16 A32])))
>>>>>>>
>>>>>>> satisfies the 'm" constraint.  I don't think LRA should call
>>>>>>> ix86_legitimate_combined_insn to validate to validate constraints on
>>>>>>> an instruction.
>>>>>>
>>>>>>Hm...
>>>>>>
>>>>>>if LRA desn't assume that generic "m" constraint implies at least
>>>>>>natural alignment of propageted operand, then your patch is the way to
>>>>>>go.
>>>>>
>>>>> I don't think it even considers alignment. Archs where alignment validity depends on the actual instruction should model this with proper constraints.
>>>>>
>>>>> But in this case, *every* SSE vector memory constraint should be
>>>>>>changed to Bm.
>>>>>
>>>>> I'd say so ...
>>>>
>>>> The "Bm" constraint should be applied only to non-move SSE
>>>> instructions with 16-byte memory operand.
>>>>
>>>
>>> Here are 3 patch which implement it.  There is one exception
>>> on SSE *mov<mode>_internal.  With Bm, LRA will crash, which
>>> may be an LRA bug.   I used m as workaround.
>>>
>>> Tested on x86-64 without regressions.  OK for trunk?
>>
>> Looking at the comment in Patch 3, I'd say let's keep
>> *mov<mode>_internal constraints unchanged. But it looks to me that we
>> have to finally relax
>>
>>       if ((TARGET_AVX || TARGET_IAMCU)
>>           && (misaligned_operand (operands[0], <MODE>mode)
>>           || misaligned_operand (operands[1], <MODE>mode)))
>>
>> condition to allow unaligned moves for all targets, not only AVX and
>> IAMCU. The rationale for this decision is that if the RA won't be able
>> to satisfy Bm constraint, it can load the value into XMM register.
>> This will be done through SSE *mov<mode> internal, so unaligned move
>> has to be generated.
>>
>> But please, double check the changes. In Patch 2, I have found:
>>
>> @ -2041,10 +2041,10 @@
>>     (set_attr "mode" "<MODE>")])
>>
>>  (define_insn "*ieee_smax<mode>3"
>> -  [(set (match_operand:VF 0 "register_operand" "=v,v")
>> +  [(set (match_operand:VF 0 "register_operand" "=x,v")
>>      (unspec:VF
>>        [(match_operand:VF 1 "register_operand" "0,v")
>> -       (match_operand:VF 2 "nonimmediate_operand" "vm,vm")]
>> +       (match_operand:VF 2 "vector_operand" "xm,vm")]
>>       UNSPEC_IEEE_MAX))]
>>    "TARGET_SSE"
>>    "@
>>
>> "xm,vm" constraint can't be right. It shoulld be "xBm,vm".
>
> Fixed.  This is only one I missed.
>
>> Reading your other explanation about 1,2,4 and 8 byte alignment
>> checks, I agree your patch is the way to go.
>>
>> Please drop Patch 3 and retest patches 1 and 2 with
>> *mov<mode>_internal changed as proposed above. Also, please also ask
>> Kirill for a review, as the patch touches his area.
>>
>
> I am testing these 2 patches.
>
> Kirill, please take a look at config/i386/subst.md changes.
>

LRA is fine.  I should use

(define_memory_constraint "Bm"
  "@internal Vector memory operand."
  (match_operand 0 "vector_memory_operand"))

instead of

(define_constraint "Bm"
  "@internal Vector memory operand."
  (match_operand 0 "vector_memory_operand"))


I am retesting these patches now.

Comments

Jakub Jelinek Jan. 5, 2016, 8:11 a.m. UTC | #1
On Mon, Jan 04, 2016 at 03:25:48PM -0800, H.J. Lu wrote:
> LRA is fine.  I should use
> 
> (define_memory_constraint "Bm"
>   "@internal Vector memory operand."
>   (match_operand 0 "vector_memory_operand"))
> 
> instead of
> 
> (define_constraint "Bm"
>   "@internal Vector memory operand."
>   (match_operand 0 "vector_memory_operand"))

I don't think so.  At least the documentation says that
define_memory_constraint is for MEM constraints where if they are not
satisfied they can be made to satisfy by forcing the address into a
register.  But that is not the case here, if a MEM is misaligned, no
equivalent changes to the XEXP (mem, 0) will make it aligned.

	Jakub
H.J. Lu Jan. 5, 2016, 12:32 p.m. UTC | #2
On Tue, Jan 5, 2016 at 12:11 AM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Mon, Jan 04, 2016 at 03:25:48PM -0800, H.J. Lu wrote:
>> LRA is fine.  I should use
>>
>> (define_memory_constraint "Bm"
>>   "@internal Vector memory operand."
>>   (match_operand 0 "vector_memory_operand"))
>>
>> instead of
>>
>> (define_constraint "Bm"
>>   "@internal Vector memory operand."
>>   (match_operand 0 "vector_memory_operand"))
>
> I don't think so.  At least the documentation says that
> define_memory_constraint is for MEM constraints where if they are not
> satisfied they can be made to satisfy by forcing the address into a
> register.  But that is not the case here, if a MEM is misaligned, no
> equivalent changes to the XEXP (mem, 0) will make it aligned.
>

You are right and *mov<mode>_internal must use the 'm' constraint
so that LRA won't keep generating the same reload for

(insn 353 322 323 8 (set (reg:V4SF 192)
        (reg:V4SF 201 [192])) 1226 {*movv4sf_internal}
     (nil))

until

x.i: In function \u2018foo\u2019:
x.i:29:1: internal compiler error: Max. number of generated reload
insns per insn is achieved (90)

 }
 ^

0xc0d635 lra_constraints(bool)
/export/gnu/import/git/sources/gcc/gcc/lra-constraints.c:4336
0xbf9854 lra(_IO_FILE*)
/export/gnu/import/git/sources/gcc/gcc/lra.c:2277
0xba6489 do_reload
/export/gnu/import/git/sources/gcc/gcc/ira.c:5385
0xba683c execute
/export/gnu/import/git/sources/gcc/gcc/ira.c:5556
Richard Sandiford Jan. 5, 2016, 9:34 p.m. UTC | #3
Jakub Jelinek <jakub@redhat.com> writes:
> On Mon, Jan 04, 2016 at 03:25:48PM -0800, H.J. Lu wrote:
>> LRA is fine.  I should use
>> 
>> (define_memory_constraint "Bm"
>>   "@internal Vector memory operand."
>>   (match_operand 0 "vector_memory_operand"))
>> 
>> instead of
>> 
>> (define_constraint "Bm"
>>   "@internal Vector memory operand."
>>   (match_operand 0 "vector_memory_operand"))
>
> I don't think so.  At least the documentation says that
> define_memory_constraint is for MEM constraints where if they are not
> satisfied they can be made to satisfy by forcing the address into a
> register.  But that is not the case here, if a MEM is misaligned, no
> equivalent changes to the XEXP (mem, 0) will make it aligned.

Yeah.  It seems like a missing feature though.  Maybe
define_memory_constraint should have two conditions, one for
when the MEM is valid and one for when the address is?
Guess that isn't stage 3 material though.  And maybe longer-term
we should avoid define_memory_constraint altogether and have an
automatic way of testing whether replacing the address with a
register is OK.

Thanks,
Richard
Uros Bizjak Jan. 6, 2016, 9:18 a.m. UTC | #4
On Tue, Jan 5, 2016 at 10:34 PM, Richard Sandiford
<rdsandiford@googlemail.com> wrote:
> Jakub Jelinek <jakub@redhat.com> writes:
>> On Mon, Jan 04, 2016 at 03:25:48PM -0800, H.J. Lu wrote:
>>> LRA is fine.  I should use
>>>
>>> (define_memory_constraint "Bm"
>>>   "@internal Vector memory operand."
>>>   (match_operand 0 "vector_memory_operand"))
>>>
>>> instead of
>>>
>>> (define_constraint "Bm"
>>>   "@internal Vector memory operand."
>>>   (match_operand 0 "vector_memory_operand"))
>>
>> I don't think so.  At least the documentation says that
>> define_memory_constraint is for MEM constraints where if they are not
>> satisfied they can be made to satisfy by forcing the address into a
>> register.  But that is not the case here, if a MEM is misaligned, no
>> equivalent changes to the XEXP (mem, 0) will make it aligned.
>
> Yeah.  It seems like a missing feature though.  Maybe
> define_memory_constraint should have two conditions, one for
> when the MEM is valid and one for when the address is?
> Guess that isn't stage 3 material though.  And maybe longer-term
> we should avoid define_memory_constraint altogether and have an
> automatic way of testing whether replacing the address with a
> register is OK.

If we are going in this direction, is it possible define_memory_constraint
> Thanks,
> Richard
Uros Bizjak Jan. 6, 2016, 9:23 a.m. UTC | #5
On Wed, Jan 6, 2016 at 10:18 AM, Uros Bizjak <ubizjak@gmail.com> wrote:
> On Tue, Jan 5, 2016 at 10:34 PM, Richard Sandiford
> <rdsandiford@googlemail.com> wrote:
>> Jakub Jelinek <jakub@redhat.com> writes:
>>> On Mon, Jan 04, 2016 at 03:25:48PM -0800, H.J. Lu wrote:
>>>> LRA is fine.  I should use
>>>>
>>>> (define_memory_constraint "Bm"
>>>>   "@internal Vector memory operand."
>>>>   (match_operand 0 "vector_memory_operand"))
>>>>
>>>> instead of
>>>>
>>>> (define_constraint "Bm"
>>>>   "@internal Vector memory operand."
>>>>   (match_operand 0 "vector_memory_operand"))
>>>
>>> I don't think so.  At least the documentation says that
>>> define_memory_constraint is for MEM constraints where if they are not
>>> satisfied they can be made to satisfy by forcing the address into a
>>> register.  But that is not the case here, if a MEM is misaligned, no
>>> equivalent changes to the XEXP (mem, 0) will make it aligned.
>>
>> Yeah.  It seems like a missing feature though.  Maybe
>> define_memory_constraint should have two conditions, one for
>> when the MEM is valid and one for when the address is?
>> Guess that isn't stage 3 material though.  And maybe longer-term
>> we should avoid define_memory_constraint altogether and have an
>> automatic way of testing whether replacing the address with a
>> register is OK.

If we are going in this direction, is it possible to make
define_memory_constraint more general, so it would return register
class, not only default BASE_REG_CLASS? Some x86_64 instructions can
use only non-REX registers (including registers in the address) when
high registers (%ah, %dh, %ch and %bh) are involved.

Uros.
diff mbox

Patch

From 98fa87ff7b7950c4f0d99a82eb5a0e4ec6046095 Mon Sep 17 00:00:00 2001
From: "H.J. Lu" <hjl.tools@gmail.com>
Date: Sat, 2 Jan 2016 07:47:30 -0800
Subject: [PATCH 2/2] Use vector_operand on SSE with 16b memory operand

Add vector_operand, which is vector_memory_operand or register_operand,
and use it, instead of nonimmediate_operand, in SSE patterns with 16-byte
memory operand.  Check of AVX and and IAMCU on misaligned operand in SSE
*mov<mode>_internal is removed so that it can handle both aligned and
misaligned memory operands.

gcc/

	PR target/68991
	* config/i386/i386.c (ix86_expand_vector_logical_operator):
	Replace nonimmediate_operand with vector_operand.
	* config/i386/predicates.md (vector_operand): New predicate.
	(general_vector_operand): Replace nonimmediate_operand with
	vector_operand.
	* config/i386/sse.md (*mov<mode>_internal): Remove check of AVX
	and IAMCU on misaligned operand.
	Rplace nonimmediate_operand with vector_operand and m constraint
	with Bm constraint on SSE patterns with 16-byte memory operand.
	* config/i386/subst.md (round_nimm_predicate): Replace
	nonimmediate_operand with vector_operand.
	(round_saeonly_nimm_predicate): Likewise.
	(round_saeonly_nimm_scalar_predicate): New.

gcc/testsuite/

	PR target/68991
	* gcc.target/i386/pr68991.c: New test.
---
 gcc/config/i386/i386.c                  |   8 +-
 gcc/config/i386/predicates.md           |   9 +-
 gcc/config/i386/sse.md                  | 490 ++++++++++++++++----------------
 gcc/config/i386/subst.md                |   5 +-
 gcc/testsuite/gcc.target/i386/pr68991.c |  32 +++
 5 files changed, 290 insertions(+), 254 deletions(-)
 create mode 100644 gcc/testsuite/gcc.target/i386/pr68991.c

diff --git a/gcc/config/i386/i386.c b/gcc/config/i386/i386.c
index db29b0a..c6c66c7 100644
--- a/gcc/config/i386/i386.c
+++ b/gcc/config/i386/i386.c
@@ -19359,11 +19359,11 @@  ix86_expand_vector_logical_operator (enum rtx_code code, machine_mode mode,
 	    {
 	      op1 = operands[1];
 	      op2 = SUBREG_REG (operands[2]);
-	      if (!nonimmediate_operand (op2, GET_MODE (dst)))
+	      if (!vector_operand (op2, GET_MODE (dst)))
 		op2 = force_reg (GET_MODE (dst), op2);
 	    }
 	  op1 = SUBREG_REG (op1);
-	  if (!nonimmediate_operand (op1, GET_MODE (dst)))
+	  if (!vector_operand (op1, GET_MODE (dst)))
 	    op1 = force_reg (GET_MODE (dst), op1);
 	  emit_insn (gen_rtx_SET (dst,
 				  gen_rtx_fmt_ee (code, GET_MODE (dst),
@@ -19374,9 +19374,9 @@  ix86_expand_vector_logical_operator (enum rtx_code code, machine_mode mode,
 	  break;
 	}
     }
-  if (!nonimmediate_operand (operands[1], mode))
+  if (!vector_operand (operands[1], mode))
     operands[1] = force_reg (mode, operands[1]);
-  if (!nonimmediate_operand (operands[2], mode))
+  if (!vector_operand (operands[2], mode))
     operands[2] = force_reg (mode, operands[2]);
   ix86_fixup_binary_operands_no_copy (code, mode, operands);
   emit_insn (gen_rtx_SET (operands[0],
diff --git a/gcc/config/i386/predicates.md b/gcc/config/i386/predicates.md
index 33b3594..14e80d9 100644
--- a/gcc/config/i386/predicates.md
+++ b/gcc/config/i386/predicates.md
@@ -958,6 +958,11 @@ 
        (ior (match_test "TARGET_AVX")
 	    (match_test "MEM_ALIGN (op) >= GET_MODE_ALIGNMENT (mode)"))))
 
+; Return true when OP is register_operand or vector_memory_operand.
+(define_predicate "vector_operand"
+  (ior (match_operand 0 "register_operand")
+       (match_operand 0 "vector_memory_operand")))
+
 ; Return true when OP is operand acceptable for standard SSE move.
 (define_predicate "vector_move_operand"
   (ior (match_operand 0 "nonimmediate_operand")
@@ -1598,9 +1603,9 @@ 
   return val == ((low << 8) | low);
 })
 
-;; Return true if OP is nonimmediate_operand or CONST_VECTOR.
+;; Return true if OP is vector_operand or CONST_VECTOR.
 (define_predicate "general_vector_operand"
-  (ior (match_operand 0 "nonimmediate_operand")
+  (ior (match_operand 0 "vector_operand")
        (match_code "const_vector")))
 
 ;; Return true if OP is either -1 constant or stored in register.
diff --git a/gcc/config/i386/sse.md b/gcc/config/i386/sse.md
index ca3a831..4b64b6e 100644
--- a/gcc/config/i386/sse.md
+++ b/gcc/config/i386/sse.md
@@ -892,9 +892,8 @@ 
 	case MODE_V16SF:
 	case MODE_V8SF:
 	case MODE_V4SF:
-	  if ((TARGET_AVX || TARGET_IAMCU)
-	      && (misaligned_operand (operands[0], <MODE>mode)
-		  || misaligned_operand (operands[1], <MODE>mode)))
+	  if (misaligned_operand (operands[0], <MODE>mode)
+	      || misaligned_operand (operands[1], <MODE>mode))
 	    return "%vmovups\t{%1, %0|%0, %1}";
 	  else
 	    return "%vmovaps\t{%1, %0|%0, %1}";
@@ -902,18 +901,16 @@ 
 	case MODE_V8DF:
 	case MODE_V4DF:
 	case MODE_V2DF:
-	  if ((TARGET_AVX || TARGET_IAMCU)
-	      && (misaligned_operand (operands[0], <MODE>mode)
-		  || misaligned_operand (operands[1], <MODE>mode)))
+	  if (misaligned_operand (operands[0], <MODE>mode)
+	      || misaligned_operand (operands[1], <MODE>mode))
 	    return "%vmovupd\t{%1, %0|%0, %1}";
 	  else
 	    return "%vmovapd\t{%1, %0|%0, %1}";
 
 	case MODE_OI:
 	case MODE_TI:
-	  if ((TARGET_AVX || TARGET_IAMCU)
-	      && (misaligned_operand (operands[0], <MODE>mode)
-		  || misaligned_operand (operands[1], <MODE>mode)))
+	  if (misaligned_operand (operands[0], <MODE>mode)
+	      || misaligned_operand (operands[1], <MODE>mode))
 	    return TARGET_AVX512VL ? "vmovdqu64\t{%1, %0|%0, %1}"
 				   : "%vmovdqu\t{%1, %0|%0, %1}";
 	  else
@@ -1629,8 +1626,8 @@ 
 (define_insn_and_split "*absneg<mode>2"
   [(set (match_operand:VF 0 "register_operand" "=x,x,v,v")
 	(match_operator:VF 3 "absneg_operator"
-	  [(match_operand:VF 1 "nonimmediate_operand" "0, xm, v, m")]))
-   (use (match_operand:VF 2 "nonimmediate_operand"    "xm, 0, vm,v"))]
+	  [(match_operand:VF 1 "vector_operand" "0,  xBm,v, m")]))
+   (use (match_operand:VF 2 "vector_operand"    "xBm,0,  vm,v"))]
   "TARGET_SSE"
   "#"
   "&& reload_completed"
@@ -1691,7 +1688,7 @@ 
 	(vec_merge:VF_128
 	  (plusminus:VF_128
 	    (match_operand:VF_128 1 "register_operand" "0,v")
-	    (match_operand:VF_128 2 "nonimmediate_operand" "xBm,<round_constraint>"))
+	    (match_operand:VF_128 2 "vector_operand" "xBm,<round_constraint>"))
 	  (match_dup 1)
 	  (const_int 1)))]
   "TARGET_SSE"
@@ -1715,7 +1712,7 @@ 
   [(set (match_operand:VF 0 "register_operand" "=x,v")
 	(mult:VF
 	  (match_operand:VF 1 "<round_nimm_predicate>" "%0,v")
-	  (match_operand:VF 2 "<round_nimm_predicate>" "xm,<round_constraint>")))]
+	  (match_operand:VF 2 "<round_nimm_predicate>" "xBm,<round_constraint>")))]
   "TARGET_SSE && ix86_binary_operator_ok (MULT, <MODE>mode, operands) && <mask_mode512bit_condition> && <round_mode512bit_condition>"
   "@
    mul<ssemodesuffix>\t{%2, %0|%0, %2}
@@ -1731,7 +1728,7 @@ 
 	(vec_merge:VF_128
 	  (multdiv:VF_128
 	    (match_operand:VF_128 1 "register_operand" "0,v")
-	    (match_operand:VF_128 2 "nonimmediate_operand" "xm,<round_constraint>"))
+	    (match_operand:VF_128 2 "vector_operand" "xBm,<round_constraint>"))
 	  (match_dup 1)
 	  (const_int 1)))]
   "TARGET_SSE"
@@ -1747,14 +1744,14 @@ 
 (define_expand "div<mode>3"
   [(set (match_operand:VF2 0 "register_operand")
 	(div:VF2 (match_operand:VF2 1 "register_operand")
-		 (match_operand:VF2 2 "nonimmediate_operand")))]
+		 (match_operand:VF2 2 "vector_operand")))]
   "TARGET_SSE2"
   "ix86_fixup_binary_operands_no_copy (DIV, <MODE>mode, operands);")
 
 (define_expand "div<mode>3"
   [(set (match_operand:VF1 0 "register_operand")
 	(div:VF1 (match_operand:VF1 1 "register_operand")
-		 (match_operand:VF1 2 "nonimmediate_operand")))]
+		 (match_operand:VF1 2 "vector_operand")))]
   "TARGET_SSE"
 {
   ix86_fixup_binary_operands_no_copy (DIV, <MODE>mode, operands);
@@ -1774,7 +1771,7 @@ 
   [(set (match_operand:VF 0 "register_operand" "=x,v")
 	(div:VF
 	  (match_operand:VF 1 "register_operand" "0,v")
-	  (match_operand:VF 2 "<round_nimm_predicate>" "xm,<round_constraint>")))]
+	  (match_operand:VF 2 "<round_nimm_predicate>" "xBm,<round_constraint>")))]
   "TARGET_SSE && <mask_mode512bit_condition> && <round_mode512bit_condition>"
   "@
    div<ssemodesuffix>\t{%2, %0|%0, %2}
@@ -1787,7 +1784,7 @@ 
 (define_insn "<sse>_rcp<mode>2"
   [(set (match_operand:VF1_128_256 0 "register_operand" "=x")
 	(unspec:VF1_128_256
-	  [(match_operand:VF1_128_256 1 "nonimmediate_operand" "xm")] UNSPEC_RCP))]
+	  [(match_operand:VF1_128_256 1 "vector_operand" "xBm")] UNSPEC_RCP))]
   "TARGET_SSE"
   "%vrcpps\t{%1, %0|%0, %1}"
   [(set_attr "type" "sse")
@@ -1842,12 +1839,12 @@ 
 
 (define_expand "sqrt<mode>2"
   [(set (match_operand:VF2 0 "register_operand")
-	(sqrt:VF2 (match_operand:VF2 1 "nonimmediate_operand")))]
+	(sqrt:VF2 (match_operand:VF2 1 "vector_operand")))]
   "TARGET_SSE2")
 
 (define_expand "sqrt<mode>2"
   [(set (match_operand:VF1 0 "register_operand")
-	(sqrt:VF1 (match_operand:VF1 1 "nonimmediate_operand")))]
+	(sqrt:VF1 (match_operand:VF1 1 "vector_operand")))]
   "TARGET_SSE"
 {
   if (TARGET_SSE_MATH
@@ -1876,7 +1873,7 @@ 
   [(set (match_operand:VF_128 0 "register_operand" "=x,v")
 	(vec_merge:VF_128
 	  (sqrt:VF_128
-	    (match_operand:VF_128 1 "nonimmediate_operand" "xm,<round_constraint>"))
+	    (match_operand:VF_128 1 "vector_operand" "xBm,<round_constraint>"))
 	  (match_operand:VF_128 2 "register_operand" "0,v")
 	  (const_int 1)))]
   "TARGET_SSE"
@@ -1893,7 +1890,7 @@ 
 (define_expand "rsqrt<mode>2"
   [(set (match_operand:VF1_128_256 0 "register_operand")
 	(unspec:VF1_128_256
-	  [(match_operand:VF1_128_256 1 "nonimmediate_operand")] UNSPEC_RSQRT))]
+	  [(match_operand:VF1_128_256 1 "vector_operand")] UNSPEC_RSQRT))]
   "TARGET_SSE_MATH"
 {
   ix86_emit_swsqrtsf (operands[0], operands[1], <MODE>mode, true);
@@ -1903,7 +1900,7 @@ 
 (define_insn "<sse>_rsqrt<mode>2"
   [(set (match_operand:VF1_128_256 0 "register_operand" "=x")
 	(unspec:VF1_128_256
-	  [(match_operand:VF1_128_256 1 "nonimmediate_operand" "xm")] UNSPEC_RSQRT))]
+	  [(match_operand:VF1_128_256 1 "vector_operand" "xBm")] UNSPEC_RSQRT))]
   "TARGET_SSE"
   "%vrsqrtps\t{%1, %0|%0, %1}"
   [(set_attr "type" "sse")
@@ -1972,7 +1969,7 @@ 
   [(set (match_operand:VF 0 "register_operand" "=x,v")
 	(smaxmin:VF
 	  (match_operand:VF 1 "<round_saeonly_nimm_predicate>" "%0,v")
-	  (match_operand:VF 2 "<round_saeonly_nimm_predicate>" "xm,<round_saeonly_constraint>")))]
+	  (match_operand:VF 2 "<round_saeonly_nimm_predicate>" "xBm,<round_saeonly_constraint>")))]
   "TARGET_SSE && flag_finite_math_only
    && ix86_binary_operator_ok (<CODE>, <MODE>mode, operands)
    && <mask_mode512bit_condition> && <round_saeonly_mode512bit_condition>"
@@ -1989,7 +1986,7 @@ 
   [(set (match_operand:VF 0 "register_operand" "=x,v")
 	(smaxmin:VF
 	  (match_operand:VF 1 "register_operand" "0,v")
-	  (match_operand:VF 2 "<round_saeonly_nimm_predicate>" "xm,<round_saeonly_constraint>")))]
+	  (match_operand:VF 2 "<round_saeonly_nimm_predicate>" "xBm,<round_saeonly_constraint>")))]
   "TARGET_SSE && !flag_finite_math_only
    && <mask_mode512bit_condition> && <round_saeonly_mode512bit_condition>"
   "@
@@ -2006,7 +2003,7 @@ 
 	(vec_merge:VF_128
 	  (smaxmin:VF_128
 	    (match_operand:VF_128 1 "register_operand" "0,v")
-	    (match_operand:VF_128 2 "nonimmediate_operand" "xm,<round_saeonly_constraint>"))
+	    (match_operand:VF_128 2 "vector_operand" "xBm,<round_saeonly_constraint>"))
 	 (match_dup 1)
 	 (const_int 1)))]
   "TARGET_SSE"
@@ -2026,10 +2023,10 @@ 
 ;; presence of -0.0 and NaN.
 
 (define_insn "*ieee_smin<mode>3"
-  [(set (match_operand:VF 0 "register_operand" "=v,v")
+  [(set (match_operand:VF 0 "register_operand" "=x,v")
 	(unspec:VF
 	  [(match_operand:VF 1 "register_operand" "0,v")
-	   (match_operand:VF 2 "nonimmediate_operand" "vm,vm")]
+	   (match_operand:VF 2 "vector_operand" "xBm,vm")]
 	 UNSPEC_IEEE_MIN))]
   "TARGET_SSE"
   "@
@@ -2041,10 +2038,10 @@ 
    (set_attr "mode" "<MODE>")])
 
 (define_insn "*ieee_smax<mode>3"
-  [(set (match_operand:VF 0 "register_operand" "=v,v")
+  [(set (match_operand:VF 0 "register_operand" "=x,v")
 	(unspec:VF
 	  [(match_operand:VF 1 "register_operand" "0,v")
-	   (match_operand:VF 2 "nonimmediate_operand" "vm,vm")]
+	   (match_operand:VF 2 "vector_operand" "xBm,vm")]
 	 UNSPEC_IEEE_MAX))]
   "TARGET_SSE"
   "@
@@ -2074,7 +2071,7 @@ 
 	(vec_merge:V2DF
 	  (minus:V2DF
 	    (match_operand:V2DF 1 "register_operand" "0,x")
-	    (match_operand:V2DF 2 "nonimmediate_operand" "xm,xm"))
+	    (match_operand:V2DF 2 "vector_operand" "xBm,xm"))
 	  (plus:V2DF (match_dup 1) (match_dup 2))
 	  (const_int 1)))]
   "TARGET_SSE3"
@@ -2106,7 +2103,7 @@ 
 	(vec_merge:V4SF
 	  (minus:V4SF
 	    (match_operand:V4SF 1 "register_operand" "0,x")
-	    (match_operand:V4SF 2 "nonimmediate_operand" "xm,xm"))
+	    (match_operand:V4SF 2 "vector_operand" "xBm,xm"))
 	  (plus:V4SF (match_dup 1) (match_dup 2))
 	  (const_int 5)))]
   "TARGET_SSE3"
@@ -2124,10 +2121,10 @@ 
 	(match_operator:VF_128_256 6 "addsub_vm_operator"
 	  [(minus:VF_128_256
 	     (match_operand:VF_128_256 1 "register_operand")
-	     (match_operand:VF_128_256 2 "nonimmediate_operand"))
+	     (match_operand:VF_128_256 2 "vector_operand"))
 	   (plus:VF_128_256
-	     (match_operand:VF_128_256 3 "nonimmediate_operand")
-	     (match_operand:VF_128_256 4 "nonimmediate_operand"))
+	     (match_operand:VF_128_256 3 "vector_operand")
+	     (match_operand:VF_128_256 4 "vector_operand"))
 	   (match_operand 5 "const_int_operand")]))]
   "TARGET_SSE3
    && can_create_pseudo_p ()
@@ -2145,11 +2142,11 @@ 
   [(set (match_operand:VF_128_256 0 "register_operand")
 	(match_operator:VF_128_256 6 "addsub_vm_operator"
 	  [(plus:VF_128_256
-	     (match_operand:VF_128_256 1 "nonimmediate_operand")
-	     (match_operand:VF_128_256 2 "nonimmediate_operand"))
+	     (match_operand:VF_128_256 1 "vector_operand")
+	     (match_operand:VF_128_256 2 "vector_operand"))
 	   (minus:VF_128_256
 	     (match_operand:VF_128_256 3 "register_operand")
-	     (match_operand:VF_128_256 4 "nonimmediate_operand"))
+	     (match_operand:VF_128_256 4 "vector_operand"))
 	   (match_operand 5 "const_int_operand")]))]
   "TARGET_SSE3
    && can_create_pseudo_p ()
@@ -2175,10 +2172,10 @@ 
 	  [(vec_concat:<ssedoublemode>
 	     (minus:VF_128_256
 	       (match_operand:VF_128_256 1 "register_operand")
-	       (match_operand:VF_128_256 2 "nonimmediate_operand"))
+	       (match_operand:VF_128_256 2 "vector_operand"))
 	     (plus:VF_128_256
-	       (match_operand:VF_128_256 3 "nonimmediate_operand")
-	       (match_operand:VF_128_256 4 "nonimmediate_operand")))
+	       (match_operand:VF_128_256 3 "vector_operand")
+	       (match_operand:VF_128_256 4 "vector_operand")))
 	   (match_parallel 5 "addsub_vs_parallel"
 	     [(match_operand 6 "const_int_operand")])]))]
   "TARGET_SSE3
@@ -2208,11 +2205,11 @@ 
 	(match_operator:VF_128_256 7 "addsub_vs_operator"
 	  [(vec_concat:<ssedoublemode>
 	     (plus:VF_128_256
-	       (match_operand:VF_128_256 1 "nonimmediate_operand")
-	       (match_operand:VF_128_256 2 "nonimmediate_operand"))
+	       (match_operand:VF_128_256 1 "vector_operand")
+	       (match_operand:VF_128_256 2 "vector_operand"))
 	     (minus:VF_128_256
 	       (match_operand:VF_128_256 3 "register_operand")
-	       (match_operand:VF_128_256 4 "nonimmediate_operand")))
+	       (match_operand:VF_128_256 4 "vector_operand")))
 	   (match_parallel 5 "addsub_vs_parallel"
 	     [(match_operand 6 "const_int_operand")])]))]
   "TARGET_SSE3
@@ -2274,7 +2271,7 @@ 
 	    (vec_select:DF (match_dup 1) (parallel [(const_int 1)])))
 	  (plus:DF
 	    (vec_select:DF
-	      (match_operand:V2DF 2 "nonimmediate_operand")
+	      (match_operand:V2DF 2 "vector_operand")
 	      (parallel [(const_int 0)]))
 	    (vec_select:DF (match_dup 2) (parallel [(const_int 1)])))))]
   "TARGET_SSE3")
@@ -2291,7 +2288,7 @@ 
 	      (parallel [(match_operand:SI 4 "const_0_to_1_operand")])))
 	  (plus:DF
 	    (vec_select:DF
-	      (match_operand:V2DF 2 "nonimmediate_operand" "xm,xm")
+	      (match_operand:V2DF 2 "vector_operand" "xBm,xm")
 	      (parallel [(match_operand:SI 5 "const_0_to_1_operand")]))
 	    (vec_select:DF
 	      (match_dup 2)
@@ -2317,7 +2314,7 @@ 
 	    (vec_select:DF (match_dup 1) (parallel [(const_int 1)])))
 	  (minus:DF
 	    (vec_select:DF
-	      (match_operand:V2DF 2 "nonimmediate_operand" "xm,xm")
+	      (match_operand:V2DF 2 "vector_operand" "xBm,xm")
 	      (parallel [(const_int 0)]))
 	    (vec_select:DF (match_dup 2) (parallel [(const_int 1)])))))]
   "TARGET_SSE3"
@@ -2424,7 +2421,7 @@ 
 	  (vec_concat:V2SF
 	    (plusminus:SF
 	      (vec_select:SF
-		(match_operand:V4SF 2 "nonimmediate_operand" "xBm,xm")
+		(match_operand:V4SF 2 "vector_operand" "xBm,xm")
 		(parallel [(const_int 0)]))
 	      (vec_select:SF (match_dup 2) (parallel [(const_int 1)])))
 	    (plusminus:SF
@@ -2650,7 +2647,7 @@ 
   [(set (match_operand:VF_128_256 0 "register_operand" "=x,x")
 	(match_operator:VF_128_256 3 "sse_comparison_operator"
 	  [(match_operand:VF_128_256 1 "register_operand" "%0,x")
-	   (match_operand:VF_128_256 2 "nonimmediate_operand" "xm,xm")]))]
+	   (match_operand:VF_128_256 2 "vector_operand" "xBm,xm")]))]
   "TARGET_SSE
    && GET_RTX_CLASS (GET_CODE (operands[3])) == RTX_COMM_COMPARE"
   "@
@@ -2666,7 +2663,7 @@ 
   [(set (match_operand:VF_128_256 0 "register_operand" "=x,x")
 	(match_operator:VF_128_256 3 "sse_comparison_operator"
 	  [(match_operand:VF_128_256 1 "register_operand" "0,x")
-	   (match_operand:VF_128_256 2 "nonimmediate_operand" "xm,xm")]))]
+	   (match_operand:VF_128_256 2 "vector_operand" "xBm,xm")]))]
   "TARGET_SSE"
   "@
    cmp%D3<ssemodesuffix>\t{%2, %0|%0, %2}
@@ -2682,7 +2679,7 @@ 
 	(vec_merge:VF_128
 	 (match_operator:VF_128 3 "sse_comparison_operator"
 	   [(match_operand:VF_128 1 "register_operand" "0,x")
-	    (match_operand:VF_128 2 "nonimmediate_operand" "xm,xm")])
+	    (match_operand:VF_128 2 "vector_operand" "xBm,xm")])
 	 (match_dup 1)
 	 (const_int 1)))]
   "TARGET_SSE"
@@ -2815,7 +2812,7 @@ 
 	    (match_operand:<ssevecmode> 0 "register_operand" "v")
 	    (parallel [(const_int 0)]))
 	  (vec_select:MODEF
-	    (match_operand:<ssevecmode> 1 "<round_saeonly_nimm_predicate>" "<round_saeonly_constraint>")
+	    (match_operand:<ssevecmode> 1 "<round_saeonly_nimm_scalar_predicate>" "<round_saeonly_constraint>")
 	    (parallel [(const_int 0)]))))]
   "SSE_FLOAT_MODE_P (<MODE>mode)"
   "%vcomi<ssemodesuffix>\t{<round_saeonly_op2>%1, %0|%0, %<iptr>1<round_saeonly_op2>}"
@@ -2835,7 +2832,7 @@ 
 	    (match_operand:<ssevecmode> 0 "register_operand" "v")
 	    (parallel [(const_int 0)]))
 	  (vec_select:MODEF
-	    (match_operand:<ssevecmode> 1 "<round_saeonly_nimm_predicate>" "<round_saeonly_constraint>")
+	    (match_operand:<ssevecmode> 1 "<round_saeonly_nimm_scalar_predicate>" "<round_saeonly_constraint>")
 	    (parallel [(const_int 0)]))))]
   "SSE_FLOAT_MODE_P (<MODE>mode)"
   "%vucomi<ssemodesuffix>\t{<round_saeonly_op2>%1, %0|%0, %<iptr>1<round_saeonly_op2>}"
@@ -2888,7 +2885,7 @@ 
   [(set (match_operand:<sseintvecmode> 0 "register_operand")
 	(match_operator:<sseintvecmode> 1 ""
 	  [(match_operand:VI124_128 2 "register_operand")
-	   (match_operand:VI124_128 3 "nonimmediate_operand")]))]
+	   (match_operand:VI124_128 3 "vector_operand")]))]
   "TARGET_SSE2"
 {
   bool ok = ix86_expand_int_vec_cmp (operands);
@@ -2900,7 +2897,7 @@ 
   [(set (match_operand:V2DI 0 "register_operand")
 	(match_operator:V2DI 1 ""
 	  [(match_operand:V2DI 2 "register_operand")
-	   (match_operand:V2DI 3 "nonimmediate_operand")]))]
+	   (match_operand:V2DI 3 "vector_operand")]))]
   "TARGET_SSE4_2"
 {
   bool ok = ix86_expand_int_vec_cmp (operands);
@@ -2924,7 +2921,7 @@ 
   [(set (match_operand:<sseintvecmode> 0 "register_operand")
 	(match_operator:<sseintvecmode> 1 ""
 	  [(match_operand:VF_128 2 "register_operand")
-	   (match_operand:VF_128 3 "nonimmediate_operand")]))]
+	   (match_operand:VF_128 3 "vector_operand")]))]
   "TARGET_SSE"
 {
   bool ok = ix86_expand_fp_vec_cmp (operands);
@@ -2972,7 +2969,7 @@ 
   [(set (match_operand:<sseintvecmode> 0 "register_operand")
 	(match_operator:<sseintvecmode> 1 ""
 	  [(match_operand:VI124_128 2 "register_operand")
-	   (match_operand:VI124_128 3 "nonimmediate_operand")]))]
+	   (match_operand:VI124_128 3 "vector_operand")]))]
   "TARGET_SSE2"
 {
   bool ok = ix86_expand_int_vec_cmp (operands);
@@ -2984,7 +2981,7 @@ 
   [(set (match_operand:V2DI 0 "register_operand")
 	(match_operator:V2DI 1 ""
 	  [(match_operand:V2DI 2 "register_operand")
-	   (match_operand:V2DI 3 "nonimmediate_operand")]))]
+	   (match_operand:V2DI 3 "vector_operand")]))]
   "TARGET_SSE4_2"
 {
   bool ok = ix86_expand_int_vec_cmp (operands);
@@ -3030,8 +3027,8 @@ 
   [(set (match_operand:V_128 0 "register_operand")
 	(if_then_else:V_128
 	  (match_operator 3 ""
-	    [(match_operand:VF_128 4 "nonimmediate_operand")
-	     (match_operand:VF_128 5 "nonimmediate_operand")])
+	    [(match_operand:VF_128 4 "vector_operand")
+	     (match_operand:VF_128 5 "vector_operand")])
 	  (match_operand:V_128 1 "general_operand")
 	  (match_operand:V_128 2 "general_operand")))]
   "TARGET_SSE
@@ -3075,7 +3072,7 @@ 
 (define_expand "vcond_mask_<mode><sseintvecmodelower>"
   [(set (match_operand:VI124_128 0 "register_operand")
 	(vec_merge:VI124_128
-	  (match_operand:VI124_128 1 "nonimmediate_operand")
+	  (match_operand:VI124_128 1 "vector_operand")
 	  (match_operand:VI124_128 2 "vector_move_operand")
 	  (match_operand:<sseintvecmode> 3 "register_operand")))]
   "TARGET_SSE2"
@@ -3088,7 +3085,7 @@ 
 (define_expand "vcond_mask_v2div2di"
   [(set (match_operand:V2DI 0 "register_operand")
 	(vec_merge:V2DI
-	  (match_operand:V2DI 1 "nonimmediate_operand")
+	  (match_operand:V2DI 1 "vector_operand")
 	  (match_operand:V2DI 2 "vector_move_operand")
 	  (match_operand:V2DI 3 "register_operand")))]
   "TARGET_SSE4_2"
@@ -3114,7 +3111,7 @@ 
 (define_expand "vcond_mask_<mode><sseintvecmodelower>"
   [(set (match_operand:VF_128 0 "register_operand")
 	(vec_merge:VF_128
-	  (match_operand:VF_128 1 "nonimmediate_operand")
+	  (match_operand:VF_128 1 "vector_operand")
 	  (match_operand:VF_128 2 "vector_move_operand")
 	  (match_operand:<sseintvecmode> 3 "register_operand")))]
   "TARGET_SSE"
@@ -3135,7 +3132,7 @@ 
 	(and:VF_128_256
 	  (not:VF_128_256
 	    (match_operand:VF_128_256 1 "register_operand" "0,v"))
-	  (match_operand:VF_128_256 2 "nonimmediate_operand" "xm,vm")))]
+	  (match_operand:VF_128_256 2 "vector_operand" "xBm,vm")))]
   "TARGET_SSE && <mask_avx512vl_condition>"
 {
   static char buf[128];
@@ -3223,8 +3220,8 @@ 
 (define_expand "<code><mode>3<mask_name>"
   [(set (match_operand:VF_128_256 0 "register_operand")
        (any_logic:VF_128_256
-         (match_operand:VF_128_256 1 "nonimmediate_operand")
-         (match_operand:VF_128_256 2 "nonimmediate_operand")))]
+         (match_operand:VF_128_256 1 "vector_operand")
+         (match_operand:VF_128_256 2 "vector_operand")))]
   "TARGET_SSE && <mask_avx512vl_condition>"
   "ix86_fixup_binary_operands_no_copy (<CODE>, <MODE>mode, operands);")
 
@@ -3239,8 +3236,8 @@ 
 (define_insn "*<code><mode>3<mask_name>"
   [(set (match_operand:VF_128_256 0 "register_operand" "=x,v")
 	(any_logic:VF_128_256
-	  (match_operand:VF_128_256 1 "nonimmediate_operand" "%0,v")
-	  (match_operand:VF_128_256 2 "nonimmediate_operand" "xBm,vm")))]
+	  (match_operand:VF_128_256 1 "vector_operand" "%0,v")
+	  (match_operand:VF_128_256 2 "vector_operand" "xBm,vm")))]
   "TARGET_SSE && <mask_avx512vl_condition>
    && ix86_binary_operator_ok (<CODE>, <MODE>mode, operands)"
 {
@@ -3328,10 +3325,10 @@ 
   [(set (match_dup 4)
 	(and:VF
 	  (not:VF (match_dup 3))
-	  (match_operand:VF 1 "nonimmediate_operand")))
+	  (match_operand:VF 1 "vector_operand")))
    (set (match_dup 5)
 	(and:VF (match_dup 3)
-		(match_operand:VF 2 "nonimmediate_operand")))
+		(match_operand:VF 2 "vector_operand")))
    (set (match_operand:VF 0 "register_operand")
 	(ior:VF (match_dup 4) (match_dup 5)))]
   "TARGET_SSE"
@@ -3393,7 +3390,7 @@ 
   [(set (match_operand:TF 0 "register_operand" "=x,x")
 	(and:TF
 	  (not:TF (match_operand:TF 1 "register_operand" "0,x"))
-	  (match_operand:TF 2 "nonimmediate_operand" "xm,xm")))]
+	  (match_operand:TF 2 "vector_operand" "xBm,xm")))]
   "TARGET_SSE"
 {
   static char buf[32];
@@ -3480,16 +3477,16 @@ 
 (define_expand "<code>tf3"
   [(set (match_operand:TF 0 "register_operand")
 	(any_logic:TF
-	  (match_operand:TF 1 "nonimmediate_operand")
-	  (match_operand:TF 2 "nonimmediate_operand")))]
+	  (match_operand:TF 1 "vector_operand")
+	  (match_operand:TF 2 "vector_operand")))]
   "TARGET_SSE"
   "ix86_fixup_binary_operands_no_copy (<CODE>, TFmode, operands);")
 
 (define_insn "*<code>tf3"
   [(set (match_operand:TF 0 "register_operand" "=x,x")
 	(any_logic:TF
-	  (match_operand:TF 1 "nonimmediate_operand" "%0,x")
-	  (match_operand:TF 2 "nonimmediate_operand" "xBm,xm")))]
+	  (match_operand:TF 1 "vector_operand" "%0,x")
+	  (match_operand:TF 2 "vector_operand" "xBm,xm")))]
   "TARGET_SSE
    && ix86_binary_operator_ok (<CODE>, TFmode, operands)"
 {
@@ -4374,7 +4371,7 @@ 
   [(set (match_operand:SI 0 "register_operand" "=r,r")
 	(fix:SI
 	  (vec_select:SF
-	    (match_operand:V4SF 1 "<round_saeonly_nimm_predicate>" "v,<round_saeonly_constraint2>")
+	    (match_operand:V4SF 1 "<round_saeonly_nimm_scalar_predicate>" "v,<round_saeonly_constraint2>")
 	    (parallel [(const_int 0)]))))]
   "TARGET_SSE"
   "%vcvttss2si\t{<round_saeonly_op2>%1, %0|%0, %k1<round_saeonly_op2>}"
@@ -4390,7 +4387,7 @@ 
   [(set (match_operand:DI 0 "register_operand" "=r,r")
 	(fix:DI
 	  (vec_select:SF
-	    (match_operand:V4SF 1 "<round_saeonly_nimm_predicate>" "v,<round_saeonly_constraint>")
+	    (match_operand:V4SF 1 "<round_saeonly_nimm_scalar_predicate>" "v,<round_saeonly_constraint>")
 	    (parallel [(const_int 0)]))))]
   "TARGET_SSE && TARGET_64BIT"
   "%vcvttss2si{q}\t{<round_saeonly_op2>%1, %0|%0, %k1<round_saeonly_op2>}"
@@ -4479,7 +4476,7 @@ 
 (define_insn "<sse2_avx_avx512f>_fix_notrunc<sf2simodelower><mode><mask_name>"
   [(set (match_operand:VI4_AVX 0 "register_operand" "=v")
 	(unspec:VI4_AVX
-	  [(match_operand:<ssePSmode> 1 "nonimmediate_operand" "vm")]
+	  [(match_operand:<ssePSmode> 1 "vector_operand" "vBm")]
 	  UNSPEC_FIX_NOTRUNC))]
   "TARGET_SSE2 && <mask_mode512bit_condition>"
   "%vcvtps2dq\t{%1, %0<mask_operand2>|%0<mask_operand2>, %1}"
@@ -4581,7 +4578,7 @@ 
 
 (define_insn "fix_truncv4sfv4si2<mask_name>"
   [(set (match_operand:V4SI 0 "register_operand" "=v")
-	(fix:V4SI (match_operand:V4SF 1 "nonimmediate_operand" "vm")))]
+	(fix:V4SI (match_operand:V4SF 1 "vector_operand" "vBm")))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
   "%vcvttps2dq\t{%1, %0<mask_operand2>|%0<mask_operand2>, %1}"
   [(set_attr "type" "ssecvt")
@@ -4732,7 +4729,7 @@ 
   [(set (match_operand:SI 0 "register_operand" "=r")
 	(unsigned_fix:SI
 	  (vec_select:SF
-	    (match_operand:V4SF 1 "<round_saeonly_nimm_predicate>" "<round_saeonly_constraint>")
+	    (match_operand:V4SF 1 "<round_saeonly_nimm_scalar_predicate>" "<round_saeonly_constraint>")
 	    (parallel [(const_int 0)]))))]
   "TARGET_AVX512F"
   "vcvttss2usi\t{<round_saeonly_op2>%1, %0|%0, %1<round_saeonly_op2>}"
@@ -4744,7 +4741,7 @@ 
   [(set (match_operand:DI 0 "register_operand" "=r")
 	(unsigned_fix:DI
 	  (vec_select:SF
-	    (match_operand:V4SF 1 "<round_saeonly_nimm_predicate>" "<round_saeonly_constraint>")
+	    (match_operand:V4SF 1 "<round_saeonly_nimm_scalar_predicate>" "<round_saeonly_constraint>")
 	    (parallel [(const_int 0)]))))]
   "TARGET_AVX512F && TARGET_64BIT"
   "vcvttss2usi\t{<round_saeonly_op2>%1, %0|%0, %1<round_saeonly_op2>}"
@@ -4782,7 +4779,7 @@ 
   [(set (match_operand:SI 0 "register_operand" "=r")
 	(unsigned_fix:SI
 	  (vec_select:DF
-	    (match_operand:V2DF 1 "<round_saeonly_nimm_predicate>" "<round_saeonly_constraint>")
+	    (match_operand:V2DF 1 "<round_saeonly_nimm_scalar_predicate>" "<round_saeonly_constraint>")
 	    (parallel [(const_int 0)]))))]
   "TARGET_AVX512F"
   "vcvttsd2usi\t{<round_saeonly_op2>%1, %0|%0, %1<round_saeonly_op2>}"
@@ -4794,7 +4791,7 @@ 
   [(set (match_operand:DI 0 "register_operand" "=r")
 	(unsigned_fix:DI
 	  (vec_select:DF
-	    (match_operand:V2DF 1 "<round_saeonly_nimm_predicate>" "<round_saeonly_constraint>")
+	    (match_operand:V2DF 1 "<round_saeonly_nimm_scalar_predicate>" "<round_saeonly_constraint>")
 	    (parallel [(const_int 0)]))))]
   "TARGET_AVX512F && TARGET_64BIT"
   "vcvttsd2usi\t{<round_saeonly_op2>%1, %0|%0, %1<round_saeonly_op2>}"
@@ -4867,7 +4864,7 @@ 
   [(set (match_operand:SI 0 "register_operand" "=r,r")
 	(fix:SI
 	  (vec_select:DF
-	    (match_operand:V2DF 1 "<round_saeonly_nimm_predicate>" "v,<round_saeonly_constraint2>")
+	    (match_operand:V2DF 1 "<round_saeonly_nimm_scalar_predicate>" "v,<round_saeonly_constraint2>")
 	    (parallel [(const_int 0)]))))]
   "TARGET_SSE2"
   "%vcvttsd2si\t{<round_saeonly_op2>%1, %0|%0, %q1<round_saeonly_op2>}"
@@ -4884,7 +4881,7 @@ 
   [(set (match_operand:DI 0 "register_operand" "=r,r")
 	(fix:DI
 	  (vec_select:DF
-	    (match_operand:V2DF 1 "<round_saeonly_nimm_predicate>" "v,<round_saeonly_constraint2>")
+	    (match_operand:V2DF 1 "<round_saeonly_nimm_scalar_predicate>" "v,<round_saeonly_constraint2>")
 	    (parallel [(const_int 0)]))))]
   "TARGET_SSE2 && TARGET_64BIT"
   "%vcvttsd2si{q}\t{<round_saeonly_op2>%1, %0|%0, %q1<round_saeonly_op2>}"
@@ -5081,7 +5078,7 @@ 
 (define_insn "sse2_cvtpd2dq<mask_name>"
   [(set (match_operand:V4SI 0 "register_operand" "=v")
 	(vec_concat:V4SI
-	  (unspec:V2SI [(match_operand:V2DF 1 "nonimmediate_operand" "vm")]
+	  (unspec:V2SI [(match_operand:V2DF 1 "vector_operand" "vBm")]
 		       UNSPEC_FIX_NOTRUNC)
 	  (const_vector:V2SI [(const_int 0) (const_int 0)])))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
@@ -5242,7 +5239,7 @@ 
 (define_insn "sse2_cvttpd2dq<mask_name>"
   [(set (match_operand:V4SI 0 "register_operand" "=v")
 	(vec_concat:V4SI
-	  (fix:V2SI (match_operand:V2DF 1 "nonimmediate_operand" "vm"))
+	  (fix:V2SI (match_operand:V2DF 1 "vector_operand" "vBm"))
 	  (const_vector:V2SI [(const_int 0) (const_int 0)])))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
 {
@@ -5285,7 +5282,7 @@ 
 	(vec_merge:V2DF
 	  (float_extend:V2DF
 	    (vec_select:V2SF
-	      (match_operand:V4SF 2 "<round_saeonly_nimm_predicate>" "x,m,<round_saeonly_constraint>")
+	      (match_operand:V4SF 2 "<round_saeonly_nimm_scalar_predicate>" "x,m,<round_saeonly_constraint>")
 	      (parallel [(const_int 0) (const_int 1)])))
 	  (match_operand:V2DF 1 "register_operand" "0,0,v")
 	  (const_int 1)))]
@@ -5328,7 +5325,7 @@ 
   [(set (match_operand:V4SF 0 "register_operand")
 	(vec_concat:V4SF
 	  (float_truncate:V2SF
-	    (match_operand:V2DF 1 "nonimmediate_operand"))
+	    (match_operand:V2DF 1 "vector_operand"))
 	  (match_dup 2)))]
   "TARGET_SSE2"
   "operands[2] = CONST0_RTX (V2SFmode);")
@@ -5338,7 +5335,7 @@ 
 	(vec_merge:V4SF
 	  (vec_concat:V4SF
 	    (float_truncate:V2SF
-	      (match_operand:V2DF 1 "nonimmediate_operand"))
+	      (match_operand:V2DF 1 "vector_operand"))
 	    (match_dup 4))
 	  (match_operand:V4SF 2 "register_operand")
 	  (match_operand:QI 3 "register_operand")))]
@@ -5349,7 +5346,7 @@ 
   [(set (match_operand:V4SF 0 "register_operand" "=v")
 	(vec_concat:V4SF
 	  (float_truncate:V2SF
-	    (match_operand:V2DF 1 "nonimmediate_operand" "vm"))
+	    (match_operand:V2DF 1 "vector_operand" "vBm"))
 	  (match_operand:V2SF 2 "const0_operand")))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
 {
@@ -5478,7 +5475,7 @@ 
   [(set (match_operand:V2DF 0 "register_operand" "=v")
 	(float_extend:V2DF
 	  (vec_select:V2SF
-	    (match_operand:V4SF 1 "nonimmediate_operand" "vm")
+	    (match_operand:V4SF 1 "vector_operand" "vBm")
 	    (parallel [(const_int 0) (const_int 1)]))))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
   "%vcvtps2pd\t{%1, %0<mask_operand2>|%0<mask_operand2>, %q1}"
@@ -5495,7 +5492,7 @@ 
    (vec_select:V4SF
      (vec_concat:V8SF
        (match_dup 2)
-       (match_operand:V4SF 1 "nonimmediate_operand"))
+       (match_operand:V4SF 1 "vector_operand"))
      (parallel [(const_int 6) (const_int 7)
 		(const_int 2) (const_int 3)])))
   (set (match_operand:V2DF 0 "register_operand")
@@ -5536,7 +5533,7 @@ 
   [(set (match_operand:V2DF 0 "register_operand")
 	(float_extend:V2DF
 	  (vec_select:V2SF
-	    (match_operand:V4SF 1 "nonimmediate_operand")
+	    (match_operand:V4SF 1 "vector_operand")
 	    (parallel [(const_int 0) (const_int 1)]))))]
   "TARGET_SSE2")
 
@@ -5608,7 +5605,7 @@ 
 (define_expand "vec_unpacks_float_hi_v4si"
   [(set (match_dup 2)
 	(vec_select:V4SI
-	  (match_operand:V4SI 1 "nonimmediate_operand")
+	  (match_operand:V4SI 1 "vector_operand")
 	  (parallel [(const_int 2) (const_int 3)
 		     (const_int 2) (const_int 3)])))
    (set (match_operand:V2DF 0 "register_operand")
@@ -5623,14 +5620,14 @@ 
   [(set (match_operand:V2DF 0 "register_operand")
 	(float:V2DF
 	  (vec_select:V2SI
-	    (match_operand:V4SI 1 "nonimmediate_operand")
+	    (match_operand:V4SI 1 "vector_operand")
 	    (parallel [(const_int 0) (const_int 1)]))))]
   "TARGET_SSE2")
 
 (define_expand "vec_unpacks_float_hi_v8si"
   [(set (match_dup 2)
 	(vec_select:V4SI
-	  (match_operand:V8SI 1 "nonimmediate_operand")
+	  (match_operand:V8SI 1 "vector_operand")
 	  (parallel [(const_int 4) (const_int 5)
 		     (const_int 6) (const_int 7)])))
    (set (match_operand:V4DF 0 "register_operand")
@@ -5676,7 +5673,7 @@ 
 (define_expand "vec_unpacku_float_hi_v4si"
   [(set (match_dup 5)
 	(vec_select:V4SI
-	  (match_operand:V4SI 1 "nonimmediate_operand")
+	  (match_operand:V4SI 1 "vector_operand")
 	  (parallel [(const_int 2) (const_int 3)
 		     (const_int 2) (const_int 3)])))
    (set (match_dup 6)
@@ -5713,7 +5710,7 @@ 
   [(set (match_dup 5)
 	(float:V2DF
 	  (vec_select:V2SI
-	    (match_operand:V4SI 1 "nonimmediate_operand")
+	    (match_operand:V4SI 1 "vector_operand")
 	    (parallel [(const_int 0) (const_int 1)]))))
    (set (match_dup 6)
 	(lt:V2DF (match_dup 5) (match_dup 3)))
@@ -5855,8 +5852,8 @@ 
 
 (define_expand "vec_pack_trunc_v2df"
   [(match_operand:V4SF 0 "register_operand")
-   (match_operand:V2DF 1 "nonimmediate_operand")
-   (match_operand:V2DF 2 "nonimmediate_operand")]
+   (match_operand:V2DF 1 "vector_operand")
+   (match_operand:V2DF 2 "vector_operand")]
   "TARGET_SSE2"
 {
   rtx tmp0, tmp1;
@@ -5917,8 +5914,8 @@ 
 
 (define_expand "vec_pack_sfix_trunc_v2df"
   [(match_operand:V4SI 0 "register_operand")
-   (match_operand:V2DF 1 "nonimmediate_operand")
-   (match_operand:V2DF 2 "nonimmediate_operand")]
+   (match_operand:V2DF 1 "vector_operand")
+   (match_operand:V2DF 2 "vector_operand")]
   "TARGET_SSE2"
 {
   rtx tmp0, tmp1, tmp2;
@@ -6014,8 +6011,8 @@ 
 
 (define_expand "vec_pack_sfix_v2df"
   [(match_operand:V4SI 0 "register_operand")
-   (match_operand:V2DF 1 "nonimmediate_operand")
-   (match_operand:V2DF 2 "nonimmediate_operand")]
+   (match_operand:V2DF 1 "vector_operand")
+   (match_operand:V2DF 2 "vector_operand")]
   "TARGET_SSE2"
 {
   rtx tmp0, tmp1, tmp2;
@@ -6218,7 +6215,7 @@ 
 	(vec_select:V4SF
 	  (vec_concat:V8SF
 	    (match_operand:V4SF 1 "register_operand" "0,v")
-	    (match_operand:V4SF 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V4SF 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 2) (const_int 6)
 		     (const_int 3) (const_int 7)])))]
   "TARGET_SSE && <mask_avx512vl_condition>"
@@ -6323,7 +6320,7 @@ 
 	(vec_select:V4SF
 	  (vec_concat:V8SF
 	    (match_operand:V4SF 1 "register_operand" "0,x")
-	    (match_operand:V4SF 2 "nonimmediate_operand" "xm,xm"))
+	    (match_operand:V4SF 2 "vector_operand" "xBm,xm"))
 	  (parallel [(const_int 0) (const_int 4)
 		     (const_int 1) (const_int 5)])))]
   "TARGET_SSE"
@@ -6357,7 +6354,7 @@ 
   [(set (match_operand:V4SF 0 "register_operand" "=v")
 	(vec_select:V4SF
 	  (vec_concat:V8SF
-	    (match_operand:V4SF 1 "nonimmediate_operand" "vm")
+	    (match_operand:V4SF 1 "vector_operand" "vBm")
 	    (match_dup 1))
 	  (parallel [(const_int 1)
 		     (const_int 1)
@@ -6410,7 +6407,7 @@ 
   [(set (match_operand:V4SF 0 "register_operand" "=v")
 	(vec_select:V4SF
 	  (vec_concat:V8SF
-	    (match_operand:V4SF 1 "nonimmediate_operand" "vm")
+	    (match_operand:V4SF 1 "vector_operand" "vBm")
 	    (match_dup 1))
 	  (parallel [(const_int 0)
 		     (const_int 0)
@@ -6505,7 +6502,7 @@ 
 (define_expand "sse_shufps<mask_expand4_name>"
   [(match_operand:V4SF 0 "register_operand")
    (match_operand:V4SF 1 "register_operand")
-   (match_operand:V4SF 2 "nonimmediate_operand")
+   (match_operand:V4SF 2 "vector_operand")
    (match_operand:SI 3 "const_int_operand")]
   "TARGET_SSE"
 {
@@ -6555,7 +6552,7 @@ 
 	(vec_select:VI4F_128
 	  (vec_concat:<ssedoublevecmode>
 	    (match_operand:VI4F_128 1 "register_operand" "0,x")
-	    (match_operand:VI4F_128 2 "nonimmediate_operand" "xm,xm"))
+	    (match_operand:VI4F_128 2 "vector_operand" "xBm,xm"))
 	  (parallel [(match_operand 3 "const_0_to_3_operand")
 		     (match_operand 4 "const_0_to_3_operand")
 		     (match_operand 5 "const_4_to_7_operand")
@@ -6651,6 +6648,7 @@ 
    %vmovaps\t{%1, %0|%0, %1}
    %vmovlps\t{%1, %d0|%d0, %q1}"
   [(set_attr "type" "ssemov")
+   (set_attr "ssememalign" "64")
    (set_attr "prefix" "maybe_vex")
    (set_attr "mode" "V2SF,V4SF,V2SF")])
 
@@ -6776,7 +6774,7 @@ 
    (set_attr "mode" "V4SF,V4SF,V4SF,V4SF,V4SF,V4SF,SF,DI,DI")])
 
 ;; ??? In theory we can match memory for the MMX alternative, but allowing
-;; nonimmediate_operand for operand 2 and *not* allowing memory for the SSE
+;; vector_operand for operand 2 and *not* allowing memory for the SSE
 ;; alternatives pretty much forces the MMX alternative to be chosen.
 (define_insn "*vec_concatv2sf_sse"
   [(set (match_operand:V2SF 0 "register_operand"     "=x,x,*y,*y")
@@ -8501,7 +8499,7 @@ 
 (define_expand "sse2_shufpd<mask_expand4_name>"
   [(match_operand:V2DF 0 "register_operand")
    (match_operand:V2DF 1 "register_operand")
-   (match_operand:V2DF 2 "nonimmediate_operand")
+   (match_operand:V2DF 2 "vector_operand")
    (match_operand:SI 3 "const_int_operand")]
   "TARGET_SSE2"
 {
@@ -8576,7 +8574,7 @@ 
 	(vec_select:V2DI
 	  (vec_concat:V4DI
 	    (match_operand:V2DI 1 "register_operand" "0,v")
-	    (match_operand:V2DI 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V2DI 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 1)
 		     (const_int 3)])))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
@@ -8626,7 +8624,7 @@ 
 	(vec_select:V2DI
 	  (vec_concat:V4DI
 	    (match_operand:V2DI 1 "register_operand" "0,v")
-	    (match_operand:V2DI 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V2DI 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 0)
 		     (const_int 2)])))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
@@ -8644,7 +8642,7 @@ 
 	(vec_select:VI8F_128
 	  (vec_concat:<ssedoublevecmode>
 	    (match_operand:VI8F_128 1 "register_operand" "0,x")
-	    (match_operand:VI8F_128 2 "nonimmediate_operand" "xm,xm"))
+	    (match_operand:VI8F_128 2 "vector_operand" "xBm,xm"))
 	  (parallel [(match_operand 3 "const_0_to_1_operand")
 		     (match_operand 4 "const_2_to_3_operand")])))]
   "TARGET_SSE2"
@@ -9615,15 +9613,15 @@ 
   [(set (match_operand:VI_AVX2 0 "register_operand")
 	(minus:VI_AVX2
 	  (match_dup 2)
-	  (match_operand:VI_AVX2 1 "nonimmediate_operand")))]
+	  (match_operand:VI_AVX2 1 "vector_operand")))]
   "TARGET_SSE2"
   "operands[2] = force_reg (<MODE>mode, CONST0_RTX (<MODE>mode));")
 
 (define_expand "<plusminus_insn><mode>3"
   [(set (match_operand:VI_AVX2 0 "register_operand")
 	(plusminus:VI_AVX2
-	  (match_operand:VI_AVX2 1 "nonimmediate_operand")
-	  (match_operand:VI_AVX2 2 "nonimmediate_operand")))]
+	  (match_operand:VI_AVX2 1 "vector_operand")
+	  (match_operand:VI_AVX2 2 "vector_operand")))]
   "TARGET_SSE2"
   "ix86_fixup_binary_operands_no_copy (<CODE>, <MODE>mode, operands);")
 
@@ -9652,8 +9650,8 @@ 
 (define_insn "*<plusminus_insn><mode>3"
   [(set (match_operand:VI_AVX2 0 "register_operand" "=x,v")
 	(plusminus:VI_AVX2
-	  (match_operand:VI_AVX2 1 "nonimmediate_operand" "<comm>0,v")
-	  (match_operand:VI_AVX2 2 "nonimmediate_operand" "xBm,vm")))]
+	  (match_operand:VI_AVX2 1 "vector_operand" "<comm>0,v")
+	  (match_operand:VI_AVX2 2 "vector_operand" "xBm,vm")))]
   "TARGET_SSE2
    && ix86_binary_operator_ok (<CODE>, <MODE>mode, operands)"
   "@
@@ -9697,16 +9695,16 @@ 
 (define_expand "<sse2_avx2>_<plusminus_insn><mode>3<mask_name>"
   [(set (match_operand:VI12_AVX2 0 "register_operand")
 	(sat_plusminus:VI12_AVX2
-	  (match_operand:VI12_AVX2 1 "nonimmediate_operand")
-	  (match_operand:VI12_AVX2 2 "nonimmediate_operand")))]
+	  (match_operand:VI12_AVX2 1 "vector_operand")
+	  (match_operand:VI12_AVX2 2 "vector_operand")))]
   "TARGET_SSE2 && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
   "ix86_fixup_binary_operands_no_copy (<CODE>, <MODE>mode, operands);")
 
 (define_insn "*<sse2_avx2>_<plusminus_insn><mode>3<mask_name>"
   [(set (match_operand:VI12_AVX2 0 "register_operand" "=x,v")
 	(sat_plusminus:VI12_AVX2
-	  (match_operand:VI12_AVX2 1 "nonimmediate_operand" "<comm>0,v")
-	  (match_operand:VI12_AVX2 2 "nonimmediate_operand" "xBm,vm")))]
+	  (match_operand:VI12_AVX2 1 "vector_operand" "<comm>0,v")
+	  (match_operand:VI12_AVX2 2 "vector_operand" "xBm,vm")))]
   "TARGET_SSE2 && <mask_mode512bit_condition> && <mask_avx512bw_condition>
    && ix86_binary_operator_ok (<CODE>, <MODE>mode, operands)"
   "@
@@ -9730,15 +9728,15 @@ 
 
 (define_expand "mul<mode>3<mask_name>"
   [(set (match_operand:VI2_AVX2 0 "register_operand")
-	(mult:VI2_AVX2 (match_operand:VI2_AVX2 1 "nonimmediate_operand")
-		       (match_operand:VI2_AVX2 2 "nonimmediate_operand")))]
+	(mult:VI2_AVX2 (match_operand:VI2_AVX2 1 "vector_operand")
+		       (match_operand:VI2_AVX2 2 "vector_operand")))]
   "TARGET_SSE2 && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
   "ix86_fixup_binary_operands_no_copy (MULT, <MODE>mode, operands);")
 
 (define_insn "*mul<mode>3<mask_name>"
   [(set (match_operand:VI2_AVX2 0 "register_operand" "=x,v")
-	(mult:VI2_AVX2 (match_operand:VI2_AVX2 1 "nonimmediate_operand" "%0,v")
-		       (match_operand:VI2_AVX2 2 "nonimmediate_operand" "xm,vm")))]
+	(mult:VI2_AVX2 (match_operand:VI2_AVX2 1 "vector_operand" "%0,v")
+		       (match_operand:VI2_AVX2 2 "vector_operand" "xBm,vm")))]
   "TARGET_SSE2
    && ix86_binary_operator_ok (MULT, <MODE>mode, operands)
    && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
@@ -9757,9 +9755,9 @@ 
 	  (lshiftrt:<ssedoublemode>
 	    (mult:<ssedoublemode>
 	      (any_extend:<ssedoublemode>
-		(match_operand:VI2_AVX2 1 "nonimmediate_operand"))
+		(match_operand:VI2_AVX2 1 "vector_operand"))
 	      (any_extend:<ssedoublemode>
-		(match_operand:VI2_AVX2 2 "nonimmediate_operand")))
+		(match_operand:VI2_AVX2 2 "vector_operand")))
 	    (const_int 16))))]
   "TARGET_SSE2
    && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
@@ -9771,9 +9769,9 @@ 
 	  (lshiftrt:<ssedoublemode>
 	    (mult:<ssedoublemode>
 	      (any_extend:<ssedoublemode>
-		(match_operand:VI2_AVX2 1 "nonimmediate_operand" "%0,v"))
+		(match_operand:VI2_AVX2 1 "vector_operand" "%0,v"))
 	      (any_extend:<ssedoublemode>
-		(match_operand:VI2_AVX2 2 "nonimmediate_operand" "xm,vm")))
+		(match_operand:VI2_AVX2 2 "vector_operand" "xBm,vm")))
 	    (const_int 16))))]
   "TARGET_SSE2
    && ix86_binary_operator_ok (MULT, <MODE>mode, operands)
@@ -9873,11 +9871,11 @@ 
 	(mult:V2DI
 	  (zero_extend:V2DI
 	    (vec_select:V2SI
-	      (match_operand:V4SI 1 "nonimmediate_operand")
+	      (match_operand:V4SI 1 "vector_operand")
 	      (parallel [(const_int 0) (const_int 2)])))
 	  (zero_extend:V2DI
 	    (vec_select:V2SI
-	      (match_operand:V4SI 2 "nonimmediate_operand")
+	      (match_operand:V4SI 2 "vector_operand")
 	      (parallel [(const_int 0) (const_int 2)])))))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
   "ix86_fixup_binary_operands_no_copy (MULT, V4SImode, operands);")
@@ -9887,11 +9885,11 @@ 
 	(mult:V2DI
 	  (zero_extend:V2DI
 	    (vec_select:V2SI
-	      (match_operand:V4SI 1 "nonimmediate_operand" "%0,v")
+	      (match_operand:V4SI 1 "vector_operand" "%0,v")
 	      (parallel [(const_int 0) (const_int 2)])))
 	  (zero_extend:V2DI
 	    (vec_select:V2SI
-	      (match_operand:V4SI 2 "nonimmediate_operand" "xm,vm")
+	      (match_operand:V4SI 2 "vector_operand" "xBm,vm")
 	      (parallel [(const_int 0) (const_int 2)])))))]
   "TARGET_SSE2 && <mask_avx512vl_condition>
    && ix86_binary_operator_ok (MULT, V4SImode, operands)"
@@ -9991,11 +9989,11 @@ 
 	(mult:V2DI
 	  (sign_extend:V2DI
 	    (vec_select:V2SI
-	      (match_operand:V4SI 1 "nonimmediate_operand")
+	      (match_operand:V4SI 1 "vector_operand")
 	      (parallel [(const_int 0) (const_int 2)])))
 	  (sign_extend:V2DI
 	    (vec_select:V2SI
-	      (match_operand:V4SI 2 "nonimmediate_operand")
+	      (match_operand:V4SI 2 "vector_operand")
 	      (parallel [(const_int 0) (const_int 2)])))))]
   "TARGET_SSE4_1 && <mask_avx512vl_condition>"
   "ix86_fixup_binary_operands_no_copy (MULT, V4SImode, operands);")
@@ -10005,11 +10003,11 @@ 
 	(mult:V2DI
 	  (sign_extend:V2DI
 	    (vec_select:V2SI
-	      (match_operand:V4SI 1 "nonimmediate_operand" "%0,0,v")
+	      (match_operand:V4SI 1 "vector_operand" "%0,0,v")
 	      (parallel [(const_int 0) (const_int 2)])))
 	  (sign_extend:V2DI
 	    (vec_select:V2SI
-	      (match_operand:V4SI 2 "nonimmediate_operand" "Yrm,*xm,vm")
+	      (match_operand:V4SI 2 "vector_operand" "YrBm,*xBm,vm")
 	      (parallel [(const_int 0) (const_int 2)])))))]
   "TARGET_SSE4_1 && <mask_avx512vl_condition>
    && ix86_binary_operator_ok (MULT, V4SImode, operands)"
@@ -10113,12 +10111,12 @@ 
 	  (mult:V4SI
 	    (sign_extend:V4SI
 	      (vec_select:V4HI
-		(match_operand:V8HI 1 "nonimmediate_operand")
+		(match_operand:V8HI 1 "vector_operand")
 		(parallel [(const_int 0) (const_int 2)
 			   (const_int 4) (const_int 6)])))
 	    (sign_extend:V4SI
 	      (vec_select:V4HI
-		(match_operand:V8HI 2 "nonimmediate_operand")
+		(match_operand:V8HI 2 "vector_operand")
 		(parallel [(const_int 0) (const_int 2)
 			   (const_int 4) (const_int 6)]))))
 	  (mult:V4SI
@@ -10139,12 +10137,12 @@ 
 	  (mult:V4SI
 	    (sign_extend:V4SI
 	      (vec_select:V4HI
-		(match_operand:V8HI 1 "nonimmediate_operand" "%0,x")
+		(match_operand:V8HI 1 "vector_operand" "%0,x")
 		(parallel [(const_int 0) (const_int 2)
 			   (const_int 4) (const_int 6)])))
 	    (sign_extend:V4SI
 	      (vec_select:V4HI
-		(match_operand:V8HI 2 "nonimmediate_operand" "xm,xm")
+		(match_operand:V8HI 2 "vector_operand" "xBm,xm")
 		(parallel [(const_int 0) (const_int 2)
 			   (const_int 4) (const_int 6)]))))
 	  (mult:V4SI
@@ -10187,9 +10185,9 @@ 
 {
   if (TARGET_SSE4_1)
     {
-      if (!nonimmediate_operand (operands[1], <MODE>mode))
+      if (!vector_operand (operands[1], <MODE>mode))
 	operands[1] = force_reg (<MODE>mode, operands[1]);
-      if (!nonimmediate_operand (operands[2], <MODE>mode))
+      if (!vector_operand (operands[2], <MODE>mode))
 	operands[2] = force_reg (<MODE>mode, operands[2]);
       ix86_fixup_binary_operands_no_copy (MULT, <MODE>mode, operands);
     }
@@ -10203,8 +10201,8 @@ 
 (define_insn "*<sse4_1_avx2>_mul<mode>3<mask_name>"
   [(set (match_operand:VI4_AVX512F 0 "register_operand" "=Yr,*x,v")
 	(mult:VI4_AVX512F
-	  (match_operand:VI4_AVX512F 1 "nonimmediate_operand" "%0,0,v")
-	  (match_operand:VI4_AVX512F 2 "nonimmediate_operand" "Yrm,*xm,vm")))]
+	  (match_operand:VI4_AVX512F 1 "vector_operand" "%0,0,v")
+	  (match_operand:VI4_AVX512F 2 "vector_operand" "YrBm,*xBm,vm")))]
   "TARGET_SSE4_1 && ix86_binary_operator_ok (MULT, <MODE>mode, operands) && <mask_mode512bit_condition>"
   "@
    pmulld\t{%2, %0|%0, %2}
@@ -10256,8 +10254,8 @@ 
 ;; named patterns, but signed V4SI needs special help for plain SSE2.
 (define_expand "vec_widen_smult_even_v4si"
   [(match_operand:V2DI 0 "register_operand")
-   (match_operand:V4SI 1 "nonimmediate_operand")
-   (match_operand:V4SI 2 "nonimmediate_operand")]
+   (match_operand:V4SI 1 "vector_operand")
+   (match_operand:V4SI 2 "vector_operand")]
   "TARGET_SSE2"
 {
   ix86_expand_mul_widen_evenodd (operands[0], operands[1], operands[2],
@@ -10313,8 +10311,8 @@ 
 (define_expand "usadv16qi"
   [(match_operand:V4SI 0 "register_operand")
    (match_operand:V16QI 1 "register_operand")
-   (match_operand:V16QI 2 "nonimmediate_operand")
-   (match_operand:V4SI 3 "nonimmediate_operand")]
+   (match_operand:V16QI 2 "vector_operand")
+   (match_operand:V4SI 3 "vector_operand")]
   "TARGET_SSE2"
 {
   rtx t1 = gen_reg_rtx (V2DImode);
@@ -10654,8 +10652,8 @@ 
 (define_expand "<code><mode>3"
   [(set (match_operand:VI124_128 0 "register_operand")
 	(smaxmin:VI124_128
-	  (match_operand:VI124_128 1 "nonimmediate_operand")
-	  (match_operand:VI124_128 2 "nonimmediate_operand")))]
+	  (match_operand:VI124_128 1 "vector_operand")
+	  (match_operand:VI124_128 2 "vector_operand")))]
   "TARGET_SSE2"
 {
   if (TARGET_SSE4_1 || <MODE>mode == V8HImode)
@@ -10693,8 +10691,8 @@ 
 (define_insn "*sse4_1_<code><mode>3<mask_name>"
   [(set (match_operand:VI14_128 0 "register_operand" "=Yr,*x,v")
 	(smaxmin:VI14_128
-	  (match_operand:VI14_128 1 "nonimmediate_operand" "%0,0,v")
-	  (match_operand:VI14_128 2 "nonimmediate_operand" "Yrm,*xm,vm")))]
+	  (match_operand:VI14_128 1 "vector_operand" "%0,0,v")
+	  (match_operand:VI14_128 2 "vector_operand" "YrBm,*xBm,vm")))]
   "TARGET_SSE4_1
    && <mask_mode512bit_condition>
    && ix86_binary_operator_ok (<CODE>, <MODE>mode, operands)"
@@ -10711,8 +10709,8 @@ 
 (define_insn "*<code>v8hi3"
   [(set (match_operand:V8HI 0 "register_operand" "=x,x")
 	(smaxmin:V8HI
-	  (match_operand:V8HI 1 "nonimmediate_operand" "%0,x")
-	  (match_operand:V8HI 2 "nonimmediate_operand" "xm,xm")))]
+	  (match_operand:V8HI 1 "vector_operand" "%0,x")
+	  (match_operand:V8HI 2 "vector_operand" "xBm,xm")))]
   "TARGET_SSE2 && ix86_binary_operator_ok (<CODE>, V8HImode, operands)"
   "@
    p<maxmin_int>w\t{%2, %0|%0, %2}
@@ -10727,8 +10725,8 @@ 
 (define_expand "<code><mode>3"
   [(set (match_operand:VI124_128 0 "register_operand")
 	(umaxmin:VI124_128
-	  (match_operand:VI124_128 1 "nonimmediate_operand")
-	  (match_operand:VI124_128 2 "nonimmediate_operand")))]
+	  (match_operand:VI124_128 1 "vector_operand")
+	  (match_operand:VI124_128 2 "vector_operand")))]
   "TARGET_SSE2"
 {
   if (TARGET_SSE4_1 || <MODE>mode == V16QImode)
@@ -10777,8 +10775,8 @@ 
 (define_insn "*sse4_1_<code><mode>3<mask_name>"
   [(set (match_operand:VI24_128 0 "register_operand" "=Yr,*x,v")
 	(umaxmin:VI24_128
-	  (match_operand:VI24_128 1 "nonimmediate_operand" "%0,0,v")
-	  (match_operand:VI24_128 2 "nonimmediate_operand" "Yrm,*xm,vm")))]
+	  (match_operand:VI24_128 1 "vector_operand" "%0,0,v")
+	  (match_operand:VI24_128 2 "vector_operand" "YrBm,*xBm,vm")))]
   "TARGET_SSE4_1
    && <mask_mode512bit_condition>
    && ix86_binary_operator_ok (<CODE>, <MODE>mode, operands)"
@@ -10795,8 +10793,8 @@ 
 (define_insn "*<code>v16qi3"
   [(set (match_operand:V16QI 0 "register_operand" "=x,x")
 	(umaxmin:V16QI
-	  (match_operand:V16QI 1 "nonimmediate_operand" "%0,x")
-	  (match_operand:V16QI 2 "nonimmediate_operand" "xm,xm")))]
+	  (match_operand:V16QI 1 "vector_operand" "%0,x")
+	  (match_operand:V16QI 2 "vector_operand" "xBm,xm")))]
   "TARGET_SSE2 && ix86_binary_operator_ok (<CODE>, V16QImode, operands)"
   "@
    p<maxmin_int>b\t{%2, %0|%0, %2}
@@ -10881,8 +10879,8 @@ 
 (define_insn "*sse4_1_eqv2di3"
   [(set (match_operand:V2DI 0 "register_operand" "=Yr,*x,x")
 	(eq:V2DI
-	  (match_operand:V2DI 1 "nonimmediate_operand" "%0,0,x")
-	  (match_operand:V2DI 2 "nonimmediate_operand" "Yrm,*xm,xm")))]
+	  (match_operand:V2DI 1 "vector_operand" "%0,0,x")
+	  (match_operand:V2DI 2 "vector_operand" "YrBm,*xBm,xm")))]
   "TARGET_SSE4_1 && ix86_binary_operator_ok (EQ, V2DImode, operands)"
   "@
    pcmpeqq\t{%2, %0|%0, %2}
@@ -10897,8 +10895,8 @@ 
 (define_insn "*sse2_eq<mode>3"
   [(set (match_operand:VI124_128 0 "register_operand" "=x,x")
 	(eq:VI124_128
-	  (match_operand:VI124_128 1 "nonimmediate_operand" "%0,x")
-	  (match_operand:VI124_128 2 "nonimmediate_operand" "xm,xm")))]
+	  (match_operand:VI124_128 1 "vector_operand" "%0,x")
+	  (match_operand:VI124_128 2 "vector_operand" "xBm,xm")))]
   "TARGET_SSE2 && !TARGET_XOP
    && ix86_binary_operator_ok (EQ, <MODE>mode, operands)"
   "@
@@ -10913,16 +10911,16 @@ 
 (define_expand "sse2_eq<mode>3"
   [(set (match_operand:VI124_128 0 "register_operand")
 	(eq:VI124_128
-	  (match_operand:VI124_128 1 "nonimmediate_operand")
-	  (match_operand:VI124_128 2 "nonimmediate_operand")))]
+	  (match_operand:VI124_128 1 "vector_operand")
+	  (match_operand:VI124_128 2 "vector_operand")))]
   "TARGET_SSE2 && !TARGET_XOP "
   "ix86_fixup_binary_operands_no_copy (EQ, <MODE>mode, operands);")
 
 (define_expand "sse4_1_eqv2di3"
   [(set (match_operand:V2DI 0 "register_operand")
 	(eq:V2DI
-	  (match_operand:V2DI 1 "nonimmediate_operand")
-	  (match_operand:V2DI 2 "nonimmediate_operand")))]
+	  (match_operand:V2DI 1 "vector_operand")
+	  (match_operand:V2DI 2 "vector_operand")))]
   "TARGET_SSE4_1"
   "ix86_fixup_binary_operands_no_copy (EQ, V2DImode, operands);")
 
@@ -10930,7 +10928,7 @@ 
   [(set (match_operand:V2DI 0 "register_operand" "=Yr,*x,x")
 	(gt:V2DI
 	  (match_operand:V2DI 1 "register_operand" "0,0,x")
-	  (match_operand:V2DI 2 "nonimmediate_operand" "Yrm,*xm,xm")))]
+	  (match_operand:V2DI 2 "vector_operand" "YrBm,*xBm,xm")))]
   "TARGET_SSE4_2"
   "@
    pcmpgtq\t{%2, %0|%0, %2}
@@ -10982,7 +10980,7 @@ 
   [(set (match_operand:VI124_128 0 "register_operand" "=x,x")
 	(gt:VI124_128
 	  (match_operand:VI124_128 1 "register_operand" "0,x")
-	  (match_operand:VI124_128 2 "nonimmediate_operand" "xm,xm")))]
+	  (match_operand:VI124_128 2 "vector_operand" "xBm,xm")))]
   "TARGET_SSE2 && !TARGET_XOP"
   "@
    pcmpgt<ssemodesuffix>\t{%2, %0|%0, %2}
@@ -11031,7 +11029,7 @@ 
   [(set (match_operand:V_128 0 "register_operand")
 	(if_then_else:V_128
 	  (match_operator 3 ""
-	    [(match_operand:VI124_128 4 "nonimmediate_operand")
+	    [(match_operand:VI124_128 4 "vector_operand")
 	     (match_operand:VI124_128 5 "general_operand")])
 	  (match_operand:V_128 1)
 	  (match_operand:V_128 2)))]
@@ -11048,7 +11046,7 @@ 
   [(set (match_operand:VI8F_128 0 "register_operand")
 	(if_then_else:VI8F_128
 	  (match_operator 3 ""
-	    [(match_operand:V2DI 4 "nonimmediate_operand")
+	    [(match_operand:V2DI 4 "vector_operand")
 	     (match_operand:V2DI 5 "general_operand")])
 	  (match_operand:VI8F_128 1)
 	  (match_operand:VI8F_128 2)))]
@@ -11097,8 +11095,8 @@ 
   [(set (match_operand:V_128 0 "register_operand")
 	(if_then_else:V_128
 	  (match_operator 3 ""
-	    [(match_operand:VI124_128 4 "nonimmediate_operand")
-	     (match_operand:VI124_128 5 "nonimmediate_operand")])
+	    [(match_operand:VI124_128 4 "vector_operand")
+	     (match_operand:VI124_128 5 "vector_operand")])
 	  (match_operand:V_128 1 "general_operand")
 	  (match_operand:V_128 2 "general_operand")))]
   "TARGET_SSE2
@@ -11114,8 +11112,8 @@ 
   [(set (match_operand:VI8F_128 0 "register_operand")
 	(if_then_else:VI8F_128
 	  (match_operator 3 ""
-	    [(match_operand:V2DI 4 "nonimmediate_operand")
-	     (match_operand:V2DI 5 "nonimmediate_operand")])
+	    [(match_operand:V2DI 4 "vector_operand")
+	     (match_operand:V2DI 5 "vector_operand")])
 	  (match_operand:VI8F_128 1 "general_operand")
 	  (match_operand:VI8F_128 2 "general_operand")))]
   "TARGET_SSE4_2"
@@ -11177,7 +11175,7 @@ 
 
 (define_expand "one_cmpl<mode>2"
   [(set (match_operand:VI 0 "register_operand")
-	(xor:VI (match_operand:VI 1 "nonimmediate_operand")
+	(xor:VI (match_operand:VI 1 "vector_operand")
 		(match_dup 2)))]
   "TARGET_SSE"
 {
@@ -11194,7 +11192,7 @@ 
   [(set (match_operand:VI_AVX2 0 "register_operand")
 	(and:VI_AVX2
 	  (not:VI_AVX2 (match_operand:VI_AVX2 1 "register_operand"))
-	  (match_operand:VI_AVX2 2 "nonimmediate_operand")))]
+	  (match_operand:VI_AVX2 2 "vector_operand")))]
   "TARGET_SSE2")
 
 (define_expand "<sse2_avx2>_andnot<mode>3_mask"
@@ -11223,7 +11221,7 @@ 
   [(set (match_operand:VI 0 "register_operand" "=x,v")
 	(and:VI
 	  (not:VI (match_operand:VI 1 "register_operand" "0,v"))
-	  (match_operand:VI 2 "nonimmediate_operand" "xm,vm")))]
+	  (match_operand:VI 2 "vector_operand" "xBm,vm")))]
   "TARGET_SSE"
 {
   static char buf[64];
@@ -11360,8 +11358,8 @@ 
 (define_insn "<mask_codefor><code><mode>3<mask_name>"
   [(set (match_operand:VI48_AVX_AVX512F 0 "register_operand" "=x,v")
 	(any_logic:VI48_AVX_AVX512F
-	  (match_operand:VI48_AVX_AVX512F 1 "nonimmediate_operand" "%0,v")
-	  (match_operand:VI48_AVX_AVX512F 2 "nonimmediate_operand" "xBm,vm")))]
+	  (match_operand:VI48_AVX_AVX512F 1 "vector_operand" "%0,v")
+	  (match_operand:VI48_AVX_AVX512F 2 "vector_operand" "xBm,vm")))]
   "TARGET_SSE && <mask_mode512bit_condition>
    && ix86_binary_operator_ok (<CODE>, <MODE>mode, operands)"
 {
@@ -11456,8 +11454,8 @@ 
 (define_insn "*<code><mode>3"
   [(set (match_operand:VI12_AVX_AVX512F 0 "register_operand" "=x,v")
 	(any_logic: VI12_AVX_AVX512F
-	  (match_operand:VI12_AVX_AVX512F 1 "nonimmediate_operand" "%0,v")
-	  (match_operand:VI12_AVX_AVX512F 2 "nonimmediate_operand" "xBm,vm")))]
+	  (match_operand:VI12_AVX_AVX512F 1 "vector_operand" "%0,v")
+	  (match_operand:VI12_AVX_AVX512F 2 "vector_operand" "xBm,vm")))]
   "TARGET_SSE && ix86_binary_operator_ok (<CODE>, <MODE>mode, operands)"
 {
   static char buf[64];
@@ -11637,7 +11635,7 @@ 
 	  (ss_truncate:<ssehalfvecmode>
 	    (match_operand:<sseunpackmode> 1 "register_operand" "0,v"))
 	  (ss_truncate:<ssehalfvecmode>
-	    (match_operand:<sseunpackmode> 2 "nonimmediate_operand" "xm,vm"))))]
+	    (match_operand:<sseunpackmode> 2 "vector_operand" "xBm,vm"))))]
   "TARGET_SSE2 && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
   "@
    packsswb\t{%2, %0|%0, %2}
@@ -11654,7 +11652,7 @@ 
 	  (ss_truncate:<ssehalfvecmode>
 	    (match_operand:<sseunpackmode> 1 "register_operand" "0,v"))
 	  (ss_truncate:<ssehalfvecmode>
-	    (match_operand:<sseunpackmode> 2 "nonimmediate_operand" "xm,vm"))))]
+	    (match_operand:<sseunpackmode> 2 "vector_operand" "xBm,vm"))))]
   "TARGET_SSE2 && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
   "@
    packssdw\t{%2, %0|%0, %2}
@@ -11671,7 +11669,7 @@ 
 	  (us_truncate:<ssehalfvecmode>
 	    (match_operand:<sseunpackmode> 1 "register_operand" "0,v"))
 	  (us_truncate:<ssehalfvecmode>
-	    (match_operand:<sseunpackmode> 2 "nonimmediate_operand" "xm,vm"))))]
+	    (match_operand:<sseunpackmode> 2 "vector_operand" "xBm,vm"))))]
   "TARGET_SSE2 && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
   "@
    packuswb\t{%2, %0|%0, %2}
@@ -11759,7 +11757,7 @@ 
 	(vec_select:V16QI
 	  (vec_concat:V32QI
 	    (match_operand:V16QI 1 "register_operand" "0,v")
-	    (match_operand:V16QI 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V16QI 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 8)  (const_int 24)
 		     (const_int 9)  (const_int 25)
 		     (const_int 10) (const_int 26)
@@ -11855,7 +11853,7 @@ 
 	(vec_select:V16QI
 	  (vec_concat:V32QI
 	    (match_operand:V16QI 1 "register_operand" "0,v")
-	    (match_operand:V16QI 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V16QI 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 0) (const_int 16)
 		     (const_int 1) (const_int 17)
 		     (const_int 2) (const_int 18)
@@ -11927,7 +11925,7 @@ 
 	(vec_select:V8HI
 	  (vec_concat:V16HI
 	    (match_operand:V8HI 1 "register_operand" "0,v")
-	    (match_operand:V8HI 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V8HI 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 4) (const_int 12)
 		     (const_int 5) (const_int 13)
 		     (const_int 6) (const_int 14)
@@ -11995,7 +11993,7 @@ 
 	(vec_select:V8HI
 	  (vec_concat:V16HI
 	    (match_operand:V8HI 1 "register_operand" "0,v")
-	    (match_operand:V8HI 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V8HI 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 0) (const_int 8)
 		     (const_int 1) (const_int 9)
 		     (const_int 2) (const_int 10)
@@ -12052,7 +12050,7 @@ 
 	(vec_select:V4SI
 	  (vec_concat:V8SI
 	    (match_operand:V4SI 1 "register_operand" "0,v")
-	    (match_operand:V4SI 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V4SI 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 2) (const_int 6)
 		     (const_int 3) (const_int 7)])))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
@@ -12106,7 +12104,7 @@ 
 	(vec_select:V4SI
 	  (vec_concat:V8SI
 	    (match_operand:V4SI 1 "register_operand" "0,v")
-	    (match_operand:V4SI 2 "nonimmediate_operand" "xm,vm"))
+	    (match_operand:V4SI 2 "vector_operand" "xBm,vm"))
 	  (parallel [(const_int 0) (const_int 4)
 		     (const_int 1) (const_int 5)])))]
   "TARGET_SSE2 && <mask_avx512vl_condition>"
@@ -12773,7 +12771,7 @@ 
 
 (define_expand "sse2_pshufd"
   [(match_operand:V4SI 0 "register_operand")
-   (match_operand:V4SI 1 "nonimmediate_operand")
+   (match_operand:V4SI 1 "vector_operand")
    (match_operand:SI 2 "const_int_operand")]
   "TARGET_SSE2"
 {
@@ -12789,7 +12787,7 @@ 
 (define_insn "sse2_pshufd_1<mask_name>"
   [(set (match_operand:V4SI 0 "register_operand" "=v")
 	(vec_select:V4SI
-	  (match_operand:V4SI 1 "nonimmediate_operand" "vm")
+	  (match_operand:V4SI 1 "vector_operand" "vBm")
 	  (parallel [(match_operand 2 "const_0_to_3_operand")
 		     (match_operand 3 "const_0_to_3_operand")
 		     (match_operand 4 "const_0_to_3_operand")
@@ -12925,7 +12923,7 @@ 
 
 (define_expand "sse2_pshuflw"
   [(match_operand:V8HI 0 "register_operand")
-   (match_operand:V8HI 1 "nonimmediate_operand")
+   (match_operand:V8HI 1 "vector_operand")
    (match_operand:SI 2 "const_int_operand")]
   "TARGET_SSE2"
 {
@@ -12941,7 +12939,7 @@ 
 (define_insn "sse2_pshuflw_1<mask_name>"
   [(set (match_operand:V8HI 0 "register_operand" "=v")
 	(vec_select:V8HI
-	  (match_operand:V8HI 1 "nonimmediate_operand" "vm")
+	  (match_operand:V8HI 1 "vector_operand" "vBm")
 	  (parallel [(match_operand 2 "const_0_to_3_operand")
 		     (match_operand 3 "const_0_to_3_operand")
 		     (match_operand 4 "const_0_to_3_operand")
@@ -13082,7 +13080,7 @@ 
 
 (define_expand "sse2_pshufhw"
   [(match_operand:V8HI 0 "register_operand")
-   (match_operand:V8HI 1 "nonimmediate_operand")
+   (match_operand:V8HI 1 "vector_operand")
    (match_operand:SI 2 "const_int_operand")]
   "TARGET_SSE2"
 {
@@ -13098,7 +13096,7 @@ 
 (define_insn "sse2_pshufhw_1<mask_name>"
   [(set (match_operand:V8HI 0 "register_operand" "=v")
 	(vec_select:V8HI
-	  (match_operand:V8HI 1 "nonimmediate_operand" "vm")
+	  (match_operand:V8HI 1 "vector_operand" "vBm")
 	  (parallel [(const_int 0)
 		     (const_int 1)
 		     (const_int 2)
@@ -13576,9 +13574,9 @@ 
 	    (plus:<ssedoublemode>
 	      (plus:<ssedoublemode>
 		(zero_extend:<ssedoublemode>
-		  (match_operand:VI12_AVX2 1 "nonimmediate_operand"))
+		  (match_operand:VI12_AVX2 1 "vector_operand"))
 		(zero_extend:<ssedoublemode>
-		  (match_operand:VI12_AVX2 2 "nonimmediate_operand")))
+		  (match_operand:VI12_AVX2 2 "vector_operand")))
 	      (match_dup <mask_expand_op3>))
 	    (const_int 1))))]
   "TARGET_SSE2 && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
@@ -13603,9 +13601,9 @@ 
 	    (plus:<ssedoublemode>
 	      (plus:<ssedoublemode>
 		(zero_extend:<ssedoublemode>
-		  (match_operand:VI12_AVX2 1 "nonimmediate_operand" "%0,v"))
+		  (match_operand:VI12_AVX2 1 "vector_operand" "%0,v"))
 		(zero_extend:<ssedoublemode>
-		  (match_operand:VI12_AVX2 2 "nonimmediate_operand" "xm,vm")))
+		  (match_operand:VI12_AVX2 2 "vector_operand" "xBm,vm")))
 	      (match_operand:VI12_AVX2 <mask_expand_op3> "const1_operand"))
 	    (const_int 1))))]
   "TARGET_SSE2 && <mask_mode512bit_condition> && <mask_avx512bw_condition>
@@ -13625,7 +13623,7 @@ 
   [(set (match_operand:VI8_AVX2_AVX512BW 0 "register_operand" "=x,v")
 	(unspec:VI8_AVX2_AVX512BW
 	  [(match_operand:<ssebytemode> 1 "register_operand" "0,v")
-	   (match_operand:<ssebytemode> 2 "nonimmediate_operand" "xm,vm")]
+	   (match_operand:<ssebytemode> 2 "vector_operand" "xBm,vm")]
 	  UNSPEC_PSADBW))]
   "TARGET_SSE2"
   "@
@@ -13891,7 +13889,7 @@ 
 	    (vec_concat:V2HI
 	      (ssse3_plusminus:HI
 		(vec_select:HI
-		  (match_operand:V8HI 2 "nonimmediate_operand" "xm,xm")
+		  (match_operand:V8HI 2 "vector_operand" "xBm,xm")
 		  (parallel [(const_int 0)]))
 		(vec_select:HI (match_dup 2) (parallel [(const_int 1)])))
 	      (ssse3_plusminus:HI
@@ -14004,7 +14002,7 @@ 
 	  (vec_concat:V2SI
 	    (plusminus:SI
 	      (vec_select:SI
-		(match_operand:V4SI 2 "nonimmediate_operand" "xm,xm")
+		(match_operand:V4SI 2 "vector_operand" "xBm,xm")
 		(parallel [(const_int 0)]))
 	      (vec_select:SI (match_dup 2) (parallel [(const_int 1)])))
 	    (plusminus:SI
@@ -14159,7 +14157,7 @@ 
 			   (const_int 12) (const_int 14)])))
 	    (sign_extend:V8HI
 	      (vec_select:V8QI
-		(match_operand:V16QI 2 "nonimmediate_operand" "xm,xm")
+		(match_operand:V16QI 2 "vector_operand" "xBm,xm")
 		(parallel [(const_int 0) (const_int 2)
 			   (const_int 4) (const_int 6)
 			   (const_int 8) (const_int 10)
@@ -14274,9 +14272,9 @@ 
 	      (lshiftrt:<ssedoublemode>
 		(mult:<ssedoublemode>
 		  (sign_extend:<ssedoublemode>
-		    (match_operand:VI2_AVX2 1 "nonimmediate_operand" "%0,v"))
+		    (match_operand:VI2_AVX2 1 "vector_operand" "%0,v"))
 		  (sign_extend:<ssedoublemode>
-		    (match_operand:VI2_AVX2 2 "nonimmediate_operand" "xm,vm")))
+		    (match_operand:VI2_AVX2 2 "vector_operand" "xBm,vm")))
 		(const_int 14))
 	      (match_operand:VI2_AVX2 3 "const1_operand"))
 	    (const_int 1))))]
@@ -14317,7 +14315,7 @@ 
   [(set (match_operand:VI1_AVX512 0 "register_operand" "=x,v")
 	(unspec:VI1_AVX512
 	  [(match_operand:VI1_AVX512 1 "register_operand" "0,v")
-	   (match_operand:VI1_AVX512 2 "nonimmediate_operand" "xm,vm")]
+	   (match_operand:VI1_AVX512 2 "vector_operand" "xBm,vm")]
 	  UNSPEC_PSHUFB))]
   "TARGET_SSSE3 && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
   "@
@@ -14347,7 +14345,7 @@ 
   [(set (match_operand:VI124_AVX2 0 "register_operand" "=x,x")
 	(unspec:VI124_AVX2
 	  [(match_operand:VI124_AVX2 1 "register_operand" "0,x")
-	   (match_operand:VI124_AVX2 2 "nonimmediate_operand" "xm,xm")]
+	   (match_operand:VI124_AVX2 2 "vector_operand" "xBm,xm")]
 	  UNSPEC_PSIGN))]
   "TARGET_SSSE3"
   "@
@@ -14399,7 +14397,7 @@ 
   [(set (match_operand:SSESCALARMODE 0 "register_operand" "=x,v")
 	(unspec:SSESCALARMODE
 	  [(match_operand:SSESCALARMODE 1 "register_operand" "0,v")
-	   (match_operand:SSESCALARMODE 2 "nonimmediate_operand" "xm,vm")
+	   (match_operand:SSESCALARMODE 2 "vector_operand" "xBm,vm")
 	   (match_operand:SI 3 "const_0_to_255_mul_8_operand" "n,n")]
 	  UNSPEC_PALIGNR))]
   "TARGET_SSSE3"
@@ -14454,7 +14452,7 @@ 
 (define_insn "*abs<mode>2"
   [(set (match_operand:VI1248_AVX512VL_AVX512BW 0 "register_operand" "=v")
 	(abs:VI1248_AVX512VL_AVX512BW
-	  (match_operand:VI1248_AVX512VL_AVX512BW 1 "nonimmediate_operand" "vm")))]
+	  (match_operand:VI1248_AVX512VL_AVX512BW 1 "vector_operand" "vBm")))]
   "TARGET_SSSE3"
   "%vpabs<ssemodesuffix>\t{%1, %0|%0, %1}"
   [(set_attr "type" "sselog1")
@@ -14492,7 +14490,7 @@ 
 (define_expand "abs<mode>2"
   [(set (match_operand:VI1248_AVX512VL_AVX512BW 0 "register_operand")
 	(abs:VI1248_AVX512VL_AVX512BW
-	  (match_operand:VI1248_AVX512VL_AVX512BW 1 "nonimmediate_operand")))]
+	  (match_operand:VI1248_AVX512VL_AVX512BW 1 "vector_operand")))]
   "TARGET_SSE2"
 {
   if (!TARGET_SSSE3)
@@ -14606,7 +14604,7 @@ 
 (define_insn "<sse4_1>_blend<ssemodesuffix><avxsizesuffix>"
   [(set (match_operand:VF_128_256 0 "register_operand" "=Yr,*x,x")
 	(vec_merge:VF_128_256
-	  (match_operand:VF_128_256 2 "nonimmediate_operand" "Yrm,*xm,xm")
+	  (match_operand:VF_128_256 2 "vector_operand" "YrBm,*xBm,xm")
 	  (match_operand:VF_128_256 1 "register_operand" "0,0,x")
 	  (match_operand:SI 3 "const_0_to_<blendbits>_operand")))]
   "TARGET_SSE4_1"
@@ -14626,7 +14624,7 @@ 
   [(set (match_operand:VF_128_256 0 "register_operand" "=Yr,*x,x")
 	(unspec:VF_128_256
 	  [(match_operand:VF_128_256 1 "register_operand" "0,0,x")
-	   (match_operand:VF_128_256 2 "nonimmediate_operand" "Yrm,*xm,xm")
+	   (match_operand:VF_128_256 2 "vector_operand" "YrBm,*xBm,xm")
 	   (match_operand:VF_128_256 3 "register_operand" "Yz,Yz,x")]
 	  UNSPEC_BLENDV))]
   "TARGET_SSE4_1"
@@ -14646,8 +14644,8 @@ 
 (define_insn "<sse4_1>_dp<ssemodesuffix><avxsizesuffix>"
   [(set (match_operand:VF_128_256 0 "register_operand" "=Yr,*x,x")
 	(unspec:VF_128_256
-	  [(match_operand:VF_128_256 1 "nonimmediate_operand" "%0,0,x")
-	   (match_operand:VF_128_256 2 "nonimmediate_operand" "Yrm,*xm,xm")
+	  [(match_operand:VF_128_256 1 "vector_operand" "%0,0,x")
+	   (match_operand:VF_128_256 2 "vector_operand" "YrBm,*xBm,xm")
 	   (match_operand:SI 3 "const_0_to_255_operand" "n,n,n")]
 	  UNSPEC_DP))]
   "TARGET_SSE4_1"
@@ -14684,7 +14682,7 @@ 
   [(set (match_operand:VI1_AVX2 0 "register_operand" "=Yr,*x,x")
 	(unspec:VI1_AVX2
 	  [(match_operand:VI1_AVX2 1 "register_operand" "0,0,x")
-	   (match_operand:VI1_AVX2 2 "nonimmediate_operand" "Yrm,*xm,xm")
+	   (match_operand:VI1_AVX2 2 "vector_operand" "YrBm,*xBm,xm")
 	   (match_operand:SI 3 "const_0_to_255_operand" "n,n,n")]
 	  UNSPEC_MPSADBW))]
   "TARGET_SSE4_1"
@@ -14707,7 +14705,7 @@ 
 	  (us_truncate:<ssehalfvecmode>
 	    (match_operand:<sseunpackmode> 1 "register_operand" "0,0,v"))
 	  (us_truncate:<ssehalfvecmode>
-	    (match_operand:<sseunpackmode> 2 "nonimmediate_operand" "Yrm,*xm,vm"))))]
+	    (match_operand:<sseunpackmode> 2 "vector_operand" "YrBm,*xBm,vm"))))]
   "TARGET_SSE4_1 && <mask_mode512bit_condition> && <mask_avx512bw_condition>"
   "@
    packusdw\t{%2, %0|%0, %2}
@@ -14723,7 +14721,7 @@ 
   [(set (match_operand:VI1_AVX2 0 "register_operand" "=Yr,*x,x")
 	(unspec:VI1_AVX2
 	  [(match_operand:VI1_AVX2 1 "register_operand"  "0,0,x")
-	   (match_operand:VI1_AVX2 2 "nonimmediate_operand" "Yrm,*xm,xm")
+	   (match_operand:VI1_AVX2 2 "vector_operand" "YrBm,*xBm,xm")
 	   (match_operand:VI1_AVX2 3 "register_operand" "Yz,Yz,x")]
 	  UNSPEC_BLENDV))]
   "TARGET_SSE4_1"
@@ -14742,7 +14740,7 @@ 
 (define_insn "sse4_1_pblendw"
   [(set (match_operand:V8HI 0 "register_operand" "=Yr,*x,x")
 	(vec_merge:V8HI
-	  (match_operand:V8HI 2 "nonimmediate_operand" "Yrm,*xm,xm")
+	  (match_operand:V8HI 2 "vector_operand" "YrBm,*xBm,xm")
 	  (match_operand:V8HI 1 "register_operand" "0,0,x")
 	  (match_operand:SI 3 "const_0_to_255_operand" "n,n,n")))]
   "TARGET_SSE4_1"
@@ -14803,7 +14801,7 @@ 
 
 (define_insn "sse4_1_phminposuw"
   [(set (match_operand:V8HI 0 "register_operand" "=Yr,*x")
-	(unspec:V8HI [(match_operand:V8HI 1 "nonimmediate_operand" "Yrm,*xm")]
+	(unspec:V8HI [(match_operand:V8HI 1 "vector_operand" "YrBm,*xBm")]
 		     UNSPEC_PHMINPOSUW))]
   "TARGET_SSE4_1"
   "%vphminposuw\t{%1, %0|%0, %1}"
@@ -15063,7 +15061,7 @@ 
 (define_insn "<sse4_1>_ptest<mode>"
   [(set (reg:CC FLAGS_REG)
 	(unspec:CC [(match_operand:V_AVX 0 "register_operand" "Yr, *x, x")
-		    (match_operand:V_AVX 1 "nonimmediate_operand" "Yrm, *xm, xm")]
+		    (match_operand:V_AVX 1 "vector_operand" "YrBm, *xBm, xm")]
 		   UNSPEC_PTEST))]
   "TARGET_SSE4_1"
   "%vptest\t{%1, %0|%0, %1}"
@@ -15081,7 +15079,7 @@ 
 (define_insn "<sse4_1>_round<ssemodesuffix><avxsizesuffix>"
   [(set (match_operand:VF_128_256 0 "register_operand" "=Yr,*x")
 	(unspec:VF_128_256
-	  [(match_operand:VF_128_256 1 "nonimmediate_operand" "Yrm,*xm")
+	  [(match_operand:VF_128_256 1 "vector_operand" "YrBm,*xBm")
 	   (match_operand:SI 2 "const_0_to_15_operand" "n,n")]
 	  UNSPEC_ROUND))]
   "TARGET_ROUND"
@@ -15099,7 +15097,7 @@ 
 
 (define_expand "<sse4_1>_round<ssemodesuffix>_sfix<avxsizesuffix>"
   [(match_operand:<sseintvecmode> 0 "register_operand")
-   (match_operand:VF1_128_256 1 "nonimmediate_operand")
+   (match_operand:VF1_128_256 1 "vector_operand")
    (match_operand:SI 2 "const_0_to_15_operand")]
   "TARGET_ROUND"
 {
@@ -15125,8 +15123,8 @@ 
 
 (define_expand "<sse4_1>_round<ssemodesuffix>_vec_pack_sfix<avxsizesuffix>"
   [(match_operand:<ssepackfltmode> 0 "register_operand")
-   (match_operand:VF2 1 "nonimmediate_operand")
-   (match_operand:VF2 2 "nonimmediate_operand")
+   (match_operand:VF2 1 "vector_operand")
+   (match_operand:VF2 2 "vector_operand")
    (match_operand:SI 3 "const_0_to_15_operand")]
   "TARGET_ROUND"
 {
@@ -16926,7 +16924,7 @@ 
 (define_insn "aesenc"
   [(set (match_operand:V2DI 0 "register_operand" "=x,x")
 	(unspec:V2DI [(match_operand:V2DI 1 "register_operand" "0,x")
-		       (match_operand:V2DI 2 "nonimmediate_operand" "xm,xm")]
+		       (match_operand:V2DI 2 "vector_operand" "xBm,xm")]
 		      UNSPEC_AESENC))]
   "TARGET_AES"
   "@
@@ -16942,7 +16940,7 @@ 
 (define_insn "aesenclast"
   [(set (match_operand:V2DI 0 "register_operand" "=x,x")
 	(unspec:V2DI [(match_operand:V2DI 1 "register_operand" "0,x")
-		       (match_operand:V2DI 2 "nonimmediate_operand" "xm,xm")]
+		       (match_operand:V2DI 2 "vector_operand" "xBm,xm")]
 		      UNSPEC_AESENCLAST))]
   "TARGET_AES"
   "@
@@ -16958,7 +16956,7 @@ 
 (define_insn "aesdec"
   [(set (match_operand:V2DI 0 "register_operand" "=x,x")
 	(unspec:V2DI [(match_operand:V2DI 1 "register_operand" "0,x")
-		       (match_operand:V2DI 2 "nonimmediate_operand" "xm,xm")]
+		       (match_operand:V2DI 2 "vector_operand" "xBm,xm")]
 		      UNSPEC_AESDEC))]
   "TARGET_AES"
   "@
@@ -16974,7 +16972,7 @@ 
 (define_insn "aesdeclast"
   [(set (match_operand:V2DI 0 "register_operand" "=x,x")
 	(unspec:V2DI [(match_operand:V2DI 1 "register_operand" "0,x")
-		       (match_operand:V2DI 2 "nonimmediate_operand" "xm,xm")]
+		       (match_operand:V2DI 2 "vector_operand" "xBm,xm")]
 		      UNSPEC_AESDECLAST))]
   "TARGET_AES"
   "@
@@ -16989,7 +16987,7 @@ 
 
 (define_insn "aesimc"
   [(set (match_operand:V2DI 0 "register_operand" "=x")
-	(unspec:V2DI [(match_operand:V2DI 1 "nonimmediate_operand" "xm")]
+	(unspec:V2DI [(match_operand:V2DI 1 "vector_operand" "xBm")]
 		      UNSPEC_AESIMC))]
   "TARGET_AES"
   "%vaesimc\t{%1, %0|%0, %1}"
@@ -17000,7 +16998,7 @@ 
 
 (define_insn "aeskeygenassist"
   [(set (match_operand:V2DI 0 "register_operand" "=x")
-	(unspec:V2DI [(match_operand:V2DI 1 "nonimmediate_operand" "xm")
+	(unspec:V2DI [(match_operand:V2DI 1 "vector_operand" "xBm")
 		      (match_operand:SI 2 "const_0_to_255_operand" "n")]
 		     UNSPEC_AESKEYGENASSIST))]
   "TARGET_AES"
@@ -17014,7 +17012,7 @@ 
 (define_insn "pclmulqdq"
   [(set (match_operand:V2DI 0 "register_operand" "=x,x")
 	(unspec:V2DI [(match_operand:V2DI 1 "register_operand" "0,x")
-		      (match_operand:V2DI 2 "nonimmediate_operand" "xm,xm")
+		      (match_operand:V2DI 2 "vector_operand" "xBm,xm")
 		      (match_operand:SI 3 "const_0_to_255_operand" "n,n")]
 		     UNSPEC_PCLMUL))]
   "TARGET_PCLMUL"
@@ -19152,7 +19150,7 @@ 
   [(set (match_operand:V4SI 0 "register_operand" "=x")
 	(unspec:V4SI
 	  [(match_operand:V4SI 1 "register_operand" "0")
-	   (match_operand:V4SI 2 "nonimmediate_operand" "xm")]
+	   (match_operand:V4SI 2 "vector_operand" "xBm")]
 	  UNSPEC_SHA1MSG1))]
   "TARGET_SHA"
   "sha1msg1\t{%2, %0|%0, %2}"
@@ -19163,7 +19161,7 @@ 
   [(set (match_operand:V4SI 0 "register_operand" "=x")
 	(unspec:V4SI
 	  [(match_operand:V4SI 1 "register_operand" "0")
-	   (match_operand:V4SI 2 "nonimmediate_operand" "xm")]
+	   (match_operand:V4SI 2 "vector_operand" "xBm")]
 	  UNSPEC_SHA1MSG2))]
   "TARGET_SHA"
   "sha1msg2\t{%2, %0|%0, %2}"
@@ -19174,7 +19172,7 @@ 
   [(set (match_operand:V4SI 0 "register_operand" "=x")
 	(unspec:V4SI
 	  [(match_operand:V4SI 1 "register_operand" "0")
-	   (match_operand:V4SI 2 "nonimmediate_operand" "xm")]
+	   (match_operand:V4SI 2 "vector_operand" "xBm")]
 	  UNSPEC_SHA1NEXTE))]
   "TARGET_SHA"
   "sha1nexte\t{%2, %0|%0, %2}"
@@ -19185,7 +19183,7 @@ 
   [(set (match_operand:V4SI 0 "register_operand" "=x")
 	(unspec:V4SI
 	  [(match_operand:V4SI 1 "register_operand" "0")
-	   (match_operand:V4SI 2 "nonimmediate_operand" "xm")
+	   (match_operand:V4SI 2 "vector_operand" "xBm")
 	   (match_operand:SI 3 "const_0_to_3_operand" "n")]
 	  UNSPEC_SHA1RNDS4))]
   "TARGET_SHA"
@@ -19198,7 +19196,7 @@ 
   [(set (match_operand:V4SI 0 "register_operand" "=x")
 	(unspec:V4SI
 	  [(match_operand:V4SI 1 "register_operand" "0")
-	   (match_operand:V4SI 2 "nonimmediate_operand" "xm")]
+	   (match_operand:V4SI 2 "vector_operand" "xBm")]
 	  UNSPEC_SHA256MSG1))]
   "TARGET_SHA"
   "sha256msg1\t{%2, %0|%0, %2}"
@@ -19209,7 +19207,7 @@ 
   [(set (match_operand:V4SI 0 "register_operand" "=x")
 	(unspec:V4SI
 	  [(match_operand:V4SI 1 "register_operand" "0")
-	   (match_operand:V4SI 2 "nonimmediate_operand" "xm")]
+	   (match_operand:V4SI 2 "vector_operand" "xBm")]
 	  UNSPEC_SHA256MSG2))]
   "TARGET_SHA"
   "sha256msg2\t{%2, %0|%0, %2}"
@@ -19220,7 +19218,7 @@ 
   [(set (match_operand:V4SI 0 "register_operand" "=x")
 	(unspec:V4SI
 	  [(match_operand:V4SI 1 "register_operand" "0")
-	   (match_operand:V4SI 2 "nonimmediate_operand" "xm")
+	   (match_operand:V4SI 2 "vector_operand" "xBm")
 	   (match_operand:V4SI 3 "register_operand" "Yz")]
 	  UNSPEC_SHA256RNDS2))]
   "TARGET_SHA"
diff --git a/gcc/config/i386/subst.md b/gcc/config/i386/subst.md
index b283dd8..d35f34c 100644
--- a/gcc/config/i386/subst.md
+++ b/gcc/config/i386/subst.md
@@ -123,7 +123,7 @@ 
 (define_subst_attr "round_constraint" "round" "vm" "v")
 (define_subst_attr "round_constraint2" "round" "m" "v")
 (define_subst_attr "round_constraint3" "round" "rm" "r")
-(define_subst_attr "round_nimm_predicate" "round" "nonimmediate_operand" "register_operand")
+(define_subst_attr "round_nimm_predicate" "round" "vector_operand" "register_operand")
 (define_subst_attr "round_prefix" "round" "vex" "evex")
 (define_subst_attr "round_mode512bit_condition" "round" "1" "(<MODE>mode == V16SFmode
 							      || <MODE>mode == V8DFmode
@@ -162,7 +162,8 @@ 
 (define_subst_attr "round_saeonly_sd_mask_op5" "round_saeonly" "" "<round_saeonly_sd_mask_operand5>")
 (define_subst_attr "round_saeonly_constraint" "round_saeonly" "vm" "v")
 (define_subst_attr "round_saeonly_constraint2" "round_saeonly" "m" "v")
-(define_subst_attr "round_saeonly_nimm_predicate" "round_saeonly" "nonimmediate_operand" "register_operand")
+(define_subst_attr "round_saeonly_nimm_predicate" "round_saeonly" "vector_operand" "register_operand")
+(define_subst_attr "round_saeonly_nimm_scalar_predicate" "round_saeonly" "nonimmediate_operand" "register_operand")
 (define_subst_attr "round_saeonly_mode512bit_condition" "round_saeonly" "1" "(<MODE>mode == V16SFmode
 									      || <MODE>mode == V8DFmode
 									      || <MODE>mode == V8DImode
diff --git a/gcc/testsuite/gcc.target/i386/pr68991.c b/gcc/testsuite/gcc.target/i386/pr68991.c
new file mode 100644
index 0000000..3fc5ac8
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/pr68991.c
@@ -0,0 +1,32 @@ 
+/* { dg-do compile } */
+/* { dg-options "-O2 -msse2 -mfpmath=sse" } */
+
+float
+foo (float a, float b, float c, float d)
+{
+  float ac, bd, ad, bc, y;
+
+  ac = a * c;
+  bd = b * d;
+  ad = a * d;
+  bc = b * c;
+
+  if (__builtin_expect (!__builtin_expect ((a) != (a), 0)
+			& !__builtin_expect (!__builtin_expect (((a) - (a)) != ((a) - (a)), 0), 1), 0)
+      || __builtin_expect (!__builtin_expect ((b) != (b), 0)
+			   & !__builtin_expect (!__builtin_expect (((b) - (b)) != ((b) - (b)), 0), 1), 0))
+    a = __builtin_copysignf (__builtin_expect (!__builtin_expect ((a) != (a), 0)
+					       & !__builtin_expect (!__builtin_expect (((a) - (a)) != ((a) - (a)), 0), 1), 0) ? 1 : 0, a);
+
+  c = __builtin_copysignf (__builtin_expect (!__builtin_expect ((c) != (c), 0) & !__builtin_expect (!__builtin_expect (((c) - (c)) != ((c) - (c)), 0), 1), 0) ? 1 : 0, c);
+  if ((__builtin_expect (!__builtin_expect ((ac) != (ac), 0)
+			 & !__builtin_expect (!__builtin_expect (((ac) - (ac)) != ((ac) - (ac)), 0), 1), 0)
+       || __builtin_expect (!__builtin_expect ((bd) != (bd), 0)
+			    & !__builtin_expect (!__builtin_expect (((bd) - (bd)) != ((bd) - (bd)), 0), 1), 0)
+       || __builtin_expect (!__builtin_expect ((bc) != (bc), 0) & !__builtin_expect (!__builtin_expect (((bc) - (bc)) != ((bc) - (bc)), 0), 1), 0)))
+    d = __builtin_copysignf (0, d);
+
+  y = a * d + b * c;
+
+  return y;
+}
-- 
2.5.0