diff mbox

PowerPC IEEE 128-bit patch #7 (revised #2)

Message ID 20151008190345.GA31984@ibm-tiger.the-meissners.org
State New
Headers show

Commit Message

Michael Meissner Oct. 8, 2015, 7:03 p.m. UTC
This is the revised patch #7 for the IEEE 128-bit software floating point
support in the PowerPC. I have tested this on big endian power7 (both 32/64
bit) and little endian power8 with no regressions. Is this ok to install in the
trunk?

This patch should allow __float128 support on VSX systems, but it won't be
usable until patch #8 goes in that adds support in libgcc. It is expected that
we will need to update the glibc sources (which are where the master definition
for the soft-fp directory comes from) to add kf mode support for IEEE 128-bit
floating point. I have intermediate patches that add the support by using sed
to transform the <...>tf<...> names into <...>kf<...>.

At a high level, the changes in this patch are:

1) As we discussed previously, the option name is changed from
-mfloat128-software and -mfloat128-none to be -mfloat128 and -mno-float128.

2) Add defines so that the user can determine if __float128 is available, and
what is the default format for long double.

3) I removed the code I previously had to disallow IFmode/KFmode if -mfloat128
is not enabled.

4) Like other vector types, don't allow pre-increment and pre-decrement memory
forms that reload eventually has to fix up.

5) Add IEEE 128-bit register allocation, and call/return as vector types. In a
few places, I had to modify tests against VECTOR_TYPE_P to use
ALTIVEC_OR_VSX_VECTOR_MODE, since the IEEE 128-bit floating point types aren't
considered a vector type by the machine independent portion of the compiler.

6) Add support for all of the emulator names. In this version of the patch, I
added support for converting between the various decimal types and __float128
(but so far those new emulated functions have not been written). In addition, I
modified the compare function to have 2 variants, __cmpukf2 which is intended
to mimic the behavior of the 'fcmpu' instruction, and __cmpokf2 which is
intended to mimic the behaviour of the 'fcmpo' instruction. At the present
time, I have not written the __cmpokf2 function, and the compiler does not
currently generated 'fcmpo'.

7) I added support for the 'q' and 'Q' suffix for explict __float128 constants.

8) I reordered the tests in MODES_TIEABLE_P so that vector types are tested
before scalar floating point types, so that __float128 pairs with vector
double, but not with long double (if IBM extended double is the long double
format) and __ibm128.

9) I updated the documentation for the new switch and for the __float128 and
__ibm128 keywords.

10) I went through rs6000.md and cloned all of the support for the 128-bit
types, so that you can use the right operation on KFmode for IEEE types, IFmode
for IBM types, and TFmode will be treated as KFmode if -mabi=ieeelongdouble and
IFmode if -mabi=ibmlongdouble.

[gcc]
2015-10-06  Michael Meissner  <meissner@linux.vnet.ibm.com>

	* config/rs6000/rs6000-cpus.def (POWERPC_MASKS): Add -mfloat128.

	* config/rs6000/rs6000.opt (-mfloat128): Replace the switches
	-mfloat128-{software,none} with -m{,no-}float128.

	* config/rs6000/rs6000-c.c (rs6000_cpu_cpp_builtins): Add defines
	to tell users whether __float128 is available, and whether long
	double is IBM extended double or IEEE 128-bit floating point.

	* config/rs6000/rs6000.c (TARGET_C_MODE_FOR_SUFFIX): Define.
	(rs6000_hard_regno_mode_ok): Remove code not allowing 128-bit
	floating types from any register if the appropriate option is not
	used.
	(rs6000_debug_reg_global): Delete -mfloat128-{software,none}
	debugging.
	(rs6000_setup_reg_addr_masks): Don't allow pre-increment or
	pre-decrement on IEEE 128-bit floating point.
	(rs6000_init_hard_regno_mode_ok): Add support for IEEE 128-bit
	floating point types.
	(rs6000_option_override_internal): Replace the switches
	-mfloat128-{software,none} with -m{,no-}float128, and move the
	tests to be closer to other ISA 2.06/2.07 options.
	(rs6000_gen_le_vsx_permute): Add support for IEEE 128-bit floating
	point.
	(init_cumulative_args): Assume that IEEE 128-bit floating point
	emulation routines have prototypes, and don't need to pass the
	arguments in the save area as well as the Altivec register set.
	(rs6000_function_arg): Likewise.
	(rs6000_arg_partial_bytes): Likewise.
	(rs6000_init_builtins): Set up ieee128_float_type_node to always
	use KFmode, and ibm128_float_type_node to always use IFmode.
	(init_float128_ibm): Split rs6000_init_libfuncs into 2 functions,
	one for IEEE 128-bit floating point, and one for IBM extended
	double floating point. For IFmode, use the traditional IBM
	extended double names. For TFmode, if -mabi=ieeelongdouble, use
	the names for KFmode.
	(init_float128_ieee): Likewise.
	(rs6000_init_libfuncs): Likewise.
	(rs6000_generate_compare): For IEEE 128-bit floating point, use
	the compare function that matches the fcmpu instruction.
	(rs6000_expand_float128_convert): Restructure code. Add support
	for using plain move if the underlying type is the same. Deal with
	src and dest being in memory.
	(rs6000_mangle_type): Use U10__float128 for IEEE 128-bit floating
	point.
	(rs6000_scalar_mode_supported_p): Add IFmode if -mfloat128.
	(rs6000_c_mode_for_suffix): Add 'q' and 'Q' suffixes for IEEE
	128-bit floating point.
	(rs6000_opt_masks): Add -mfloat128.
	(struct rs6000_opt_var): Fix typo in comment.
	(chain_contains_only_swaps): Change from VECTOR_MODE_P to
	ALTIVEC_OR_VSX_VECTOR_MODE to add IEEE 128-bit floating point
	values in VSX registers.
	(mark_swaps_for_removal): Likewise.
	(rs6000_analyze_swaps): Likewise.

	* config/rs6000/rs6000.h (ALTIVEC_VECTOR_MODE): Add IEEE 128-bit
	floating point types that go in VSX registers.
	(MODES_TIEABLE_P): Move vector tests above floating point scalar
	tests, so that IEEE 128-bit floating point values in VSX tie with
	other vector types.
	(struct rs6000_args): Add libcall field.

	* config/rs6000/rs6000.md (FP iterator): Allow IEEE 128-bit
	floating point.
	(FLOAT128_SFDFTF iterator): Delete, rework conversion insns.
	(IFKF iterator): Delete, replace with IBM128 iterator. Add
	explicit FLOAT128_IBM_P (<mode>) tests.
	(IBM128 iterator): Likewise.
	(IEEE128 iterator): New iterator for IEEE 128-bit floating point
	support.
	(TFIFKF iterator): Delete, replace with FLOAT128 iterator.
	(FLOAT128 iterator): Likewise.
	(signbit<mode>2): Add IEEE 128-bit floating point support.
	(extenddftf2): 	Rework IEEE 128-bit floating point code. Use
	iterators to make IFmode and KFmode patterns to mirror the TFmode
	patterns. Delete FLOAT128_SFDFTF, IFKF and TFIFKF iterators. Add
	IEEE128, and FLOAT128 iterators.
	(extenddf<mode>2): Likewise.
	(extenddftf2_fprs): Likewise.
	(extenddf<mode>2_fprs): Likewise.
	(extenddf<mode>2_vsx): Likewise.
	(extenddftf2_internal): Likewise.
	(extendsftf2): Likewise.
	(extendsf<mode>2): Likewise.
	(trunctfdf2): Rework 128-bit floating point truncation patterns.
	(trunc<mode>df2): Likewise.
	(trunctfdf2_internal1): Likewise.
	(trunc<mode>df2_internal1): Likewise.
	(trunctfdf2_internal2): Likewise.
	(trunc<mode>df2_internal2): Likewise.
	(trunctfsf2): Likewise.
	(trunc<mode>sf2): Likewise.
	(trunctfsf2_fprs): Likewise.
	(trunc<mode>sf2_fprs): Likewise.
	(floatsitf2): Rework conversion patterns from 32-bit integer types
	to 128-bit floating point.
	(floatsi<mode>2): Likewise.
	(fix_trunc_helper): Rework conversions patterns from 128-bit
	floating point types to integers.
	(fix_trunc_helper<mode>): Likewise.
	(fix_trunctfsi2): Likewise.
	(fix_trunc<mode>si2): Likewise.
	(fix_trunctfsi2_fprs): Likewise.
	(fix_trunc<mode>si2_fprs): Likewise.
	(fix_trunctfsi2_internal): Likewise.
	(fix_trunc<mode>si2_internal): Likewise.
	(fix_trunctfdi2): Likewise.
	(fix_trunc<mode>di2): Likewise.
	(fixuns_trunctf<mode>2): Likewise.
	(fixuns_trunc<IEEE128:mode><SDI:mode>2): Likewise.
	(floatditf2): Rework conversion patterns from 64-bit integer types
	to 128-bit floating point.
	(floatdi<mode>2): Likewise.
	(floatuns<mode>tf2): Likewise.
	(floatuns<IEEE128:mode><SDI:mode>2): Likewise.
	(extend<FLOAT128_SFDFTF:mode><IFKF:mode>2): Rework conversions
	between 128-bit floating point types.
	(extendifkf2): Likewise.
	(trunc<IFKF:mode><FLOAT128_SFDFTF:mode>2): Likewise.
	(extendtfkf2): Likewise.
	(fix_trunc<IFKF:mode><SDI:mode>2): Likewise.
	(extendtfkf2): Likewise.
	(fix_trunc<IFKF:mode><SDI:mode>2): Likewise.
	(trunciftf2): Likewise.
	(fixuns_trunc<IFKF:mode><SDI:mode>2): Likewise.
	(truncifkf2): Likewise.
	(float<SDI:mode><IFKF:mode>2): Likewise.
	(trunckftf2): Likewise.
	(floatuns<SDI:mode><IFKF:mode>2): Likewise.
	(trunctfif2): Likewise.
	(neg<mode>2): Rework IEEE 128-bit floating point negation and
	absolute value.
	(negtf2_internal): Likewise.
	(neg<mode>2_internal): Likewise.
	(abs<mode>2): Likewise.
	(abstf2_internal): Likewise.
	(abs<mode>2_internal): Likewise.
	(ieee_128bit_vsx_neg<mode>2): Likewise.
	(ieee_128bit_vsx_neg<mode>2_internal): Likewise.
	(ieee_128bit_vsx_abs<mode>2): Likewise.
	(ieee_128bit_vsx_abs<mode>2_internal): Likewise.
	(ieee_128bit_vsx_nabs<mode>2): Likewise.
	(ieee_128bit_vsx_nabs<mode>2_internal): Likewise.
	(cmptf_internal1): Add IFmode 128-bit comparison.
	(cmp<mode>_internal1): Likewise.

	* config/rs6000/rs6000-opts.h (enum float128_type_t):
	Delete. Switch from -mfloat128-{none,software} to
	-m{,no-}float128.

	* config/rs6000/predicates.md (reg_or_indexed_operand): Allow
	SUBREG's.

	* doc/extend.texi (additional floating types): Document PowerPC
	use of __float128 and __ibm128 types.

	* doc/invoke.texi (RS/6000 and PowerPC Options): Document
	-mfloat128 and -mno-float128.

[gcc/testsuite]
2015-10-06  Michael Meissner  <meissner@linux.vnet.ibm.com>

	* gcc.target/powerpc/float128-call.c: New function, test whether
	IEEE 128-bit floating point in VSX registers is passed correctly.

Comments

Joseph Myers Oct. 8, 2015, 9:30 p.m. UTC | #1
Question: what happens if you mix __float128 and __ibm128 in an arithmetic 
or conditional expression?

__float128 a;
__ibm128 b;
int x;
/* ... */
a + b;
x ? a : b;

(And likewise if one or both are the corresponding complex types.)  As I 
suggested in <https://gcc.gnu.org/ml/gcc/2014-06/msg00033.html> I think 
this would best be rejected for both C and C++ (with testcases).  That 
accords with TS 18661-3 (just published) making floating-point conversions 
undefined where neither type's set of values is a subset of the other's.

The invalid_binary_op target hook should be usable to cover the case where 
a binary operation would cause implicit conversions, but I don't see an 
existing hook that would deal with ? : expressions.
Michael Meissner Oct. 13, 2015, 6:10 p.m. UTC | #2
On Thu, Oct 08, 2015 at 09:30:45PM +0000, Joseph Myers wrote:
> Question: what happens if you mix __float128 and __ibm128 in an arithmetic 
> or conditional expression?
> 
> __float128 a;
> __ibm128 b;
> int x;
> /* ... */
> a + b;
> x ? a : b;
> 
> (And likewise if one or both are the corresponding complex types.)  As I 
> suggested in <https://gcc.gnu.org/ml/gcc/2014-06/msg00033.html> I think 
> this would best be rejected for both C and C++ (with testcases).  That 
> accords with TS 18661-3 (just published) making floating-point conversions 
> undefined where neither type's set of values is a subset of the other's.
> 
> The invalid_binary_op target hook should be usable to cover the case where 
> a binary operation would cause implicit conversions, but I don't see an 
> existing hook that would deal with ? : expressions.

Good points. I hadn't delved into error conditions, but I will code up a target
hook not allowing inter-mixing different formats.

I believe every non-NaN value that IBM extended double supports is
representable in IEEE 754R 128-bit floating point, since IEEE has 112 bits of
mantissa plus the hidden bit, while IBM extended double has 106 bits (52 bits
of mantissa for each part plus 2 hidden bits). Even with all of the extra bits
that you can hand craft into silent/signalling NaNs, you should be able
represent those values in IEEE 128-bit floating point as similar NaNs.
Joseph Myers Oct. 13, 2015, 11:44 p.m. UTC | #3
On Tue, 13 Oct 2015, Michael Meissner wrote:

> I believe every non-NaN value that IBM extended double supports is
> representable in IEEE 754R 128-bit floating point, since IEEE has 112 bits of
> mantissa plus the hidden bit, while IBM extended double has 106 bits (52 bits
> of mantissa for each part plus 2 hidden bits). Even with all of the extra bits

No, because IBM long double can represent values with discontiguous 
mantissa bits (these are the values that 
libgcc/config/rs6000/ibm-ldouble-format describes as denormal but not 
subnormal).

> that you can hand craft into silent/signalling NaNs, you should be able
> represent those values in IEEE 128-bit floating point as similar NaNs.

The low part of a NaN in IBM long double is documented as don't-care (so 
all IBM long double NaNs are correctly converted to binary128 simply by 
converting the high part - as is also the case with zeroes).
diff mbox

Patch

Index: gcc/config/rs6000/rs6000-cpus.def
===================================================================
--- gcc/config/rs6000/rs6000-cpus.def	(revision 228506)
+++ gcc/config/rs6000/rs6000-cpus.def	(working copy)
@@ -80,6 +80,7 @@ 
 				 | OPTION_MASK_DIRECT_MOVE		\
 				 | OPTION_MASK_DLMZB			\
 				 | OPTION_MASK_EFFICIENT_UNALIGNED_VSX	\
+				 | OPTION_MASK_FLOAT128			\
 				 | OPTION_MASK_FPRND			\
 				 | OPTION_MASK_HTM			\
 				 | OPTION_MASK_ISEL			\
Index: gcc/config/rs6000/rs6000.opt
===================================================================
--- gcc/config/rs6000/rs6000.opt	(revision 228506)
+++ gcc/config/rs6000/rs6000.opt	(working copy)
@@ -601,18 +601,6 @@  moptimize-swaps
 Target Undocumented Var(rs6000_optimize_swaps) Init(1) Save
 Analyze and remove doubleword swaps from VSX computations.
 
-mfloat128-
-Target RejectNegative Joined Enum(float128_type_t) Var(TARGET_FLOAT128) Init(FLOAT128_UNSET) Save
--mfloat128-{software,none} - Specify how IEEE 128-bit floating point is used.
-
-Enum
-Name(float128_type_t) Type(enum float128_type_t)
-
-EnumValue
-Enum(float128_type_t) String(none) Value(FLOAT128_NONE)
-
-EnumValue
-Enum(float128_type_t) String(software) Value(FLOAT128_SW)
-
-EnumValue
-Enum(float128_type_t) String(sw) Value(FLOAT128_SW)
+mfloat128
+Target Report Mask(FLOAT128) Var(rs6000_isa_flags)
+Enable/disable IEEE 128-bit floating point via the __float128 keyword.
Index: gcc/config/rs6000/rs6000-c.c
===================================================================
--- gcc/config/rs6000/rs6000-c.c	(revision 228506)
+++ gcc/config/rs6000/rs6000-c.c	(working copy)
@@ -410,6 +410,8 @@  rs6000_cpu_cpp_builtins (cpp_reader *pfi
     builtin_define ("__RSQRTE__");
   if (TARGET_FRSQRTES)
     builtin_define ("__RSQRTEF__");
+  if (TARGET_FLOAT128)
+    builtin_define ("__FLOAT128__");
 
   if (TARGET_EXTRA_BUILTINS && cpp_get_options (pfile)->lang != CLK_ASM)
     {
@@ -483,6 +485,11 @@  rs6000_cpu_cpp_builtins (cpp_reader *pfi
     {
       builtin_define ("__LONG_DOUBLE_128__");
       builtin_define ("__LONGDOUBLE128");
+
+      if (TARGET_IEEEQUAD)
+	builtin_define ("__LONG_DOUBLE_IEEE128__");
+      else
+	builtin_define ("__LONG_DOUBLE_IBM128__");
     }
 
   switch (TARGET_CMODEL)
Index: gcc/config/rs6000/rs6000.c
===================================================================
--- gcc/config/rs6000/rs6000.c	(revision 228506)
+++ gcc/config/rs6000/rs6000.c	(working copy)
@@ -1690,6 +1690,9 @@  static const struct attribute_spec rs600
 #define TARGET_LIBGCC_SHIFT_COUNT_MODE rs6000_abi_word_mode
 #undef TARGET_UNWIND_WORD_MODE
 #define TARGET_UNWIND_WORD_MODE rs6000_abi_word_mode
+
+#undef TARGET_C_MODE_FOR_SUFFIX
+#define TARGET_C_MODE_FOR_SUFFIX rs6000_c_mode_for_suffix
 
 
 /* Processor table.  */
@@ -1788,16 +1791,6 @@  rs6000_hard_regno_mode_ok (int regno, ma
 	    && IN_RANGE (last_regno, FIRST_GPR_REGNO, LAST_GPR_REGNO)
 	    && ((regno & 1) == 0));
 
-  /* If we don't allow 128-bit binary floating point, disallow the 128-bit
-     types from going in any registers.  Similarly if __float128 is not
-     supported, don't allow __float128/__ibm128 types.  */
-  if (!TARGET_LONG_DOUBLE_128
-      && (mode == TFmode || mode == KFmode || mode == IFmode))
-    return false;
-
-  if (!TARGET_FLOAT128 && (mode == KFmode || mode == IFmode))
-    return false;
-
   /* VSX registers that overlap the FPR registers are larger than for non-VSX
      implementations.  Don't allow an item to be split between a FP register
      and an Altivec register.  Allow TImode in all VSX registers if the user
@@ -2071,7 +2064,6 @@  rs6000_debug_reg_global (void)
   const char *trace_str;
   const char *abi_str;
   const char *cmodel_str;
-  const char *float128_str;
   struct cl_target_option cl_opts;
 
   /* Modes we want tieable information on.  */
@@ -2437,15 +2429,6 @@  rs6000_debug_reg_global (void)
   fprintf (stderr, DEBUG_FMT_S, "e500_double",
 	   (TARGET_E500_DOUBLE ? "true" : "false"));
 
-  switch (TARGET_FLOAT128)
-    {
-    case FLOAT128_NONE:	float128_str = "none";		break;
-    case FLOAT128_SW:	float128_str = "software";	break;
-    default:		float128_str = "unknown";	break;
-    }
-
-  fprintf (stderr, DEBUG_FMT_S, "float128", float128_str);
-
   if (TARGET_LINK_STACK)
     fprintf (stderr, DEBUG_FMT_S, "link_stack", "true");
 
@@ -2528,6 +2511,7 @@  rs6000_setup_reg_addr_masks (void)
 		  && (rc == RELOAD_REG_GPR || rc == RELOAD_REG_FPR)
 		  && GET_MODE_SIZE (m2) <= 8
 		  && !VECTOR_MODE_P (m2)
+		  && !FLOAT128_VECTOR_P (m2)
 		  && !COMPLEX_MODE_P (m2)
 		  && !indexed_only_p
 		  && !(TARGET_E500_DOUBLE && GET_MODE_SIZE (m2) == 8))
@@ -2673,6 +2657,20 @@  rs6000_init_hard_regno_mode_ok (bool glo
       align32 = 128;
     }
 
+  /* KF mode (IEEE 128-bit in VSX registers).  We do not have arithmetic, so
+     only set the memory modes.  Include TFmode if -mabi=ieeelongdouble.  */
+  if (TARGET_FLOAT128)
+    {
+      rs6000_vector_mem[KFmode] = VECTOR_VSX;
+      rs6000_vector_align[KFmode] = 128;
+
+      if (FLOAT128_IEEE_P (TFmode))
+	{
+	  rs6000_vector_mem[TFmode] = VECTOR_VSX;
+	  rs6000_vector_align[TFmode] = 128;
+	}
+    }
+
   /* V2DF mode, VSX only.  */
   if (TARGET_VSX)
     {
@@ -2866,7 +2864,7 @@  rs6000_init_hard_regno_mode_ok (bool glo
   if (TARGET_FLOAT128)
     {
       rs6000_constraints[RS6000_CONSTRAINT_wq] = VSX_REGS;	/* KFmode  */
-      if (rs6000_ieeequad)
+      if (FLOAT128_IEEE_P (TFmode))
 	rs6000_constraints[RS6000_CONSTRAINT_wp] = VSX_REGS;	/* TFmode  */
     }
 
@@ -2889,6 +2887,8 @@  rs6000_init_hard_regno_mode_ok (bool glo
 	  reg_addr[V4SFmode].reload_load   = CODE_FOR_reload_v4sf_di_load;
 	  reg_addr[V2DFmode].reload_store  = CODE_FOR_reload_v2df_di_store;
 	  reg_addr[V2DFmode].reload_load   = CODE_FOR_reload_v2df_di_load;
+	  reg_addr[KFmode].reload_store    = CODE_FOR_reload_kf_di_store;
+	  reg_addr[KFmode].reload_load     = CODE_FOR_reload_kf_di_load;
 	  reg_addr[DFmode].reload_store    = CODE_FOR_reload_df_di_store;
 	  reg_addr[DFmode].reload_load     = CODE_FOR_reload_df_di_load;
 	  reg_addr[DDmode].reload_store    = CODE_FOR_reload_dd_di_store;
@@ -2896,6 +2896,12 @@  rs6000_init_hard_regno_mode_ok (bool glo
 	  reg_addr[SFmode].reload_store    = CODE_FOR_reload_sf_di_store;
 	  reg_addr[SFmode].reload_load     = CODE_FOR_reload_sf_di_load;
 
+	  if (FLOAT128_IEEE_P (TFmode))
+	    {
+	      reg_addr[TFmode].reload_store = CODE_FOR_reload_tf_di_store;
+	      reg_addr[TFmode].reload_load  = CODE_FOR_reload_tf_di_load;
+	    }
+
 	  /* Only provide a reload handler for SDmode if lfiwzx/stfiwx are
 	     available.  */
 	  if (TARGET_NO_SDMODE_STACK)
@@ -2949,6 +2955,8 @@  rs6000_init_hard_regno_mode_ok (bool glo
 	  reg_addr[V4SFmode].reload_load   = CODE_FOR_reload_v4sf_si_load;
 	  reg_addr[V2DFmode].reload_store  = CODE_FOR_reload_v2df_si_store;
 	  reg_addr[V2DFmode].reload_load   = CODE_FOR_reload_v2df_si_load;
+	  reg_addr[KFmode].reload_store    = CODE_FOR_reload_kf_si_store;
+	  reg_addr[KFmode].reload_load     = CODE_FOR_reload_kf_si_load;
 	  reg_addr[DFmode].reload_store    = CODE_FOR_reload_df_si_store;
 	  reg_addr[DFmode].reload_load     = CODE_FOR_reload_df_si_load;
 	  reg_addr[DDmode].reload_store    = CODE_FOR_reload_dd_si_store;
@@ -2956,6 +2964,12 @@  rs6000_init_hard_regno_mode_ok (bool glo
 	  reg_addr[SFmode].reload_store    = CODE_FOR_reload_sf_si_store;
 	  reg_addr[SFmode].reload_load     = CODE_FOR_reload_sf_si_load;
 
+	  if (FLOAT128_IEEE_P (TFmode))
+	    {
+	      reg_addr[TFmode].reload_store = CODE_FOR_reload_tf_si_store;
+	      reg_addr[TFmode].reload_load  = CODE_FOR_reload_tf_si_load;
+	    }
+
 	  /* Only provide a reload handler for SDmode if lfiwzx/stfiwx are
 	     available.  */
 	  if (TARGET_NO_SDMODE_STACK)
@@ -3711,13 +3725,6 @@  rs6000_option_override_internal (bool gl
       && optimize >= 3)
     rs6000_isa_flags |= OPTION_MASK_P8_FUSION_SIGN;
 
-  /* Set the appropriate IEEE 128-bit floating option.  Do not enable float128
-     support by default until the libgcc support is added.  */
-  if (TARGET_FLOAT128 == FLOAT128_UNSET)
-    TARGET_FLOAT128 = FLOAT128_NONE;
-  else if (TARGET_FLOAT128 == FLOAT128_SW && !TARGET_VSX)
-    error ("-mfloat128-software requires VSX support");
-
   /* Set -mallow-movmisalign to explicitly on if we have full ISA 2.07
      support. If we only have ISA 2.06 support, and the user did not specify
      the switch, leave it set to -1 so the movmisalign patterns are enabled,
@@ -3757,6 +3764,15 @@  rs6000_option_override_internal (bool gl
 	}
     }
 
+  /* __float128 requires VSX support.  */
+  if (TARGET_FLOAT128 && !TARGET_VSX)
+    {
+      if ((rs6000_isa_flags_explicit & OPTION_MASK_FLOAT128) != 0)
+	error ("-mfloat128 requires VSX support");
+
+      rs6000_isa_flags &= ~OPTION_MASK_FLOAT128;
+    }
+
   if (TARGET_DEBUG_REG || TARGET_DEBUG_TARGET)
     rs6000_print_isa_options (stderr, 0, "after defaults", rs6000_isa_flags);
 
@@ -3837,7 +3853,8 @@  rs6000_option_override_internal (bool gl
      unless the altivec ABI was set.  This is set by default for 64-bit, but
      not for 32-bit.  */
   if (main_target_opt != NULL && !main_target_opt->x_rs6000_altivec_abi)
-    rs6000_isa_flags &= ~((OPTION_MASK_VSX | OPTION_MASK_ALTIVEC)
+    rs6000_isa_flags &= ~((OPTION_MASK_VSX | OPTION_MASK_ALTIVEC
+			   | OPTION_MASK_FLOAT128)
 			  & ~rs6000_isa_flags_explicit);
 
   /* Enable Altivec ABI for AIX -maltivec.  */
@@ -8467,8 +8484,14 @@  rs6000_const_vec (machine_mode mode)
 rtx
 rs6000_gen_le_vsx_permute (rtx source, machine_mode mode)
 {
-  rtx par = gen_rtx_PARALLEL (VOIDmode, rs6000_const_vec (mode));
-  return gen_rtx_VEC_SELECT (mode, source, par);
+  /* Use ROTATE instead of VEC_SELECT on IEEE 128-bit floating point.  */
+  if (FLOAT128_VECTOR_P (mode))
+    return gen_rtx_ROTATE (mode, source, GEN_INT (64));
+  else
+    {
+      rtx par = gen_rtx_PARALLEL (VOIDmode, rs6000_const_vec (mode));
+      return gen_rtx_VEC_SELECT (mode, source, par);
+    }
 }
 
 /* Emit a little-endian load from vector memory location SOURCE to VSX
@@ -9447,6 +9470,7 @@  init_cumulative_args (CUMULATIVE_ARGS *c
 		      ? CALL_LIBCALL : CALL_NORMAL);
   cum->sysv_gregno = GP_ARG_MIN_REG;
   cum->stdarg = stdarg_p (fntype);
+  cum->libcall = libcall;
 
   cum->nargs_prototype = 0;
   if (incoming || cum->prototype)
@@ -10609,9 +10633,11 @@  rs6000_function_arg (cumulative_args_t c
       rtx r, off;
       int i, k = 0;
 
-      /* Do we also need to pass this argument in the parameter
-	 save area?  */
-      if (TARGET_64BIT && ! cum->prototype)
+      /* Do we also need to pass this argument in the parameter save area?
+	 Library support functions for IEEE 128-bit are assumed to not need the
+	 value passed both in GPRs and in vector registers.  */
+      if (TARGET_64BIT && !cum->prototype
+	  && (!cum->libcall || !FLOAT128_VECTOR_P (elt_mode)))
 	{
 	  int align_words = (cum->words + 1) & ~1;
 	  k = rs6000_psave_function_arg (mode, type, align_words, rvec);
@@ -10842,11 +10868,14 @@  rs6000_arg_partial_bytes (cumulative_arg
 
   if (USE_ALTIVEC_FOR_ARG_P (cum, elt_mode, named))
     {
-      /* If we are passing this arg in the fixed parameter save area
-         (gprs or memory) as well as VRs, we do not use the partial
-	 bytes mechanism; instead, rs6000_function_arg will return a
-	 PARALLEL including a memory element as necessary.  */
-      if (TARGET_64BIT && ! cum->prototype)
+      /* If we are passing this arg in the fixed parameter save area (gprs or
+         memory) as well as VRs, we do not use the partial bytes mechanism;
+         instead, rs6000_function_arg will return a PARALLEL including a memory
+         element as necessary.  Library support functions for IEEE 128-bit are
+         assumed to not need the value passed both in GPRs and in vector
+         registers.  */
+      if (TARGET_64BIT && !cum->prototype
+	  && (!cum->libcall || !FLOAT128_VECTOR_P (elt_mode)))
 	return 0;
 
       /* Otherwise, we pass in VRs only.  Check for partial copies.  */
@@ -14431,8 +14460,6 @@  rs6000_init_builtins (void)
   tree tdecl;
   tree ftype;
   machine_mode mode;
-  machine_mode ieee128_mode;
-  machine_mode ibm128_mode;
 
   if (TARGET_DEBUG_BUILTIN)
     fprintf (stderr, "rs6000_init_builtins%s%s%s%s\n",
@@ -14505,26 +14532,24 @@  rs6000_init_builtins (void)
      TFmode will be either IEEE 128-bit floating point or the IBM double-double
      format that uses a pair of doubles, depending on the switches and
      defaults.  */
-  if (TARGET_IEEEQUAD)
-    {
-      ieee128_mode = TFmode;
-      ibm128_mode = IFmode;
-    }
-  else
+  if (TARGET_FLOAT128)
     {
-      ieee128_mode = KFmode;
-      ibm128_mode = TFmode;
-    }
+      ibm128_float_type_node = make_node (REAL_TYPE);
+      TYPE_PRECISION (ibm128_float_type_node) = 128;
+      layout_type (ibm128_float_type_node);
+      SET_TYPE_MODE (ibm128_float_type_node, IFmode);
+
+      ieee128_float_type_node = make_node (REAL_TYPE);
+      TYPE_PRECISION (ieee128_float_type_node) = 128;
+      layout_type (ieee128_float_type_node);
+      SET_TYPE_MODE (ieee128_float_type_node, KFmode);
 
-  ieee128_float_type_node = make_node (REAL_TYPE);
-  TYPE_PRECISION (ieee128_float_type_node) = 128;
-  layout_type (ieee128_float_type_node);
-  SET_TYPE_MODE (ieee128_float_type_node, ieee128_mode);
+      lang_hooks.types.register_builtin_type (ieee128_float_type_node,
+					      "__float128");
 
-  ibm128_float_type_node = make_node (REAL_TYPE);
-  TYPE_PRECISION (ibm128_float_type_node) = 128;
-  layout_type (ibm128_float_type_node);
-  SET_TYPE_MODE (ibm128_float_type_node, ibm128_mode);
+      lang_hooks.types.register_builtin_type (ibm128_float_type_node,
+					      "__ibm128");
+    }
 
   /* Initialize the modes for builtin_function_type, mapping a machine mode to
      tree type node.  */
@@ -16030,75 +16055,184 @@  rs6000_common_init_builtins (void)
     }
 }
 
+/* Set up AIX/Darwin/64-bit Linux quad floating point routines.  */
 static void
-rs6000_init_libfuncs (void)
+init_float128_ibm (machine_mode mode)
 {
-  if (!TARGET_IEEEQUAD)
-      /* AIX/Darwin/64-bit Linux quad floating point routines.  */
-    if (!TARGET_XL_COMPAT)
-      {
-	set_optab_libfunc (add_optab, TFmode, "__gcc_qadd");
-	set_optab_libfunc (sub_optab, TFmode, "__gcc_qsub");
-	set_optab_libfunc (smul_optab, TFmode, "__gcc_qmul");
-	set_optab_libfunc (sdiv_optab, TFmode, "__gcc_qdiv");
+  if (!TARGET_XL_COMPAT)
+    {
+      set_optab_libfunc (add_optab, mode, "__gcc_qadd");
+      set_optab_libfunc (sub_optab, mode, "__gcc_qsub");
+      set_optab_libfunc (smul_optab, mode, "__gcc_qmul");
+      set_optab_libfunc (sdiv_optab, mode, "__gcc_qdiv");
 
-	if (!(TARGET_HARD_FLOAT && (TARGET_FPRS || TARGET_E500_DOUBLE)))
-	  {
-	    set_optab_libfunc (neg_optab, TFmode, "__gcc_qneg");
-	    set_optab_libfunc (eq_optab, TFmode, "__gcc_qeq");
-	    set_optab_libfunc (ne_optab, TFmode, "__gcc_qne");
-	    set_optab_libfunc (gt_optab, TFmode, "__gcc_qgt");
-	    set_optab_libfunc (ge_optab, TFmode, "__gcc_qge");
-	    set_optab_libfunc (lt_optab, TFmode, "__gcc_qlt");
-	    set_optab_libfunc (le_optab, TFmode, "__gcc_qle");
-
-	    set_conv_libfunc (sext_optab, TFmode, SFmode, "__gcc_stoq");
-	    set_conv_libfunc (sext_optab, TFmode, DFmode, "__gcc_dtoq");
-	    set_conv_libfunc (trunc_optab, SFmode, TFmode, "__gcc_qtos");
-	    set_conv_libfunc (trunc_optab, DFmode, TFmode, "__gcc_qtod");
-	    set_conv_libfunc (sfix_optab, SImode, TFmode, "__gcc_qtoi");
-	    set_conv_libfunc (ufix_optab, SImode, TFmode, "__gcc_qtou");
-	    set_conv_libfunc (sfloat_optab, TFmode, SImode, "__gcc_itoq");
-	    set_conv_libfunc (ufloat_optab, TFmode, SImode, "__gcc_utoq");
-	  }
+      if (!(TARGET_HARD_FLOAT && (TARGET_FPRS || TARGET_E500_DOUBLE)))
+	{
+	  set_optab_libfunc (neg_optab, mode, "__gcc_qneg");
+	  set_optab_libfunc (eq_optab, mode, "__gcc_qeq");
+	  set_optab_libfunc (ne_optab, mode, "__gcc_qne");
+	  set_optab_libfunc (gt_optab, mode, "__gcc_qgt");
+	  set_optab_libfunc (ge_optab, mode, "__gcc_qge");
+	  set_optab_libfunc (lt_optab, mode, "__gcc_qlt");
+	  set_optab_libfunc (le_optab, mode, "__gcc_qle");
 
-	if (!(TARGET_HARD_FLOAT && TARGET_FPRS))
-	  set_optab_libfunc (unord_optab, TFmode, "__gcc_qunord");
-      }
-    else
-      {
-	set_optab_libfunc (add_optab, TFmode, "_xlqadd");
-	set_optab_libfunc (sub_optab, TFmode, "_xlqsub");
-	set_optab_libfunc (smul_optab, TFmode, "_xlqmul");
-	set_optab_libfunc (sdiv_optab, TFmode, "_xlqdiv");
-      }
+	  set_conv_libfunc (sext_optab, mode, SFmode, "__gcc_stoq");
+	  set_conv_libfunc (sext_optab, mode, DFmode, "__gcc_dtoq");
+	  set_conv_libfunc (trunc_optab, SFmode, mode, "__gcc_qtos");
+	  set_conv_libfunc (trunc_optab, DFmode, mode, "__gcc_qtod");
+	  set_conv_libfunc (sfix_optab, SImode, mode, "__gcc_qtoi");
+	  set_conv_libfunc (ufix_optab, SImode, mode, "__gcc_qtou");
+	  set_conv_libfunc (sfloat_optab, mode, SImode, "__gcc_itoq");
+	  set_conv_libfunc (ufloat_optab, mode, SImode, "__gcc_utoq");
+	}
+
+      if (!(TARGET_HARD_FLOAT && TARGET_FPRS))
+	set_optab_libfunc (unord_optab, mode, "__gcc_qunord");
+    }
   else
     {
-      /* 32-bit SVR4 quad floating point routines.  */
+      set_optab_libfunc (add_optab, mode, "_xlqadd");
+      set_optab_libfunc (sub_optab, mode, "_xlqsub");
+      set_optab_libfunc (smul_optab, mode, "_xlqmul");
+      set_optab_libfunc (sdiv_optab, mode, "_xlqdiv");
+    }
+
+  /* Add various conversions for IFmode to use the traditional TFmode
+     names.  */
+  if (mode == IFmode)
+    {
+      set_conv_libfunc (sext_optab, mode, SDmode, "__dpd_extendsdtf2");
+      set_conv_libfunc (sext_optab, mode, DDmode, "__dpd_extendddtf2");
+      set_conv_libfunc (trunc_optab, mode, TDmode, "__dpd_trunctftd2");
+      set_conv_libfunc (trunc_optab, SDmode, mode, "__dpd_trunctfsd2");
+      set_conv_libfunc (trunc_optab, DDmode, mode, "__dpd_trunctfdd2");
+      set_conv_libfunc (sext_optab, TDmode, mode, "__dpd_extendtdtf2");
 
-      set_optab_libfunc (add_optab, TFmode, "_q_add");
-      set_optab_libfunc (sub_optab, TFmode, "_q_sub");
-      set_optab_libfunc (neg_optab, TFmode, "_q_neg");
-      set_optab_libfunc (smul_optab, TFmode, "_q_mul");
-      set_optab_libfunc (sdiv_optab, TFmode, "_q_div");
+      if (TARGET_POWERPC64)
+	{
+	  set_conv_libfunc (sfix_optab, TImode, mode, "__fixtfti");
+	  set_conv_libfunc (ufix_optab, TImode, mode, "__fixunstfti");
+	  set_conv_libfunc (sfloat_optab, mode, TImode, "__floattitf");
+	  set_conv_libfunc (ufloat_optab, mode, TImode, "__floatuntitf");
+	}
+    }
+}
+
+/* Set up IEEE 128-bit floating point routines.  Use different names if the
+   arguments can be passed in a vector register.  The historical PowerPC
+   implementation of IEEE 128-bit floating point used _q_<op> for the names, so
+   continue to use that if we aren't using vector registers to pass IEEE
+   128-bit floating point.  */
+
+static void
+init_float128_ieee (machine_mode mode)
+{
+  if (FLOAT128_VECTOR_P (mode))
+    {
+      set_optab_libfunc (add_optab, mode, "__addkf3");
+      set_optab_libfunc (sub_optab, mode, "__subkf3");
+      set_optab_libfunc (neg_optab, mode, "__negkf2");
+      set_optab_libfunc (smul_optab, mode, "__mulkf3");
+      set_optab_libfunc (sdiv_optab, mode, "__divkf3");
+      set_optab_libfunc (sqrt_optab, mode, "__sqrtkf2");
+      set_optab_libfunc (abs_optab, mode, "__abstkf2");
+
+      set_optab_libfunc (eq_optab, mode, "__eqkf2");
+      set_optab_libfunc (ne_optab, mode, "__nekf2");
+      set_optab_libfunc (gt_optab, mode, "__gtkf2");
+      set_optab_libfunc (ge_optab, mode, "__gekf2");
+      set_optab_libfunc (lt_optab, mode, "__ltkf2");
+      set_optab_libfunc (le_optab, mode, "__lekf2");
+      set_optab_libfunc (unord_optab, mode, "__unordkf2");
+      set_optab_libfunc (cmp_optab, mode, "__cmpokf2");		/* fcmpo */
+      set_optab_libfunc (ucmp_optab, mode, "__cmpukf2");	/* fcmpu */
+
+      set_conv_libfunc (sext_optab, mode, SFmode, "__extendsfkf2");
+      set_conv_libfunc (sext_optab, mode, DFmode, "__extenddfkf2");
+      set_conv_libfunc (trunc_optab, SFmode, mode, "__trunckfsf2");
+      set_conv_libfunc (trunc_optab, DFmode, mode, "__trunckfdf2");
+
+      set_conv_libfunc (sext_optab, mode, IFmode, "__extendtfkf2");
+      if (mode != TFmode && FLOAT128_IBM_P (TFmode))
+	set_conv_libfunc (sext_optab, mode, TFmode, "__extendtfkf2");
+
+      set_conv_libfunc (trunc_optab, IFmode, mode, "__trunckftf2");
+      if (mode != TFmode && FLOAT128_IBM_P (TFmode))
+	set_conv_libfunc (trunc_optab, TFmode, mode, "__trunckftf2");
+
+      set_conv_libfunc (sext_optab, mode, SDmode, "__dpd_extendsdkf2");
+      set_conv_libfunc (sext_optab, mode, DDmode, "__dpd_extendddkf2");
+      set_conv_libfunc (trunc_optab, mode, TDmode, "__dpd_trunckftd2");
+      set_conv_libfunc (trunc_optab, SDmode, mode, "__dpd_trunckfsd2");
+      set_conv_libfunc (trunc_optab, DDmode, mode, "__dpd_trunckfdd2");
+      set_conv_libfunc (sext_optab, TDmode, mode, "__dpd_extendtdkf2");
+
+      set_conv_libfunc (sfix_optab, SImode, mode, "__fixkfsi");
+      set_conv_libfunc (ufix_optab, SImode, mode, "__fixunskfsi");
+      set_conv_libfunc (sfix_optab, DImode, mode, "__fixkfdi");
+      set_conv_libfunc (ufix_optab, DImode, mode, "__fixunskfdi");
+
+      set_conv_libfunc (sfloat_optab, mode, SImode, "__floatsikf");
+      set_conv_libfunc (ufloat_optab, mode, SImode, "__floatunsikf");
+      set_conv_libfunc (sfloat_optab, mode, DImode, "__floatdikf");
+      set_conv_libfunc (ufloat_optab, mode, DImode, "__floatundikf");
+
+      if (TARGET_POWERPC64)
+	{
+	  set_conv_libfunc (sfix_optab, TImode, mode, "__fixkfti");
+	  set_conv_libfunc (ufix_optab, TImode, mode, "__fixunskfti");
+	  set_conv_libfunc (sfloat_optab, mode, TImode, "__floattikf");
+	  set_conv_libfunc (ufloat_optab, mode, TImode, "__floatuntikf");
+	}
+    }
+
+  else
+    {
+      set_optab_libfunc (add_optab, mode, "_q_add");
+      set_optab_libfunc (sub_optab, mode, "_q_sub");
+      set_optab_libfunc (neg_optab, mode, "_q_neg");
+      set_optab_libfunc (smul_optab, mode, "_q_mul");
+      set_optab_libfunc (sdiv_optab, mode, "_q_div");
       if (TARGET_PPC_GPOPT)
-	set_optab_libfunc (sqrt_optab, TFmode, "_q_sqrt");
+	set_optab_libfunc (sqrt_optab, mode, "_q_sqrt");
 
-      set_optab_libfunc (eq_optab, TFmode, "_q_feq");
-      set_optab_libfunc (ne_optab, TFmode, "_q_fne");
-      set_optab_libfunc (gt_optab, TFmode, "_q_fgt");
-      set_optab_libfunc (ge_optab, TFmode, "_q_fge");
-      set_optab_libfunc (lt_optab, TFmode, "_q_flt");
-      set_optab_libfunc (le_optab, TFmode, "_q_fle");
-
-      set_conv_libfunc (sext_optab, TFmode, SFmode, "_q_stoq");
-      set_conv_libfunc (sext_optab, TFmode, DFmode, "_q_dtoq");
-      set_conv_libfunc (trunc_optab, SFmode, TFmode, "_q_qtos");
-      set_conv_libfunc (trunc_optab, DFmode, TFmode, "_q_qtod");
-      set_conv_libfunc (sfix_optab, SImode, TFmode, "_q_qtoi");
-      set_conv_libfunc (ufix_optab, SImode, TFmode, "_q_qtou");
-      set_conv_libfunc (sfloat_optab, TFmode, SImode, "_q_itoq");
-      set_conv_libfunc (ufloat_optab, TFmode, SImode, "_q_utoq");
+      set_optab_libfunc (eq_optab, mode, "_q_feq");
+      set_optab_libfunc (ne_optab, mode, "_q_fne");
+      set_optab_libfunc (gt_optab, mode, "_q_fgt");
+      set_optab_libfunc (ge_optab, mode, "_q_fge");
+      set_optab_libfunc (lt_optab, mode, "_q_flt");
+      set_optab_libfunc (le_optab, mode, "_q_fle");
+
+      set_conv_libfunc (sext_optab, mode, SFmode, "_q_stoq");
+      set_conv_libfunc (sext_optab, mode, DFmode, "_q_dtoq");
+      set_conv_libfunc (trunc_optab, SFmode, mode, "_q_qtos");
+      set_conv_libfunc (trunc_optab, DFmode, mode, "_q_qtod");
+      set_conv_libfunc (sfix_optab, SImode, mode, "_q_qtoi");
+      set_conv_libfunc (ufix_optab, SImode, mode, "_q_qtou");
+      set_conv_libfunc (sfloat_optab, mode, SImode, "_q_itoq");
+      set_conv_libfunc (ufloat_optab, mode, SImode, "_q_utoq");
+    }
+}
+
+static void
+rs6000_init_libfuncs (void)
+{
+  /* __float128 support.  */
+  if (TARGET_FLOAT128)
+    {
+      init_float128_ibm (IFmode);
+      init_float128_ieee (KFmode);
+    }
+
+  /* AIX/Darwin/64-bit Linux quad floating point routines.  */
+  if (TARGET_LONG_DOUBLE_128)
+    {
+      if (!TARGET_IEEEQUAD)
+	init_float128_ibm (TFmode);
+
+      /* IEEE 128-bit including 32-bit SVR4 quad floating point routines.  */
+      else
+	init_float128_ieee (TFmode);
     }
 }
 
@@ -19983,17 +20117,18 @@  rs6000_generate_compare (rtx cmp, machin
       emit_insn (cmp);
     }
 
-  /* IEEE 128-bit support in VSX registers.  The comparison function (__cmpkf2)
-     returns 0..15 that is laid out the same way as the PowerPC CR register
-     would for a normal floating point comparison.  */
+  /* IEEE 128-bit support in VSX registers.  The comparison functions
+     (__cmpokf2 and __cmpukf2) returns 0..15 that is laid out the same way as
+     the PowerPC CR register would for a normal floating point comparison from
+     the fcmpo and fcmpu instructions.  */
   else if (FLOAT128_IEEE_P (mode))
     {
       rtx and_reg = gen_reg_rtx (SImode);
       rtx dest = gen_reg_rtx (SImode);
-      rtx libfunc = optab_libfunc (cmp_optab, mode);
+      rtx libfunc = optab_libfunc (ucmp_optab, mode);
       HOST_WIDE_INT mask_value = 0;
 
-      /* Values that __cmpkf2 returns.  */
+      /* Values that __cmpokf2/__cmpukf2 returns.  */
 #define PPC_CMP_UNORDERED	0x1		/* isnan (a) || isnan (b).  */
 #define PPC_CMP_EQUAL		0x2		/* a == b.  */
 #define PPC_CMP_GREATER_THEN	0x4		/* a > b.  */
@@ -20164,7 +20299,7 @@  rs6000_generate_compare (rtx cmp, machin
   return gen_rtx_fmt_ee (code, VOIDmode, compare_result, const0_rtx);
 }
 
-
+
 /* Expand floating point conversion to/from __float128 and __ibm128.  */
 
 void
@@ -20173,60 +20308,121 @@  rs6000_expand_float128_convert (rtx dest
   machine_mode dest_mode = GET_MODE (dest);
   machine_mode src_mode = GET_MODE (src);
   convert_optab cvt = unknown_optab;
+  bool do_move = false;
   rtx libfunc = NULL_RTX;
   rtx dest2;
 
   if (dest_mode == src_mode)
     gcc_unreachable ();
 
+  /* Eliminate memory operations.  */
+  if (MEM_P (src))
+    src = force_reg (src_mode, src);
+
+  if (MEM_P (dest))
+    {
+      rtx tmp = gen_reg_rtx (dest_mode);
+      rs6000_expand_float128_convert (tmp, src, unsigned_p);
+      rs6000_emit_move (dest, tmp, dest_mode);
+      return;
+    }
+
+  /* Convert to IEEE 128-bit floating point.  */
   if (FLOAT128_IEEE_P (dest_mode))
     {
-      if (src_mode == SFmode
-	  || src_mode == DFmode
-	  || FLOAT128_IBM_P (src_mode))
-	cvt = sext_optab;
+      switch (src_mode)
+	{
+	case DFmode:
+	  cvt = sext_optab;
+	  break;
 
-      else if (GET_MODE_CLASS (src_mode) == MODE_INT)
-	cvt = (unsigned_p) ? ufloat_optab : sfloat_optab;
+	case SFmode:
+	  cvt = sext_optab;
+	  break;
 
-      else if (FLOAT128_IEEE_P (src_mode))
-	emit_move_insn (dest, gen_lowpart (dest_mode, src));
+	case KFmode:
+	case IFmode:
+	case TFmode:
+	  if (FLOAT128_IBM_P (src_mode))
+	    cvt = sext_optab;
+	  else
+	    do_move = true;
+	  break;
 
-      else
-	gcc_unreachable ();
+	case SImode:
+	case DImode:
+	  cvt = (unsigned_p) ? ufloat_optab : sfloat_optab;
+	  break;
+
+	default:
+	  gcc_unreachable ();
+	}
     }
 
+  /* Convert from IEEE 128-bit floating point.  */
   else if (FLOAT128_IEEE_P (src_mode))
     {
-      if (dest_mode == SFmode
-	  || dest_mode == DFmode
-	  || FLOAT128_IBM_P (dest_mode))
-	cvt = trunc_optab;
+      switch (dest_mode)
+	{
+	case DFmode:
+	  cvt = trunc_optab;
+	  break;
 
-      else if (GET_MODE_CLASS (dest_mode) == MODE_INT)
-	cvt = (unsigned_p) ? ufix_optab : sfix_optab;
+	case SFmode:
+	  cvt = trunc_optab;
+	  break;
 
-      else
-	gcc_unreachable ();
+	case KFmode:
+	case IFmode:
+	case TFmode:
+	  if (FLOAT128_IBM_P (dest_mode))
+	    cvt = trunc_optab;
+	  else
+	    do_move = true;
+	  break;
+
+	case SImode:
+	case DImode:
+	  cvt = (unsigned_p) ? ufix_optab : sfix_optab;
+	  break;
+
+	default:
+	  gcc_unreachable ();
+	}
     }
 
+  /* Both IBM format.  */
+  else if (FLOAT128_IBM_P (dest_mode) && FLOAT128_IBM_P (src_mode))
+    do_move = true;
+
   else
     gcc_unreachable ();
 
-  gcc_assert (cvt != unknown_optab);
-  libfunc = convert_optab_libfunc (cvt, dest_mode, src_mode);
-  gcc_assert (libfunc != NULL_RTX);
-
-  dest2 = emit_library_call_value (libfunc, dest, LCT_CONST, dest_mode, 1, src,
-				   src_mode);
-
-  gcc_assert (dest != NULL_RTX);
-  if (!rtx_equal_p (dest, dest2))
-    emit_move_insn (dest, dest2);
+  /* Handle conversion between TFmode/KFmode.  */
+  if (do_move)
+    emit_move_insn (dest, gen_lowpart (dest_mode, src));
+
+  /* Call an external function to do the conversion.  */
+  else if (cvt != unknown_optab)
+    {
+      libfunc = convert_optab_libfunc (cvt, dest_mode, src_mode);
+      gcc_assert (libfunc != NULL_RTX);
+
+      dest2 = emit_library_call_value (libfunc, dest, LCT_CONST, dest_mode, 1, src,
+				       src_mode);
+
+      gcc_assert (dest2 != NULL_RTX);
+      if (!rtx_equal_p (dest, dest2))
+	emit_move_insn (dest, dest2);
+    }
+
+  else
+    gcc_unreachable ();
 
   return;
 }
 
+
 /* Emit the RTL for an sISEL pattern.  */
 
 void
@@ -29831,6 +30027,21 @@  rs6000_mangle_type (const_tree type)
   if (type == bool_int_type_node) return "U6__booli";
   if (type == bool_long_type_node) return "U6__booll";
 
+  /* Use a unique name for __float128 rather than trying to use "e" or "g". Use
+     "g" for IBM extended double, no matter whether it is long double (using
+     -mabi=ibmlongdouble) or the distinct __ibm128 type.  */
+  if (TARGET_FLOAT128)
+    {
+      if (type == ieee128_float_type_node)
+	return "U10__float128";
+
+      if (type == ibm128_float_type_node)
+	return "g";
+
+      if (type == long_double_type_node && TARGET_LONG_DOUBLE_128)
+	return (TARGET_IEEEQUAD) ? "U10__float128" : "g";
+    }
+
   /* Mangle IBM extended float long double as `g' (__float128) on
      powerpc*-linux where long-double-64 previously was the default.  */
   if (TYPE_MAIN_VARIANT (type) == long_double_type_node
@@ -33116,8 +33327,8 @@  rs6000_scalar_mode_supported_p (machine_
 
   if (DECIMAL_FLOAT_MODE_P (mode))
     return default_decimal_float_supported_p ();
-  else if (mode == KFmode)
-    return TARGET_FLOAT128;
+  else if (TARGET_FLOAT128 && (mode == KFmode || mode == IFmode))
+    return true;
   else
     return default_scalar_mode_supported_p (mode);
 }
@@ -33143,6 +33354,26 @@  rs6000_vector_mode_supported_p (machine_
     return false;
 }
 
+/* Target hook for c_mode_for_suffix.  */
+static machine_mode
+rs6000_c_mode_for_suffix (char suffix)
+{
+  if (TARGET_FLOAT128)
+    {
+      if (suffix == 'q' || suffix == 'Q')
+	return (FLOAT128_IEEE_P (TFmode)) ? TFmode : KFmode;
+
+      /* At the moment, we are not defining a suffix for IBM extended double.
+	 If/when the default for -mabi=ieeelongdouble is changed, and we want
+	 to support __ibm128 constants in legacy library code, we may need to
+	 re-evalaute this decision.  Currently, c-lex.c only supports 'w' and
+	 'q' as machine dependent suffixes.  The x86_64 port uses 'w' for
+	 __float80 constants.  */
+    }
+
+  return VOIDmode;
+}
+
 /* Target hook for invalid_arg_for_unprototyped_fn. */
 static const char *
 invalid_arg_for_unprototyped_fn (const_tree typelist, const_tree funcdecl, const_tree val)
@@ -33228,6 +33459,7 @@  static struct rs6000_opt_mask const rs60
   { "dlmzb",			OPTION_MASK_DLMZB,		false, true  },
   { "efficient-unaligned-vsx",	OPTION_MASK_EFFICIENT_UNALIGNED_VSX,
 								false, true  },
+  { "float128",			OPTION_MASK_FLOAT128,		false, true  },
   { "fprnd",			OPTION_MASK_FPRND,		false, true  },
   { "hard-dfp",			OPTION_MASK_DFP,		false, true  },
   { "htm",			OPTION_MASK_HTM,		false, true  },
@@ -33306,7 +33538,7 @@  static struct rs6000_opt_mask const rs60
 struct rs6000_opt_var {
   const char *name;		/* option name */
   size_t global_offset;		/* offset of the option in global_options.  */
-  size_t target_offset;		/* offset of the option in target optiosn.  */
+  size_t target_offset;		/* offset of the option in target options.  */
 };
 
 static struct rs6000_opt_var const rs6000_opt_vars[] =
@@ -35611,7 +35843,7 @@  chain_contains_only_swaps (swap_web_entr
 
   for (; link; link = link->next)
     {
-      if (!VECTOR_MODE_P (GET_MODE (DF_REF_REG (link->ref))))
+      if (!ALTIVEC_OR_VSX_VECTOR_MODE (GET_MODE (DF_REF_REG (link->ref))))
 	continue;
 
       if (DF_REF_IS_ARTIFICIAL (link->ref))
@@ -35710,7 +35942,7 @@  mark_swaps_for_removal (swap_web_entry *
 	{
 	  /* Ignore uses for addressability.  */
 	  machine_mode mode = GET_MODE (DF_REF_REG (use));
-	  if (!VECTOR_MODE_P (mode))
+	  if (!ALTIVEC_OR_VSX_VECTOR_MODE (mode))
 	    continue;
 
 	  struct df_link *link = DF_REF_CHAIN (use);
@@ -36224,10 +36456,11 @@  rs6000_analyze_swaps (function *fun)
 		    mode = V4SImode;
 		}
 
-	      if (VECTOR_MODE_P (mode) || mode == TImode)
+	      if (ALTIVEC_OR_VSX_VECTOR_MODE (mode) || mode == TImode)
 		{
 		  insn_entry[uid].is_relevant = 1;
-		  if (mode == TImode || mode == V1TImode)
+		  if (mode == TImode || mode == V1TImode
+		      || FLOAT128_VECTOR_P (mode))
 		    insn_entry[uid].is_128_int = 1;
 		  if (DF_REF_INSN_INFO (mention))
 		    insn_entry[uid].contains_subreg
@@ -36248,13 +36481,14 @@  rs6000_analyze_swaps (function *fun)
 		 isn't sufficient to ensure we union the call into the
 		 web with the parameter setup code.  */
 	      if (mode == DImode && GET_CODE (insn) == SET
-		  && VECTOR_MODE_P (GET_MODE (SET_DEST (insn))))
+		  && ALTIVEC_OR_VSX_VECTOR_MODE (GET_MODE (SET_DEST (insn))))
 		mode = GET_MODE (SET_DEST (insn));
 
-	      if (VECTOR_MODE_P (mode) || mode == TImode)
+	      if (ALTIVEC_OR_VSX_VECTOR_MODE (mode) || mode == TImode)
 		{
 		  insn_entry[uid].is_relevant = 1;
-		  if (mode == TImode || mode == V1TImode)
+		  if (mode == TImode || mode == V1TImode
+		      || FLOAT128_VECTOR_P (mode))
 		    insn_entry[uid].is_128_int = 1;
 		  if (DF_REF_INSN_INFO (mention))
 		    insn_entry[uid].contains_subreg
Index: gcc/config/rs6000/rs6000.h
===================================================================
--- gcc/config/rs6000/rs6000.h	(revision 228506)
+++ gcc/config/rs6000/rs6000.h	(working copy)
@@ -1217,11 +1217,16 @@  enum data_align { align_abi, align_opt, 
 	 ((MODE) == V4SFmode		\
 	  || (MODE) == V2DFmode)	\
 
-#define ALTIVEC_VECTOR_MODE(MODE)	\
-	 ((MODE) == V16QImode		\
-	  || (MODE) == V8HImode		\
-	  || (MODE) == V4SFmode		\
-	  || (MODE) == V4SImode)
+/* Note KFmode and possibly TFmode (i.e. IEEE 128-bit floating point) are not
+   really a vector, but we want to treat it as a vector for moves, and
+   such.  */
+
+#define ALTIVEC_VECTOR_MODE(MODE)					\
+  ((MODE) == V16QImode							\
+   || (MODE) == V8HImode						\
+   || (MODE) == V4SFmode						\
+   || (MODE) == V4SImode						\
+   || FLOAT128_VECTOR_P (MODE))
 
 #define ALTIVEC_OR_VSX_VECTOR_MODE(MODE)				\
   (ALTIVEC_VECTOR_MODE (MODE) || VSX_VECTOR_MODE (MODE)			\
@@ -1248,12 +1253,19 @@  enum data_align { align_abi, align_opt, 
 
    PTImode cannot tie with other modes because PTImode is restricted to even
    GPR registers, and TImode can go in any GPR as well as VSX registers (PR
-   57744).  */
+   57744).
+
+   Altivec/VSX vector tests were moved ahead of scalar float mode, so that IEEE
+   128-bit floating point on VSX systems ties with other vectors.  */
 #define MODES_TIEABLE_P(MODE1, MODE2)		\
   ((MODE1) == PTImode				\
    ? (MODE2) == PTImode				\
    : (MODE2) == PTImode				\
    ? 0						\
+   : ALTIVEC_OR_VSX_VECTOR_MODE (MODE1)		\
+   ? ALTIVEC_OR_VSX_VECTOR_MODE (MODE2)		\
+   : ALTIVEC_OR_VSX_VECTOR_MODE (MODE2)		\
+   ? 0						\
    : SCALAR_FLOAT_MODE_P (MODE1)		\
    ? SCALAR_FLOAT_MODE_P (MODE2)		\
    : SCALAR_FLOAT_MODE_P (MODE2)		\
@@ -1266,10 +1278,6 @@  enum data_align { align_abi, align_opt, 
    ? SPE_VECTOR_MODE (MODE2)			\
    : SPE_VECTOR_MODE (MODE2)			\
    ? 0						\
-   : ALTIVEC_OR_VSX_VECTOR_MODE (MODE1)		\
-   ? ALTIVEC_OR_VSX_VECTOR_MODE (MODE2)		\
-   : ALTIVEC_OR_VSX_VECTOR_MODE (MODE2)		\
-   ? 0						\
    : 1)
 
 /* Post-reload, we can't use any new AltiVec registers, as we already
@@ -1801,6 +1809,7 @@  typedef struct rs6000_args
 				   GPR space (darwin64) */
   int named;			/* false for varargs params */
   int escapes;			/* if function visible outside tu */
+  int libcall;			/* If this is a compiler generated call.  */
 } CUMULATIVE_ARGS;
 
 /* Initialize a variable CUM of type CUMULATIVE_ARGS
Index: gcc/config/rs6000/rs6000.md
===================================================================
--- gcc/config/rs6000/rs6000.md	(revision 228511)
+++ gcc/config/rs6000/rs6000.md	(working copy)
@@ -347,8 +347,7 @@  (define_mode_iterator FP [
    && ((TARGET_FPRS && TARGET_SINGLE_FLOAT) || TARGET_E500_SINGLE)")
   (DF "TARGET_HARD_FLOAT 
    && ((TARGET_FPRS && TARGET_DOUBLE_FLOAT) || TARGET_E500_DOUBLE)")
-  (TF "!TARGET_IEEEQUAD
-   && TARGET_HARD_FLOAT
+  (TF "TARGET_HARD_FLOAT
    && (TARGET_FPRS || TARGET_E500_DOUBLE)
    && TARGET_LONG_DOUBLE_128")
   (IF "TARGET_FLOAT128")
@@ -448,24 +447,18 @@  (define_mode_iterator RECIPF [SF DF V4SF
 ; Iterator for just SF/DF
 (define_mode_iterator SFDF [SF DF])
 
-; Iterator for float128 floating conversions
-(define_mode_iterator FLOAT128_SFDFTF [
-    (SF "TARGET_FLOAT128")
-    (DF "TARGET_FLOAT128")
-    (TF "FLOAT128_IBM_P (TFmode)")
-    (IF "TARGET_FLOAT128")])
-
-; Iterator for special 128-bit floating point.  This is for non-default
-; conversions, so TFmode is not used here.
-(define_mode_iterator IFKF [IF KF])
-
 ; Iterator for 128-bit floating point that uses the IBM double-double format
-(define_mode_iterator IBM128 [IF TF])
+(define_mode_iterator IBM128 [(IF "FLOAT128_IBM_P (IFmode)")
+			      (TF "FLOAT128_IBM_P (TFmode)")])
+
+; Iterator for 128-bit floating point that uses IEEE 128-bit float
+(define_mode_iterator IEEE128 [(KF "FLOAT128_IEEE_P (KFmode)")
+			       (TF "FLOAT128_IEEE_P (TFmode)")])
 
 ; Iterator for 128-bit floating point
-(define_mode_iterator TFIFKF [(KF "TARGET_FLOAT128")
-			      (IF "TARGET_FLOAT128")
-			      (TF "TARGET_LONG_DOUBLE_128")])
+(define_mode_iterator FLOAT128 [(KF "TARGET_FLOAT128")
+				(IF "TARGET_FLOAT128")
+				(TF "TARGET_LONG_DOUBLE_128")])
 
 ; SF/DF suffix for traditional floating instructions
 (define_mode_attr Ftrad		[(SF "s") (DF "")])
@@ -4248,8 +4241,7 @@  (define_expand "signbit<mode>2"
    	(match_dup 5))
    (set (match_operand:SI 0 "gpc_reg_operand" "")
   	(match_dup 6))]
-  "FLOAT128_IBM_P (<MODE>mode)
-   && TARGET_HARD_FLOAT
+  "TARGET_HARD_FLOAT
    && (TARGET_FPRS || TARGET_E500_DOUBLE)"
 {
   operands[2] = gen_reg_rtx (DFmode);
@@ -6476,34 +6468,51 @@  (define_insn_and_split "*mov<mode>_softf
 { rs6000_split_multireg_move (operands[0], operands[1]); DONE; }
   [(set_attr "length" "20,20,16")])
 
-(define_expand "extenddftf2"
-  [(set (match_operand:TF 0 "gpc_reg_operand" "")
-	(float_extend:TF (match_operand:DF 1 "gpc_reg_operand" "")))]
+(define_expand "extenddf<mode>2"
+  [(set (match_operand:FLOAT128 0 "gpc_reg_operand" "")
+	(float_extend:FLOAT128 (match_operand:DF 1 "gpc_reg_operand" "")))]
   "TARGET_HARD_FLOAT && (TARGET_FPRS || TARGET_E500_DOUBLE)
    && TARGET_LONG_DOUBLE_128"
 {
-  if (TARGET_IEEEQUAD)
+  if (FLOAT128_IEEE_P (<MODE>mode))
     rs6000_expand_float128_convert (operands[0], operands[1], false);
   else if (TARGET_E500_DOUBLE)
-    emit_insn (gen_spe_extenddftf2 (operands[0], operands[1]));
+    {
+      gcc_assert (<MODE>mode == TFmode);
+      emit_insn (gen_spe_extenddftf2 (operands[0], operands[1]));
+    }
   else if (TARGET_VSX)
-    emit_insn (gen_extenddftf2_vsx (operands[0], operands[1]));
-  else
+    {
+      if (<MODE>mode == TFmode)
+	emit_insn (gen_extenddftf2_vsx (operands[0], operands[1]));
+      else if (<MODE>mode == IFmode)
+	emit_insn (gen_extenddfif2_vsx (operands[0], operands[1]));
+      else
+	gcc_unreachable ();
+    }
+   else
     {
       rtx zero = gen_reg_rtx (DFmode);
       rs6000_emit_move (zero, CONST0_RTX (DFmode), DFmode);
-      emit_insn (gen_extenddftf2_fprs (operands[0], operands[1], zero));
+
+      if (<MODE>mode == TFmode)
+	emit_insn (gen_extenddftf2_fprs (operands[0], operands[1], zero));
+      else if (<MODE>mode == IFmode)
+	emit_insn (gen_extenddfif2_fprs (operands[0], operands[1], zero));
+      else
+	gcc_unreachable ();
     }
   DONE;
 })
 
 ;; Allow memory operands for the source to be created by the combiner.
-(define_insn_and_split "extenddftf2_fprs"
-  [(set (match_operand:TF 0 "gpc_reg_operand" "=d,d,&d")
-	(float_extend:TF (match_operand:DF 1 "nonimmediate_operand" "d,m,d")))
+(define_insn_and_split "extenddf<mode>2_fprs"
+  [(set (match_operand:IBM128 0 "gpc_reg_operand" "=d,d,&d")
+	(float_extend:IBM128
+	 (match_operand:DF 1 "nonimmediate_operand" "d,m,d")))
    (use (match_operand:DF 2 "nonimmediate_operand" "m,m,d"))]
   "!TARGET_VSX && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_DOUBLE_FLOAT
-   && TARGET_LONG_DOUBLE_128 && !TARGET_IEEEQUAD"
+   && TARGET_LONG_DOUBLE_128 && FLOAT128_IBM_P (<MODE>mode)"
   "#"
   "&& reload_completed"
   [(set (match_dup 3) (match_dup 1))
@@ -6512,14 +6521,15 @@  (define_insn_and_split "extenddftf2_fprs
   const int lo_word = LONG_DOUBLE_LARGE_FIRST ? GET_MODE_SIZE (DFmode) : 0;
   const int hi_word = LONG_DOUBLE_LARGE_FIRST ? 0 : GET_MODE_SIZE (DFmode);
 
-  operands[3] = simplify_gen_subreg (DFmode, operands[0], TFmode, hi_word);
-  operands[4] = simplify_gen_subreg (DFmode, operands[0], TFmode, lo_word);
+  operands[3] = simplify_gen_subreg (DFmode, operands[0], <MODE>mode, hi_word);
+  operands[4] = simplify_gen_subreg (DFmode, operands[0], <MODE>mode, lo_word);
 })
 
-(define_insn_and_split "extenddftf2_vsx"
-  [(set (match_operand:TF 0 "gpc_reg_operand" "=d,d")
-	(float_extend:TF (match_operand:DF 1 "nonimmediate_operand" "ws,m")))]
-  "TARGET_LONG_DOUBLE_128 && TARGET_VSX && !TARGET_IEEEQUAD"
+(define_insn_and_split "extenddf<mode>2_vsx"
+  [(set (match_operand:IBM128 0 "gpc_reg_operand" "=d,d")
+	(float_extend:IBM128
+	 (match_operand:DF 1 "nonimmediate_operand" "ws,m")))]
+  "TARGET_LONG_DOUBLE_128 && TARGET_VSX && FLOAT128_IBM_P (<MODE>mode)"
   "#"
   "&& reload_completed"
   [(set (match_dup 2) (match_dup 1))
@@ -6528,47 +6538,48 @@  (define_insn_and_split "extenddftf2_vsx"
   const int lo_word = LONG_DOUBLE_LARGE_FIRST ? GET_MODE_SIZE (DFmode) : 0;
   const int hi_word = LONG_DOUBLE_LARGE_FIRST ? 0 : GET_MODE_SIZE (DFmode);
 
-  operands[2] = simplify_gen_subreg (DFmode, operands[0], TFmode, hi_word);
-  operands[3] = simplify_gen_subreg (DFmode, operands[0], TFmode, lo_word);
+  operands[2] = simplify_gen_subreg (DFmode, operands[0], <MODE>mode, hi_word);
+  operands[3] = simplify_gen_subreg (DFmode, operands[0], <MODE>mode, lo_word);
   operands[4] = CONST0_RTX (DFmode);
 })
 
-(define_expand "extendsftf2"
-  [(set (match_operand:TF 0 "gpc_reg_operand" "")
-	(float_extend:TF (match_operand:SF 1 "gpc_reg_operand" "")))]
+(define_expand "extendsf<mode>2"
+  [(set (match_operand:FLOAT128 0 "gpc_reg_operand" "")
+	(float_extend:FLOAT128 (match_operand:SF 1 "gpc_reg_operand" "")))]
   "TARGET_HARD_FLOAT
    && (TARGET_FPRS || TARGET_E500_DOUBLE)
    && TARGET_LONG_DOUBLE_128"
 {
-  if (TARGET_IEEEQUAD)
+  if (FLOAT128_IEEE_P (<MODE>mode))
     rs6000_expand_float128_convert (operands[0], operands[1], false);
   else
     {
       rtx tmp = gen_reg_rtx (DFmode);
       emit_insn (gen_extendsfdf2 (tmp, operands[1]));
-      emit_insn (gen_extenddftf2 (operands[0], tmp));
+      emit_insn (gen_extenddf<mode>2 (operands[0], tmp));
     }
   DONE;
 })
 
-(define_expand "trunctfdf2"
+(define_expand "trunc<mode>df2"
   [(set (match_operand:DF 0 "gpc_reg_operand" "")
-	(float_truncate:DF (match_operand:TF 1 "gpc_reg_operand" "")))]
+	(float_truncate:DF (match_operand:FLOAT128 1 "gpc_reg_operand" "")))]
   "TARGET_HARD_FLOAT
    && (TARGET_FPRS || TARGET_E500_DOUBLE)
    && TARGET_LONG_DOUBLE_128"
 {
-  if (TARGET_IEEEQUAD)
+  if (FLOAT128_IEEE_P (<MODE>mode))
     {
       rs6000_expand_float128_convert (operands[0], operands[1], false);
       DONE;
     }
 })
 
-(define_insn_and_split "trunctfdf2_internal1"
+(define_insn_and_split "trunc<mode>df2_internal1"
   [(set (match_operand:DF 0 "gpc_reg_operand" "=d,?d")
-	(float_truncate:DF (match_operand:TF 1 "gpc_reg_operand" "0,d")))]
-  "!TARGET_IEEEQUAD && !TARGET_XL_COMPAT
+	(float_truncate:DF
+	 (match_operand:IBM128 1 "gpc_reg_operand" "0,d")))]
+  "FLOAT128_IBM_P (<MODE>mode) && !TARGET_XL_COMPAT
    && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_LONG_DOUBLE_128"
   "@
    #
@@ -6581,39 +6592,44 @@  (define_insn_and_split "trunctfdf2_inter
 }
   [(set_attr "type" "fp")])
 
-(define_insn "trunctfdf2_internal2"
+(define_insn "trunc<mode>df2_internal2"
   [(set (match_operand:DF 0 "gpc_reg_operand" "=d")
-	(float_truncate:DF (match_operand:TF 1 "gpc_reg_operand" "d")))]
-  "!TARGET_IEEEQUAD && TARGET_XL_COMPAT
-   && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_DOUBLE_FLOAT 
-   && TARGET_LONG_DOUBLE_128"
+	(float_truncate:DF (match_operand:IBM128 1 "gpc_reg_operand" "d")))]
+  "FLOAT128_IBM_P (<MODE>mode) && TARGET_XL_COMPAT && TARGET_HARD_FLOAT
+   && TARGET_FPRS && TARGET_DOUBLE_FLOAT && TARGET_LONG_DOUBLE_128"
   "fadd %0,%1,%L1"
   [(set_attr "type" "fp")
    (set_attr "fp_type" "fp_addsub_d")])
 
-(define_expand "trunctfsf2"
+(define_expand "trunc<mode>sf2"
   [(set (match_operand:SF 0 "gpc_reg_operand" "")
-	(float_truncate:SF (match_operand:TF 1 "gpc_reg_operand" "")))]
+	(float_truncate:SF (match_operand:FLOAT128 1 "gpc_reg_operand" "")))]
   "TARGET_HARD_FLOAT
    && (TARGET_FPRS || TARGET_E500_DOUBLE)
    && TARGET_LONG_DOUBLE_128"
 {
-  if (TARGET_IEEEQUAD)
+  if (FLOAT128_IEEE_P (<MODE>mode))
     rs6000_expand_float128_convert (operands[0], operands[1], false);
   else if (TARGET_E500_DOUBLE)
-    emit_insn (gen_spe_trunctfsf2 (operands[0], operands[1]));
-  else
+    {
+      gcc_assert (<MODE>mode == TFmode);
+      emit_insn (gen_spe_trunctfsf2 (operands[0], operands[1]));
+    }
+  else if (<MODE>mode == TFmode)
     emit_insn (gen_trunctfsf2_fprs (operands[0], operands[1]));
+  else if (<MODE>mode == IFmode)
+    emit_insn (gen_truncifsf2_fprs (operands[0], operands[1]));
+  else
+    gcc_unreachable ();
   DONE;
 })
 
-(define_insn_and_split "trunctfsf2_fprs"
+(define_insn_and_split "trunc<mode>sf2_fprs"
   [(set (match_operand:SF 0 "gpc_reg_operand" "=f")
-	(float_truncate:SF (match_operand:TF 1 "gpc_reg_operand" "d")))
+	(float_truncate:SF (match_operand:IBM128 1 "gpc_reg_operand" "d")))
    (clobber (match_scratch:DF 2 "=d"))]
-  "!TARGET_IEEEQUAD
-   && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_SINGLE_FLOAT 
-   && TARGET_LONG_DOUBLE_128"
+  "TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_SINGLE_FLOAT 
+   && TARGET_LONG_DOUBLE_128 && FLOAT128_IBM_P (<MODE>mode)"
   "#"
   "&& reload_completed"
   [(set (match_dup 2)
@@ -6622,56 +6638,69 @@  (define_insn_and_split "trunctfsf2_fprs"
 	(float_truncate:SF (match_dup 2)))]
   "")
 
-(define_expand "floatsitf2"
-  [(set (match_operand:TF 0 "gpc_reg_operand" "")
-        (float:TF (match_operand:SI 1 "gpc_reg_operand" "")))]
-  "!TARGET_IEEEQUAD
-   && TARGET_HARD_FLOAT
+(define_expand "floatsi<mode>2"
+  [(set (match_operand:FLOAT128 0 "gpc_reg_operand" "")
+        (float:FLOAT128 (match_operand:SI 1 "gpc_reg_operand" "")))]
+  "TARGET_HARD_FLOAT
    && (TARGET_FPRS || TARGET_E500_DOUBLE)
    && TARGET_LONG_DOUBLE_128"
 {
-  rtx tmp = gen_reg_rtx (DFmode);
-  expand_float (tmp, operands[1], false);
-  emit_insn (gen_extenddftf2 (operands[0], tmp));
+  if (FLOAT128_IEEE_P (<MODE>mode))
+    rs6000_expand_float128_convert (operands[0], operands[1], false);
+  else
+    {
+      rtx tmp = gen_reg_rtx (DFmode);
+      expand_float (tmp, operands[1], false);
+      if (<MODE>mode == TFmode)
+	emit_insn (gen_extenddftf2 (operands[0], tmp));
+      else if (<MODE>mode == IFmode)
+	emit_insn (gen_extenddfif2 (operands[0], tmp));
+      else
+	gcc_unreachable ();
+    }
   DONE;
 })
 
 ; fadd, but rounding towards zero.
 ; This is probably not the optimal code sequence.
-(define_insn "fix_trunc_helper"
+(define_insn "fix_trunc_helper<mode>"
   [(set (match_operand:DF 0 "gpc_reg_operand" "=d")
-	(unspec:DF [(match_operand:TF 1 "gpc_reg_operand" "d")]
+	(unspec:DF [(match_operand:IBM128 1 "gpc_reg_operand" "d")]
 		   UNSPEC_FIX_TRUNC_TF))
    (clobber (match_operand:DF 2 "gpc_reg_operand" "=&d"))]
-  "TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_DOUBLE_FLOAT"
+  "TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_DOUBLE_FLOAT
+   && FLOAT128_IBM_P (<MODE>mode)"
   "mffs %2\n\tmtfsb1 31\n\tmtfsb0 30\n\tfadd %0,%1,%L1\n\tmtfsf 1,%2"
   [(set_attr "type" "fp")
    (set_attr "length" "20")])
 
-(define_expand "fix_trunctfsi2"
+(define_expand "fix_trunc<mode>si2"
   [(set (match_operand:SI 0 "gpc_reg_operand" "")
-	(fix:SI (match_operand:TF 1 "gpc_reg_operand" "")))]
+	(fix:SI (match_operand:FLOAT128 1 "gpc_reg_operand" "")))]
   "TARGET_HARD_FLOAT
    && (TARGET_FPRS || TARGET_E500_DOUBLE) && TARGET_LONG_DOUBLE_128"
 {
-  if (TARGET_IEEEQUAD)
+  if (FLOAT128_IEEE_P (<MODE>mode))
     rs6000_expand_float128_convert (operands[0], operands[1], false);
-  else if (TARGET_E500_DOUBLE)
+  else if (TARGET_E500_DOUBLE && <MODE>mode == TFmode)
     emit_insn (gen_spe_fix_trunctfsi2 (operands[0], operands[1]));
-  else
+  else if (<MODE>mode == TFmode)
     emit_insn (gen_fix_trunctfsi2_fprs (operands[0], operands[1]));
+  else if (<MODE>mode == IFmode)
+    emit_insn (gen_fix_truncifsi2_fprs (operands[0], operands[1]));
+  else
+    gcc_unreachable ();
   DONE;
 })
 
-(define_expand "fix_trunctfsi2_fprs"
+(define_expand "fix_trunc<mode>si2_fprs"
   [(parallel [(set (match_operand:SI 0 "gpc_reg_operand" "")
-		   (fix:SI (match_operand:TF 1 "gpc_reg_operand" "")))
+		   (fix:SI (match_operand:IBM128 1 "gpc_reg_operand" "")))
 	      (clobber (match_dup 2))
 	      (clobber (match_dup 3))
 	      (clobber (match_dup 4))
 	      (clobber (match_dup 5))])]
-  "!TARGET_IEEEQUAD
-   && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_LONG_DOUBLE_128"
+  "TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_LONG_DOUBLE_128"
 {
   operands[2] = gen_reg_rtx (DFmode);
   operands[3] = gen_reg_rtx (DFmode);
@@ -6679,21 +6708,21 @@  (define_expand "fix_trunctfsi2_fprs"
   operands[5] = assign_stack_temp (DImode, GET_MODE_SIZE (DImode));
 })
 
-(define_insn_and_split "*fix_trunctfsi2_internal"
+(define_insn_and_split "*fix_trunc<mode>si2_internal"
   [(set (match_operand:SI 0 "gpc_reg_operand" "=r")
-        (fix:SI (match_operand:TF 1 "gpc_reg_operand" "d")))
+        (fix:SI (match_operand:IBM128 1 "gpc_reg_operand" "d")))
    (clobber (match_operand:DF 2 "gpc_reg_operand" "=d"))
    (clobber (match_operand:DF 3 "gpc_reg_operand" "=&d"))
    (clobber (match_operand:DI 4 "gpc_reg_operand" "=d"))
    (clobber (match_operand:DI 5 "offsettable_mem_operand" "=o"))]
-  "!TARGET_IEEEQUAD
-   && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_LONG_DOUBLE_128"
+  "TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_LONG_DOUBLE_128"
   "#"
   ""
   [(pc)]
 {
   rtx lowword;
-  emit_insn (gen_fix_trunc_helper (operands[2], operands[1], operands[3]));
+  emit_insn (gen_fix_trunc_helper<mode> (operands[2], operands[1],
+					 operands[3]));
 
   gcc_assert (MEM_P (operands[5]));
   lowword = adjust_address (operands[5], SImode, WORDS_BIG_ENDIAN ? 4 : 0);
@@ -6704,45 +6733,45 @@  (define_insn_and_split "*fix_trunctfsi2_
   DONE;
 })
 
-(define_expand "fix_trunctfdi2"
-  [(set (match_operand:DI 0 "nonimmediate_operand" "")
-	(fix:DI (match_operand:TF 1 "gpc_reg_operand" "")))]
-  "TARGET_IEEEQUAD && TARGET_LONG_DOUBLE_128"
+(define_expand "fix_trunc<mode>di2"
+  [(set (match_operand:DI 0 "gpc_reg_operand" "")
+	(fix:DI (match_operand:IEEE128 1 "gpc_reg_operand" "")))]
+  "TARGET_FLOAT128"
 {
   rs6000_expand_float128_convert (operands[0], operands[1], false);
   DONE;
 })
 
-(define_expand "fixuns_trunctf<mode>2"
-  [(set (match_operand:SDI 0 "nonimmediate_operand" "")
-	(unsigned_fix:SDI (match_operand:TF 1 "gpc_reg_operand" "")))]
-  "TARGET_IEEEQUAD && TARGET_LONG_DOUBLE_128"
+(define_expand "fixuns_trunc<IEEE128:mode><SDI:mode>2"
+  [(set (match_operand:SDI 0 "gpc_reg_operand" "")
+	(unsigned_fix:SDI (match_operand:IEEE128 1 "gpc_reg_operand" "")))]
+  "TARGET_FLOAT128"
 {
   rs6000_expand_float128_convert (operands[0], operands[1], true);
   DONE;
 })
 
-(define_expand "floatditf2"
-  [(set (match_operand:TF 0 "nonimmediate_operand" "")
-	(float:TF (match_operand:DI 1 "gpc_reg_operand" "")))]
-  "TARGET_IEEEQUAD && TARGET_LONG_DOUBLE_128"
+(define_expand "floatdi<mode>2"
+  [(set (match_operand:IEEE128 0 "gpc_reg_operand" "")
+	(float:IEEE128 (match_operand:DI 1 "gpc_reg_operand" "")))]
+  "TARGET_FLOAT128"
 {
   rs6000_expand_float128_convert (operands[0], operands[1], false);
   DONE;
 })
 
-(define_expand "floatuns<mode>tf2"
-  [(set (match_operand:TF 0 "nonimmediate_operand" "")
-	(unsigned_float:TF (match_operand:SDI 1 "gpc_reg_operand" "")))]
-  "TARGET_IEEEQUAD && TARGET_LONG_DOUBLE_128"
+(define_expand "floatuns<SDI:mode><IEEE128:mode>2"
+  [(set (match_operand:IEEE128 0 "gpc_reg_operand" "")
+	(unsigned_float:IEEE128 (match_operand:SDI 1 "gpc_reg_operand" "")))]
+  "TARGET_FLOAT128"
 {
   rs6000_expand_float128_convert (operands[0], operands[1], true);
   DONE;
 })
 
 (define_expand "neg<mode>2"
-  [(set (match_operand:TFIFKF 0 "gpc_reg_operand" "")
-	(neg:TFIFKF (match_operand:TFIFKF 1 "gpc_reg_operand" "")))]
+  [(set (match_operand:FLOAT128 0 "gpc_reg_operand" "")
+	(neg:FLOAT128 (match_operand:FLOAT128 1 "gpc_reg_operand" "")))]
   "FLOAT128_IEEE_P (<MODE>mode)
    || (FLOAT128_IBM_P (<MODE>mode)
        && TARGET_HARD_FLOAT
@@ -6752,7 +6781,14 @@  (define_expand "neg<mode>2"
   if (FLOAT128_IEEE_P (<MODE>mode))
     {
       if (TARGET_FLOAT128)
-	emit_insn (gen_ieee_128bit_vsx_neg<mode>2 (operands[0], operands[1]));
+	{
+	  if (<MODE>mode == TFmode)
+	    emit_insn (gen_ieee_128bit_vsx_negtf2 (operands[0], operands[1]));
+	  else if (<MODE>mode == KFmode)
+	    emit_insn (gen_ieee_128bit_vsx_negkf2 (operands[0], operands[1]));
+	  else
+	    gcc_unreachable ();
+	}
       else
 	{
 	  rtx libfunc = optab_libfunc (neg_optab, <MODE>mode);
@@ -6767,9 +6803,9 @@  (define_expand "neg<mode>2"
     }
 }")
 
-(define_insn "negtf2_internal"
-  [(set (match_operand:TF 0 "gpc_reg_operand" "=d")
-	(neg:TF (match_operand:TF 1 "gpc_reg_operand" "d")))]
+(define_insn "neg<mode>2_internal"
+  [(set (match_operand:IBM128 0 "gpc_reg_operand" "=d")
+	(neg:IBM128 (match_operand:IBM128 1 "gpc_reg_operand" "d")))]
   "TARGET_HARD_FLOAT && TARGET_FPRS && FLOAT128_IBM_P (TFmode)"
   "*
 {
@@ -6782,8 +6818,8 @@  (define_insn "negtf2_internal"
    (set_attr "length" "8")])
 
 (define_expand "abs<mode>2"
-  [(set (match_operand:TFIFKF 0 "gpc_reg_operand" "")
-	(abs:TFIFKF (match_operand:TFIFKF 1 "gpc_reg_operand" "")))]
+  [(set (match_operand:FLOAT128 0 "gpc_reg_operand" "")
+	(abs:FLOAT128 (match_operand:FLOAT128 1 "gpc_reg_operand" "")))]
   "FLOAT128_IEEE_P (<MODE>mode)
    || (FLOAT128_IBM_P (<MODE>mode)
        && TARGET_HARD_FLOAT
@@ -6796,7 +6832,12 @@  (define_expand "abs<mode>2"
     {
       if (TARGET_FLOAT128)
 	{
-	  emit_insn (gen_ieee_128bit_vsx_abs<mode>2 (operands[0], operands[1]));
+	  if (<MODE>mode == TFmode)
+	    emit_insn (gen_ieee_128bit_vsx_abstf2 (operands[0], operands[1]));
+	  else if (<MODE>mode == KFmode)
+	    emit_insn (gen_ieee_128bit_vsx_abskf2 (operands[0], operands[1]));
+	  else
+	    FAIL;
 	  DONE;
 	}
       else
@@ -6804,22 +6845,26 @@  (define_expand "abs<mode>2"
     }
 
   label = gen_label_rtx ();
-  if (TARGET_E500_DOUBLE)
+  if (TARGET_E500_DOUBLE && <MODE>mode == TFmode)
     {
       if (flag_finite_math_only && !flag_trapping_math)
 	emit_insn (gen_spe_abstf2_tst (operands[0], operands[1], label));
       else
 	emit_insn (gen_spe_abstf2_cmp (operands[0], operands[1], label));
     }
-  else
+  else if (<MODE>mode == TFmode)
     emit_insn (gen_abstf2_internal (operands[0], operands[1], label));
+  else if (<MODE>mode == TFmode)
+    emit_insn (gen_absif2_internal (operands[0], operands[1], label));
+  else
+    FAIL;
   emit_label (label);
   DONE;
 }")
 
-(define_expand "abstf2_internal"
-  [(set (match_operand:TF 0 "gpc_reg_operand" "")
-	(match_operand:TF 1 "gpc_reg_operand" ""))
+(define_expand "abs<mode>2_internal"
+  [(set (match_operand:IBM128 0 "gpc_reg_operand" "")
+	(match_operand:IBM128 1 "gpc_reg_operand" ""))
    (set (match_dup 3) (match_dup 5))
    (set (match_dup 5) (abs:DF (match_dup 5)))
    (set (match_dup 4) (compare:CCFP (match_dup 3) (match_dup 5)))
@@ -6827,8 +6872,7 @@  (define_expand "abstf2_internal"
 			   (label_ref (match_operand 2 "" ""))
 			   (pc)))
    (set (match_dup 6) (neg:DF (match_dup 6)))]
-  "!TARGET_IEEEQUAD
-   && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_DOUBLE_FLOAT 
+  "TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_DOUBLE_FLOAT 
    && TARGET_LONG_DOUBLE_128"
   "
 {
@@ -6836,8 +6880,8 @@  (define_expand "abstf2_internal"
   const int lo_word = LONG_DOUBLE_LARGE_FIRST ? GET_MODE_SIZE (DFmode) : 0;
   operands[3] = gen_reg_rtx (DFmode);
   operands[4] = gen_reg_rtx (CCFPmode);
-  operands[5] = simplify_gen_subreg (DFmode, operands[0], TFmode, hi_word);
-  operands[6] = simplify_gen_subreg (DFmode, operands[0], TFmode, lo_word);
+  operands[5] = simplify_gen_subreg (DFmode, operands[0], <MODE>mode, hi_word);
+  operands[6] = simplify_gen_subreg (DFmode, operands[0], <MODE>mode, lo_word);
 }")
 
 
@@ -6870,14 +6914,14 @@  (define_expand "ieee_128bit_negative_zer
 ;; neg/abs to create the constant just once.
 
 (define_insn_and_split "ieee_128bit_vsx_neg<mode>2"
-  [(set (match_operand:TFIFKF 0 "register_operand" "=wa")
-	(neg:TFIFKF (match_operand:TFIFKF 1 "register_operand" "wa")))
+  [(set (match_operand:IEEE128 0 "register_operand" "=wa")
+	(neg:IEEE128 (match_operand:IEEE128 1 "register_operand" "wa")))
    (clobber (match_scratch:V16QI 2 "=v"))]
-  "TARGET_FLOAT128 && FLOAT128_IEEE_P (<MODE>mode)"
+  "TARGET_FLOAT128"
   "#"
   "&& 1"
   [(parallel [(set (match_dup 0)
-		   (neg:TFIFKF (match_dup 1)))
+		   (neg:IEEE128 (match_dup 1)))
 	      (use (match_dup 2))])]
 {
   if (GET_CODE (operands[2]) == SCRATCH)
@@ -6890,8 +6934,8 @@  (define_insn_and_split "ieee_128bit_vsx_
    (set_attr "type" "vecsimple")])
 
 (define_insn "*ieee_128bit_vsx_neg<mode>2_internal"
-  [(set (match_operand:TFIFKF 0 "register_operand" "=wa")
-	(neg:TFIFKF (match_operand:TFIFKF 1 "register_operand" "wa")))
+  [(set (match_operand:IEEE128 0 "register_operand" "=wa")
+	(neg:IEEE128 (match_operand:IEEE128 1 "register_operand" "wa")))
    (use (match_operand:V16QI 2 "register_operand" "=v"))]
   "TARGET_FLOAT128"
   "xxlxor %x0,%x1,%x2"
@@ -6899,14 +6943,14 @@  (define_insn "*ieee_128bit_vsx_neg<mode>
 
 ;; IEEE 128-bit absolute value
 (define_insn_and_split "ieee_128bit_vsx_abs<mode>2"
-  [(set (match_operand:TFIFKF 0 "register_operand" "=wa")
-	(abs:TFIFKF (match_operand:TFIFKF 1 "register_operand" "wa")))
+  [(set (match_operand:IEEE128 0 "register_operand" "=wa")
+	(abs:IEEE128 (match_operand:IEEE128 1 "register_operand" "wa")))
    (clobber (match_scratch:V16QI 2 "=v"))]
   "TARGET_FLOAT128 && FLOAT128_IEEE_P (<MODE>mode)"
   "#"
   "&& 1"
   [(parallel [(set (match_dup 0)
-		   (abs:TFIFKF (match_dup 1)))
+		   (abs:IEEE128 (match_dup 1)))
 	      (use (match_dup 2))])]
 {
   if (GET_CODE (operands[2]) == SCRATCH)
@@ -6919,8 +6963,8 @@  (define_insn_and_split "ieee_128bit_vsx_
    (set_attr "type" "vecsimple")])
 
 (define_insn "*ieee_128bit_vsx_abs<mode>2_internal"
-  [(set (match_operand:TFIFKF 0 "register_operand" "=wa")
-	(abs:TFIFKF (match_operand:TFIFKF 1 "register_operand" "wa")))
+  [(set (match_operand:IEEE128 0 "register_operand" "=wa")
+	(abs:IEEE128 (match_operand:IEEE128 1 "register_operand" "wa")))
    (use (match_operand:V16QI 2 "register_operand" "=v"))]
   "TARGET_FLOAT128"
   "xxlandc %x0,%x1,%x2"
@@ -6928,16 +6972,16 @@  (define_insn "*ieee_128bit_vsx_abs<mode>
 
 ;; IEEE 128-bit negative absolute value
 (define_insn_and_split "*ieee_128bit_vsx_nabs<mode>2"
-  [(set (match_operand:TFIFKF 0 "register_operand" "=wa")
-	(neg:TFIFKF
-	 (abs:TFIFKF
-	  (match_operand:TFIFKF 1 "register_operand" "wa"))))
+  [(set (match_operand:IEEE128 0 "register_operand" "=wa")
+	(neg:IEEE128
+	 (abs:IEEE128
+	  (match_operand:IEEE128 1 "register_operand" "wa"))))
    (clobber (match_scratch:V16QI 2 "=v"))]
   "TARGET_FLOAT128 && FLOAT128_IEEE_P (<MODE>mode)"
   "#"
   "&& 1"
   [(parallel [(set (match_dup 0)
-		   (abs:TFIFKF (match_dup 1)))
+		   (abs:IEEE128 (match_dup 1)))
 	      (use (match_dup 2))])]
 {
   if (GET_CODE (operands[2]) == SCRATCH)
@@ -6950,70 +6994,78 @@  (define_insn_and_split "*ieee_128bit_vsx
    (set_attr "type" "vecsimple")])
 
 (define_insn "*ieee_128bit_vsx_nabs<mode>2_internal"
-  [(set (match_operand:TFIFKF 0 "register_operand" "=wa")
-	(neg:TFIFKF
-	 (abs:TFIFKF
-	  (match_operand:TFIFKF 1 "register_operand" "wa"))))
+  [(set (match_operand:IEEE128 0 "register_operand" "=wa")
+	(neg:IEEE128
+	 (abs:IEEE128
+	  (match_operand:IEEE128 1 "register_operand" "wa"))))
    (use (match_operand:V16QI 2 "register_operand" "=v"))]
   "TARGET_FLOAT128"
   "xxlor %x0,%x1,%x2"
   [(set_attr "type" "vecsimple")])
 
 ;; Float128 conversion functions.  These expand to library function calls.
+;; We use expand to convert from IBM double double to IEEE 128-bit
+;; and trunc for the opposite.
+(define_expand "extendiftf2"
+  [(set (match_operand:TF 0 "gpc_reg_operand" "")
+	(float_extend:TF (match_operand:IF 1 "gpc_reg_operand" "")))]
+  "TARGET_FLOAT128"
+{
+  rs6000_expand_float128_convert (operands[0], operands[1], false);
+  DONE;
+})
 
-(define_expand "extend<FLOAT128_SFDFTF:mode><IFKF:mode>2"
-  [(set (match_operand:IFKF 0 "nonimmediate_operand" "")
-	(float_extend:IFKF
-	 (match_operand:FLOAT128_SFDFTF 1 "gpc_reg_operand" "")))]
+(define_expand "extendifkf2"
+  [(set (match_operand:KF 0 "gpc_reg_operand" "")
+	(float_extend:KF (match_operand:IF 1 "gpc_reg_operand" "")))]
   "TARGET_FLOAT128"
 {
   rs6000_expand_float128_convert (operands[0], operands[1], false);
   DONE;
 })
 
-(define_expand "trunc<IFKF:mode><FLOAT128_SFDFTF:mode>2"
-  [(set (match_operand:FLOAT128_SFDFTF 0 "nonimmediate_operand" "")
-	(float_truncate:FLOAT128_SFDFTF
-	 (match_operand:IFKF 1 "gpc_reg_operand" "")))]
+(define_expand "extendtfkf2"
+  [(set (match_operand:KF 0 "gpc_reg_operand" "")
+	(float_extend:KF (match_operand:TF 1 "gpc_reg_operand" "")))]
   "TARGET_FLOAT128"
 {
   rs6000_expand_float128_convert (operands[0], operands[1], false);
   DONE;
 })
 
-(define_expand "fix_trunc<IFKF:mode><SDI:mode>2"
-  [(set (match_operand:SDI 0 "nonimmediate_operand" "")
-	(fix:SDI (match_operand:IFKF 1 "gpc_reg_operand" "")))]
+(define_expand "trunciftf2"
+  [(set (match_operand:IF 0 "gpc_reg_operand" "")
+	(float_truncate:IF (match_operand:TF 1 "gpc_reg_operand" "")))]
   "TARGET_FLOAT128"
 {
   rs6000_expand_float128_convert (operands[0], operands[1], false);
   DONE;
 })
 
-(define_expand "fixuns_trunc<IFKF:mode><SDI:mode>2"
-  [(set (match_operand:SDI 0 "nonimmediate_operand" "")
-	(unsigned_fix:SDI (match_operand:IFKF 1 "gpc_reg_operand" "")))]
+(define_expand "truncifkf2"
+  [(set (match_operand:IF 0 "gpc_reg_operand" "")
+	(float_truncate:IF (match_operand:KF 1 "gpc_reg_operand" "")))]
   "TARGET_FLOAT128"
 {
-  rs6000_expand_float128_convert (operands[0], operands[1], true);
+  rs6000_expand_float128_convert (operands[0], operands[1], false);
   DONE;
 })
 
-(define_expand "float<SDI:mode><IFKF:mode>2"
-  [(set (match_operand:IFKF 0 "nonimmediate_operand" "")
-	(float:KF (match_operand:SDI 1 "gpc_reg_operand" "")))]
+(define_expand "trunckftf2"
+  [(set (match_operand:TF 0 "gpc_reg_operand" "")
+	(float_truncate:TF (match_operand:KF 1 "gpc_reg_operand" "")))]
   "TARGET_FLOAT128"
 {
   rs6000_expand_float128_convert (operands[0], operands[1], false);
   DONE;
 })
 
-(define_expand "floatuns<SDI:mode><IFKF:mode>2"
-  [(set (match_operand:IFKF 0 "nonimmediate_operand" "")
-	(unsigned_float:IFKF (match_operand:SDI 1 "gpc_reg_operand" "")))]
+(define_expand "trunctfif2"
+  [(set (match_operand:IF 0 "gpc_reg_operand" "")
+	(float_truncate:IF (match_operand:TF 1 "gpc_reg_operand" "")))]
   "TARGET_FLOAT128"
 {
-  rs6000_expand_float128_convert (operands[0], operands[1], true);
+  rs6000_expand_float128_convert (operands[0], operands[1], false);
   DONE;
 })
 
@@ -10530,20 +10582,20 @@  (define_split
    (set (match_dup 0) (plus:SI (match_dup 1) (match_dup 4)))])
 
 ;; Only need to compare second words if first words equal
-(define_insn "*cmptf_internal1"
+(define_insn "*cmp<mode>_internal1"
   [(set (match_operand:CCFP 0 "cc_reg_operand" "=y")
-	(compare:CCFP (match_operand:TF 1 "gpc_reg_operand" "d")
-		      (match_operand:TF 2 "gpc_reg_operand" "d")))]
-  "!TARGET_IEEEQUAD && !TARGET_XL_COMPAT
+	(compare:CCFP (match_operand:IBM128 1 "gpc_reg_operand" "d")
+		      (match_operand:IBM128 2 "gpc_reg_operand" "d")))]
+  "!TARGET_XL_COMPAT
    && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_DOUBLE_FLOAT && TARGET_LONG_DOUBLE_128"
   "fcmpu %0,%1,%2\;bne %0,$+8\;fcmpu %0,%L1,%L2"
   [(set_attr "type" "fpcompare")
    (set_attr "length" "12")])
 
-(define_insn_and_split "*cmptf_internal2"
+(define_insn_and_split "*cmp<mode>_internal2"
   [(set (match_operand:CCFP 0 "cc_reg_operand" "=y")
-	(compare:CCFP (match_operand:TF 1 "gpc_reg_operand" "d")
-		      (match_operand:TF 2 "gpc_reg_operand" "d")))
+	(compare:CCFP (match_operand:IEEE128 1 "gpc_reg_operand" "d")
+		      (match_operand:IEEE128 2 "gpc_reg_operand" "d")))
     (clobber (match_scratch:DF 3 "=d"))
     (clobber (match_scratch:DF 4 "=d"))
     (clobber (match_scratch:DF 5 "=d"))
@@ -10553,7 +10605,7 @@  (define_insn_and_split "*cmptf_internal2
     (clobber (match_scratch:DF 9 "=d"))
     (clobber (match_scratch:DF 10 "=d"))
     (clobber (match_scratch:GPR 11 "=b"))]
-  "!TARGET_IEEEQUAD && TARGET_XL_COMPAT
+  "TARGET_XL_COMPAT
    && TARGET_HARD_FLOAT && TARGET_FPRS && TARGET_DOUBLE_FLOAT && TARGET_LONG_DOUBLE_128"
   "#"
   "&& reload_completed"
@@ -10577,10 +10629,10 @@  (define_insn_and_split "*cmptf_internal2
   const int lo_word = LONG_DOUBLE_LARGE_FIRST ? GET_MODE_SIZE (DFmode) : 0;
   const int hi_word = LONG_DOUBLE_LARGE_FIRST ? 0 : GET_MODE_SIZE (DFmode);
 
-  operands[5] = simplify_gen_subreg (DFmode, operands[1], TFmode, hi_word);
-  operands[6] = simplify_gen_subreg (DFmode, operands[1], TFmode, lo_word);
-  operands[7] = simplify_gen_subreg (DFmode, operands[2], TFmode, hi_word);
-  operands[8] = simplify_gen_subreg (DFmode, operands[2], TFmode, lo_word);
+  operands[5] = simplify_gen_subreg (DFmode, operands[1], <MODE>mode, hi_word);
+  operands[6] = simplify_gen_subreg (DFmode, operands[1], <MODE>mode, lo_word);
+  operands[7] = simplify_gen_subreg (DFmode, operands[2], <MODE>mode, hi_word);
+  operands[8] = simplify_gen_subreg (DFmode, operands[2], <MODE>mode, lo_word);
   operands[12] = gen_label_rtx ();
   operands[13] = gen_label_rtx ();
   real_inf (&rv);
Index: gcc/config/rs6000/rs6000-opts.h
===================================================================
--- gcc/config/rs6000/rs6000-opts.h	(revision 228506)
+++ gcc/config/rs6000/rs6000-opts.h	(working copy)
@@ -80,14 +80,6 @@  enum fpu_type_t
 };
 
 
-/* Float128 support.  */
-enum float128_type_t
-{
-  FLOAT128_UNSET	= -1,	/* Initial value.  */
-  FLOAT128_NONE,		/* No __float128 support.  */
-  FLOAT128_SW			/* software __float128 support.  */
-};
-
 /* Types of costly dependences.  */
 enum rs6000_dependence_cost
 {
Index: gcc/config/rs6000/predicates.md
===================================================================
--- gcc/config/rs6000/predicates.md	(revision 228545)
+++ gcc/config/rs6000/predicates.md	(working copy)
@@ -684,7 +684,7 @@  (define_predicate "indexed_or_indirect_o
 ;; Like indexed_or_indirect_operand, but also allow a GPR register if direct
 ;; moves are supported.
 (define_predicate "reg_or_indexed_operand"
-  (match_code "mem,reg")
+  (match_code "mem,reg,subreg")
 {
   if (MEM_P (op))
     return indexed_or_indirect_operand (op, mode);
Index: gcc/doc/extend.texi
===================================================================
--- gcc/doc/extend.texi	(revision 228506)
+++ gcc/doc/extend.texi	(working copy)
@@ -929,6 +929,7 @@  examine and set these two fictitious var
 @cindex additional floating types
 @cindex @code{__float80} data type
 @cindex @code{__float128} data type
+@cindex @code{__ibm128} data type
 @cindex @code{w} floating point suffix
 @cindex @code{q} floating point suffix
 @cindex @code{W} floating point suffix
@@ -941,19 +942,39 @@  Support for additional types includes th
 add, subtract, multiply, divide; unary arithmetic operators;
 relational operators; equality operators; and conversions to and from
 integer and other floating types.  Use a suffix @samp{w} or @samp{W}
-in a literal constant of type @code{__float80} and @samp{q} or @samp{Q}
-for @code{_float128}.  You can declare complex types using the
-corresponding internal complex type, @code{XCmode} for @code{__float80}
-type and @code{TCmode} for @code{__float128} type:
+in a literal constant of type @code{__float80} or type
+@code{__ibm128}.  Use a suffix @samp{q} or @samp{Q} for @code{_float128}.
+
+On the i386, x86_64, IA-64, and HP-UX targets, you can declare complex
+types using the corresponding internal complex type, @code{XCmode} for
+@code{__float80} type and @code{TCmode} for @code{__float128} type:
 
 @smallexample
 typedef _Complex float __attribute__((mode(TC))) _Complex128;
 typedef _Complex float __attribute__((mode(XC))) _Complex80;
 @end smallexample
 
+On PowerPC Linux, Freebsd and Darwin systems, the default for
+@code{long double} is to use the IBM extended floating point format
+that uses a pair of @code{double} values to extend the precision.
+This means that the mode @code{TCmode} was already used by the
+traditional IBM long double format, and you would need to use the mode
+@code{KCmode}:
+
+@smallexample
+typedef _Complex float __attribute__((mode(KC))) _Complex128;
+@end smallexample
+
 Not all targets support additional floating-point types.  @code{__float80}
 and @code{__float128} types are supported on x86 and IA-64 targets.
-The @code{__float128} type is supported on hppa HP-UX targets.
+The @code{__float128} type is supported on hppa HP-UX.
+The @code{__float128} type is supported on PowerPC systems by default
+if the vector scalar instruction set (VSX) is enabled.
+
+On the PowerPC, @code{__ibm128} provides access to the IBM extended
+double format, and it is intended to be used by the library functions
+that handle conversions if/when long double is changed to be IEEE
+128-bit floating point.
 
 @node Half-Precision
 @section Half-Precision Floating Point
@@ -13329,6 +13350,8 @@  uint64_t __builtin_ppc_get_timebase ();
 unsigned long __builtin_ppc_mftb ();
 double __builtin_unpack_longdouble (long double, int);
 long double __builtin_pack_longdouble (double, double);
+double __builtin_unpack_ibm128 (long double, int);
+__ibm128 __builtin_pack_ibm128 (double, double);
 @end smallexample
 
 The @code{vec_rsqrt}, @code{__builtin_rsqrt}, and
Index: gcc/doc/invoke.texi
===================================================================
--- gcc/doc/invoke.texi	(revision 228506)
+++ gcc/doc/invoke.texi	(working copy)
@@ -943,7 +943,8 @@  See RS/6000 and PowerPC Options.
 -mquad-memory-atomic -mno-quad-memory-atomic @gol
 -mcompat-align-parm -mno-compat-align-parm @gol
 -mupper-regs-df -mno-upper-regs-df -mupper-regs-sf -mno-upper-regs-sf @gol
--mupper-regs -mno-upper-regs}
+-mupper-regs -mno-upper-regs @gol
+-mfloat128 -mno-float128}
 
 @emph{RX Options}
 @gccoptlist{-m64bit-doubles  -m32bit-doubles  -fpu  -nofpu@gol
@@ -19466,6 +19467,17 @@  floating point register set, depending o
 If the @option{-mno-upper-regs} option is used, it turns off both
 @option{-mupper-regs-sf} and @option{-mupper-regs-df} options.
 
+@item -mfloat128
+@itemx -mno-float128
+@opindex mfloat128
+@opindex mno-float128
+Enable/disable the @var{__float128} keyword for IEEE 128-bit floating point
+and use software emulation for IEEE 128-bit floating point.
+
+The VSX instruction set (@option{-mvsx}, @option{-mcpu=power7}, or
+@option{-mcpu=power8}) must be enabled to use the @option{-mfloat128}
+option.
+
 @item -mfloat-gprs=@var{yes/single/double/no}
 @itemx -mfloat-gprs
 @opindex mfloat-gprs
Index: gcc/testsuite/gcc.target/powerpc/float128-call.c
===================================================================
--- gcc/testsuite/gcc.target/powerpc/float128-call.c	(revision 0)
+++ gcc/testsuite/gcc.target/powerpc/float128-call.c	(revision 0)
@@ -0,0 +1,27 @@ 
+/* { dg-do compile { target { powerpc*-*-linux* } } } */
+/* { dg-skip-if "" { powerpc*-*-darwin* } { "*" } { "" } } */
+/* { dg-require-effective-target powerpc_vsx_ok } */
+/* { dg-skip-if "do not override -mcpu" { powerpc*-*-* } { "-mcpu=*" } { "-mcpu=power7" } } */
+/* { dg-options "-O2 -mcpu=power7 -mfloat128 -mno-regnames" } */
+
+#ifndef __FLOAT128__
+#error "-mfloat128 is not supported."
+#endif
+
+#ifdef __LONG_DOUBLE_IEEE128__
+#define TYPE long double
+#define ONE  1.0L
+
+#else
+#define TYPE __float128
+#define ONE  1.0Q
+#endif
+
+/* Test to make sure vector registers are used for passing IEEE 128-bit
+   floating point values and returning them. Also make sure the 'q' suffix is
+   handled.  */
+TYPE one (void) { return ONE; }
+void store (TYPE a, TYPE *p) { *p = a; }
+
+/* { dg-final { scan-assembler "lxvd2x 34"  } } */
+/* { dg-final { scan-assembler "stxvd2x 34" } } */