Patchwork Making it easy to use -flto for bootstrap

login
register
mail settings
Submitter Alexandre Oliva
Date June 17, 2010, 7:18 p.m.
Message ID <orzkytii3a.fsf@livre.localdomain>
Download mbox | patch
Permalink /patch/56086/
State New
Headers show

Comments

Alexandre Oliva - June 17, 2010, 7:18 p.m.
I've been playing with -flto bootstraps.

The first problem I ran into was that LTO sections may differ depending
on the presence of debug stmts, so compare-debug needed some tweaking.

Furthermore, simply adding -flto to BOOT_CFLAGS didn't quite work,
because Ada doesn't seem to be ready for LTO.  I got a crash during
linking and didn't pursue it any further.  Adding -fno-lto to
BOOT_ADAFLAGS was the work-around I ended up with.

To make my partly-automated testing easier, I added bootstrap-lto as a
BUILD_CONFIG option.

I didn't have much luck with -flto in target libraries (bad relocs
creating shared libs on x86_64-linux-gnu IIRC), so I left it out.  Was
-flto supposed to work for shared libs?

I ran into -fcompare-debug failures with -flto on x86_64-linux-gnu, and
also with bootstrap-O3 on ia64-lixnu-gnu, but I haven't looked into them
yet.

This is the patch I'm going to install if there aren't objections within
24 hours or so.
Eric Botcazou - June 17, 2010, 7:32 p.m.
> Furthermore, simply adding -flto to BOOT_CFLAGS didn't quite work,
> because Ada doesn't seem to be ready for LTO.

No, it's supposed to work on mainline, there are LTO tests in gnat.dg.
Richard Guenther - June 18, 2010, 9:26 a.m.
On Thu, Jun 17, 2010 at 9:32 PM, Eric Botcazou <ebotcazou@adacore.com> wrote:
>> Furthermore, simply adding -flto to BOOT_CFLAGS didn't quite work,
>> because Ada doesn't seem to be ready for LTO.
>
> No, it's supposed to work on mainline, there are LTO tests in gnat.dg.

LTO bootstrap indeed fails with Ada (first you run into ICEs due
to -g, then if building without that you get incompatible types of
global syms inside the RTS for some Ada to C bindings if I
understand correctly).

To reproduce:

../configure --enable-stage1-languages=c,lto --enable-gold
--with-plugin-ld=/usr/bin/gold
make BOOT_CFLAGS="-O2 -g -fuse-linker-plugin -fwhopr -B
/path-to-build/prev-lto-plugin/.libs"

same with -flto, for convenience parallelize link with -fwhopr=N.  Add
STAGE1_CFLAGS="-O2 -g" for your convenience.

It would be nice to have small reproducers for the -g ICEs.

Richard.

> --
> Eric Botcazou
>
Richard Guenther - June 18, 2010, 9:31 a.m.
On Thu, Jun 17, 2010 at 9:18 PM, Alexandre Oliva <aoliva@redhat.com> wrote:
> I've been playing with -flto bootstraps.
>
> The first problem I ran into was that LTO sections may differ depending
> on the presence of debug stmts, so compare-debug needed some tweaking.
>
> Furthermore, simply adding -flto to BOOT_CFLAGS didn't quite work,
> because Ada doesn't seem to be ready for LTO.  I got a crash during
> linking and didn't pursue it any further.  Adding -fno-lto to
> BOOT_ADAFLAGS was the work-around I ended up with.

The build-config should use WHOPR, not LTO.  So -fwhopr=N
and -fno-whopr (but where and how to get N from?).

I thought about removing the current --enable-itermodule and
add --with-whopr=N or so.  Can the build-config get that N somehow?

Note that LTO bootstrap needs gold to do anything reasonable.
Note that BOOT_CFLAGs can also use -fwhole-program (if gold
and the linker-plugin is used).  Note that plugins defeat that ...

> To make my partly-automated testing easier, I added bootstrap-lto as a
> BUILD_CONFIG option.
>
> I didn't have much luck with -flto in target libraries (bad relocs
> creating shared libs on x86_64-linux-gnu IIRC), so I left it out.  Was
> -flto supposed to work for shared libs?

Yes.  But I never tried.  Note that it will be more useful to have
LTO sections in static libraries (in fact I think GNU ld will remove
all LTO sections from shared libs at link time).

> I ran into -fcompare-debug failures with -flto on x86_64-linux-gnu, and
> also with bootstrap-O3 on ia64-lixnu-gnu, but I haven't looked into them
> yet.

Hm, that works for me.

Richard.

> This is the patch I'm going to install if there aren't objections within
> 24 hours or so.
>
>
>
>
> --
> Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
> You must be the change you wish to see in the world. -- Gandhi
> Be Free! -- http://FSFLA.org/   FSF Latin America board member
> Free Software Evangelist      Red Hat Brazil Compiler Engineer
>
>
Alexandre Oliva - June 21, 2010, 12:21 a.m.
On Jun 18, 2010, Richard Guenther <richard.guenther@gmail.com> wrote:

> The build-config should use WHOPR, not LTO.

I'm not sure I fully understand the difference, even after looking at
the docs.  Clearly using -flto had some effect, and we have tests for
both -flto and -fwhopr, so I went ahead and installed this patch.  I
suppose it would make sense to introduce bootstrap-whopr as well.

> So -fwhopr=N and -fno-whopr (but where and how to get N from?).

Perhaps some --with-whopr configure option could optionally set a
Makefile variable, which a bootstrap-whopr configuration could use.  The
Makefile variable could always be overridden in the command line, of
course.
Jan Hubicka - June 21, 2010, 8:29 a.m.
> On Jun 18, 2010, Richard Guenther <richard.guenther@gmail.com> wrote:
> 
> > The build-config should use WHOPR, not LTO.
> 
> I'm not sure I fully understand the difference, even after looking at
> the docs.  Clearly using -flto had some effect, and we have tests for
> both -flto and -fwhopr, so I went ahead and installed this patch.  I
> suppose it would make sense to introduce bootstrap-whopr as well.

-fwhopr allows parallel build, so it gets you faster to resulting binary and
use less memory. Result should be pretty much same as with -flto (because of
partitioning process the some of late IPA propagation might be lost, but
according to SPEC benchmark there is almost 0 difference).

So I am not sure we need both LTO and WHOPR make, WHOPR should be IMO sufficient.

The difference in between -fwhopr and -flto is indeed bit difficult to parse
for normal user. In future I would like to whopr to be enabled with -flto (or
be -flto subswitch), but I did not had chance to propose change like this.

I would also like to have way to combine profiling and whopr/lto.
What should be interface here?  Make profiledbootstrap-whopr?

Honza
> 
> > So -fwhopr=N and -fno-whopr (but where and how to get N from?).
> 
> Perhaps some --with-whopr configure option could optionally set a
> Makefile variable, which a bootstrap-whopr configuration could use.  The
> Makefile variable could always be overridden in the command line, of
> course.
> 
> -- 
> Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
> You must be the change you wish to see in the world. -- Gandhi
> Be Free! -- http://FSFLA.org/   FSF Latin America board member
> Free Software Evangelist      Red Hat Brazil Compiler Engineer
Jan Hubicka - June 21, 2010, 8:37 a.m.
> +# This option enables LTO for stage2 and stage3.  It requires lto to
> +# be enabled for stage1 with --enable-stage1-languages.
> +
> +STAGE2_CFLAGS += -flto
> +STAGE3_CFLAGS += -flto

Also to make this useful (i.e. produce better binary), one needs -fwhole-program. This also need
to build with Gold plugin as used in Richard's example.

Honza
> +
> +# Ada fails to build with LTO, turn it off for now.
> +BOOT_ADAFLAGS += -fno-lto
> Index: gcc/doc/install.texi
> ===================================================================
> --- gcc/doc/install.texi.orig	2010-06-16 16:56:26.000000000 -0300
> +++ gcc/doc/install.texi	2010-06-16 16:58:10.000000000 -0300
> @@ -2150,6 +2150,11 @@ Removes any @option{-O}-started option f
>  @item @samp{bootstrap-O3}
>  Analogous to @code{bootstrap-O1}.
>  
> +@item @samp{bootstrap-lto}
> +Enables Link-Time Optimization for host tools during bootstrapping.
> +@samp{BUILD_CONFIG=bootstrap-lto} is equivalent to adding
> +@option{-flto} to @samp{BOOT_CFLAGS}.
> +
>  @item @samp{bootstrap-debug}
>  Verifies that the compiler generates the same executable code, whether
>  or not it is asked to emit debug information.  To this end, this

> 
> 
> -- 
> Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
> You must be the change you wish to see in the world. -- Gandhi
> Be Free! -- http://FSFLA.org/   FSF Latin America board member
> Free Software Evangelist      Red Hat Brazil Compiler Engineer

Patch

for  contrib/ChangeLog
from  Alexandre Oliva  <aoliva@redhat.com>

	* compare-debug: Drop LTO sections.

for  config/ChangeLog
from  Alexandre Oliva  <aoliva@redhat.com>

	* bootstrap-lto.mk: New.

for  gcc/ChangeLog
from  Alexandre Oliva  <aoliva@redhat.com>

	* doc/install.texi: Document bootstrap-lto.

Index: contrib/compare-debug
===================================================================
--- contrib/compare-debug.orig	2010-06-16 10:30:16.000000000 -0300
+++ contrib/compare-debug	2010-06-16 10:44:18.000000000 -0300
@@ -100,9 +100,11 @@  else
   done
 
   # If we found .eh_frame in one but not the other, or if we could not
-  # find a command to tell, try to strip off the .eh_frame section
-  # from both.
-  if test "x$cmp1" != "x$cmp2" || test "x$cmd" = "x"; then
+  # find a command to tell, or if there are LTO sections, try to strip
+  # off the .eh_frame and LTO sections from both.
+  if test "x$cmp1" != "x$cmp2" || test "x$cmd" = "x" ||
+     $cmd --section-headers "$1.$suf1" | grep '.gnu.lto_' > /dev/null ||
+     $cmd --section-headers "$2.$suf2" | grep '.gnu.lto_' > /dev/null ; then
     suf3=$suf1.
     while test -f "$1.$suf3"; do
       suf3=$suf3.
@@ -115,21 +117,27 @@  else
 
     trap 'rm -f "$1.$suf1" "$2.$suf2" "$1.$suf3" "$2.$suf4"' 0 1 2 15
 
-    echo stripping off .eh_frame, then retrying >&2
+    echo stripping off .eh_frame and LTO sections, then retrying >&2
+
+    seclist=".eh_frame .rel.eh_frame .rela.eh_frame"
+    if test "x$cmd" != "x"; then
+      seclist="$seclist "`{ $cmd --section-headers "$1.$suf1"; $cmd --section-headers "$2.$suf2"; } | sed -n 's,.* \(\.gnu\.lto_[^ 	]*\).*,\1,p' | sort -u`
+    fi
+    rsopts=`for sec in $seclist; do echo " --remove-section $sec"; done`
 
     if (objcopy -v) 2>&1 | grep ' --remove-section' > /dev/null; then
-      objcopy --remove-section .eh_frame --remove-section .rel.eh_frame --remove-section .rela.eh_frame "$1.$suf1" "$1.$suf3"
+      objcopy $rsopts "$1.$suf1" "$1.$suf3"
       mv "$1.$suf3" "$1.$suf1"
 
-      objcopy --remove-section .eh_frame --remove-section .rel.eh_frame --remove-section .rela.eh_frame "$2.$suf2" "$2.$suf4"
+      objcopy $rsopts "$2.$suf2" "$2.$suf4"
       mv "$2.$suf4" "$2.$suf2"
     elif (strip --help) 2>&1 | grep ' --remove-section' > /dev/null; then
       cp "$1.$suf1" "$1.$suf3"
-      strip --remove-section .eh_frame --remove-section .rel.eh_frame --remove-section .rela.eh_frame "$1.$suf3"
+      strip $rsopts "$1.$suf3"
       mv "$1.$suf3" "$1.$suf1"
 
       cp "$2.$suf2" "$2.$suf4"
-      strip --remove-section .eh_frame --remove-section .rel.eh_frame --remove-section .rela.eh_frame "$2.$suf4"
+      strip $rsopts "$2.$suf4"
       mv "$2.$suf4" "$2.$suf2"
     else
       echo failed to strip off .eh_frame >&2
Index: config/bootstrap-lto.mk
===================================================================
--- /dev/null	1970-01-01 00:00:00.000000000 +0000
+++ config/bootstrap-lto.mk	2010-06-16 16:56:28.000000000 -0300
@@ -0,0 +1,8 @@ 
+# This option enables LTO for stage2 and stage3.  It requires lto to
+# be enabled for stage1 with --enable-stage1-languages.
+
+STAGE2_CFLAGS += -flto
+STAGE3_CFLAGS += -flto
+
+# Ada fails to build with LTO, turn it off for now.
+BOOT_ADAFLAGS += -fno-lto
Index: gcc/doc/install.texi
===================================================================
--- gcc/doc/install.texi.orig	2010-06-16 16:56:26.000000000 -0300
+++ gcc/doc/install.texi	2010-06-16 16:58:10.000000000 -0300
@@ -2150,6 +2150,11 @@  Removes any @option{-O}-started option f
 @item @samp{bootstrap-O3}
 Analogous to @code{bootstrap-O1}.
 
+@item @samp{bootstrap-lto}
+Enables Link-Time Optimization for host tools during bootstrapping.
+@samp{BUILD_CONFIG=bootstrap-lto} is equivalent to adding
+@option{-flto} to @samp{BOOT_CFLAGS}.
+
 @item @samp{bootstrap-debug}
 Verifies that the compiler generates the same executable code, whether
 or not it is asked to emit debug information.  To this end, this