-fwhopr and -flto options reorg

> On Fri, Nov 12, 2010 at 09:15, Richard Guenther <rguenther@suse.de> wrote:
> 
> >> Where we have internals document for this kind of things?
> >
> > We don't.  Just keep it in invoke.texi.
> 
> I was thinking of a new file doc/lto.texi linked from gccint.texi

Hi,
I guess I do not care much where the WPA option is documented. It is clearly
mentioned as internal option, so it is OK. We still might drop the "whole
program" from documentation and just stick with random 3 letters.
This is what I usually do with WHOPR and WPA, just not expand the acronym.

I overviewed the docs and noticed that I left quite bit of -fwhopr references
in it.  Also I think the -flto section is long and needs to be reordered to
give most interesting info first.  For this reason I moved info about
parallelizm early and also added info about whole program assumption being
needed for effecitve optimization.

This is what I have now.  It would be great if somone with better Enlish
reviewed the LTO section and tried to make to guide users to correct
configuration more effectively. I.e. make them to realize basic thinks.
I think priority order could be something like this.

 1) that LTO can do parallel linking, how -flto=n and -flto=jobserv works
    and that '+' rule is needed
 2) that plugin or -fwhole-program is needed to get effective code
 3) that .a archives will not get LTOed unless you have linker support
    (it now works with GNU LD, right?)
    We probaby can also tell a bit on internal visibility and shared libraries
 4) rest of LTO use and need for compatible command line options.

Honza

On Fri, Nov 12, 2010 at 12:18, Jan Hubicka <hubicka@ucw.cz> wrote:

> This is what I have now.  It would be great if somone with better Enlish
> reviewed the LTO section and tried to make to guide users to correct
> configuration more effectively. I.e. make them to realize basic thinks.

I'll incorporate these changes into the patch I've got that moves the
internal stuff into lto.texi.


Thanks.  Diego.

See my comments inlined below.

On Fri, Nov 12, 2010 at 9:18 AM, Jan Hubicka <hubicka@ucw.cz> wrote:
>> On Fri, Nov 12, 2010 at 09:15, Richard Guenther <rguenther@suse.de> wrote:
>>
>> >> Where we have internals document for this kind of things?
>> >
>> > We don't.  Just keep it in invoke.texi.
>>
>> I was thinking of a new file doc/lto.texi linked from gccint.texi
>
> Hi,
> I guess I do not care much where the WPA option is documented. It is clearly
> mentioned as internal option, so it is OK. We still might drop the "whole
> program" from documentation and just stick with random 3 letters.
> This is what I usually do with WHOPR and WPA, just not expand the acronym.
>
> I overviewed the docs and noticed that I left quite bit of -fwhopr references
> in it.  Also I think the -flto section is long and needs to be reordered to
> give most interesting info first.  For this reason I moved info about
> parallelizm early and also added info about whole program assumption being
> needed for effecitve optimization.
>
> This is what I have now.  It would be great if somone with better Enlish
> reviewed the LTO section and tried to make to guide users to correct
> configuration more effectively. I.e. make them to realize basic thinks.
> I think priority order could be something like this.
>
>  1) that LTO can do parallel linking, how -flto=n and -flto=jobserv works
>    and that '+' rule is needed
>  2) that plugin or -fwhole-program is needed to get effective code
>  3) that .a archives will not get LTOed unless you have linker support
>    (it now works with GNU LD, right?)
>    We probaby can also tell a bit on internal visibility and shared libraries
>  4) rest of LTO use and need for compatible command line options.
>
> Honza
>
> Index: invoke.texi
> ===================================================================
> --- invoke.texi (revision 166652)
> +++ invoke.texi (working copy)
> @@ -399,7 +399,7 @@ Objective-C and Objective-C++ Dialects}.
>  -funit-at-a-time -funroll-all-loops -funroll-loops @gol
>  -funsafe-loop-optimizations -funsafe-math-optimizations -funswitch-loops @gol
>  -fvariable-expansion-in-unroller -fvect-cost-model -fvpt -fweb @gol
> --fwhole-program -fwhopr[=@var{n}] -fwpa -fuse-linker-plugin @gol
> +-fwhole-program -fwpa -fuse-linker-plugin @gol
>  --param @var{name}=@var{value}
>  -O  -O0  -O1  -O2  -O3  -Os -Ofast}
>
> @@ -7455,6 +7455,22 @@ file.  When the object files are linked
>  bodies are read from these ELF sections and instantiated as if they
>  had been part of the same translation unit.
>
> +If you specify the optional @var{n} the link stage is executed in
> +parallel using @var{n} parallel jobs by utilizing an installed
> +@command{make} program.

link stage or backend optimization+codegen ?

What is the default for n?

> The environment variable @env{MAKE} may be
> +used to override the program used.  See also option
> +@option{--param lto-partitions=@var{value}}). GCC by default split
> +program to fixed number of partitions. This parameter can be used
> +to increase the parallelism and also to reduce memory usage in the
> +later stage of link time optimization process.

default partition ?

> +
> +You can also specify @option{-flto=jobserver} to use GNU make's
> +job server mode to determine the number of parallel jobs. This
> +is useful when the Makefile calling GCC is already parallel.
> +The parent Makefile will need a @samp{+} prepended to the command recipe
> +for this to work. This will likely only work if @env{MAKE} is
> +GNU make.
> +
>  To use the link-timer optimizer, @option{-flto} needs to be specified at
>  compile time and during the final link.  For example,
>
> @@ -7489,6 +7505,14 @@ The only important thing to keep in mind
>  optimizations the @option{-flto} flag needs to be passed to both the
>  compile and the link commands.
>
> +To make while program optimization effective, it is neccesary to make certain
> +level of whole program assumptions.  Compiler needs to know what functions and
> +variables can be accessed by libraries and runtime outside of the link time
> +optimized unit.  When supported by linker, the linker plugin (see
> +@option{-fuse-linker-plugin}) pass to the compiler information about used and
> +externally visible symbols. When linker plugin is not availble,

availble -->available.

> +@option{-fwhole-program} should be used to get goo code quality.

goo --> good.

> +
>  Note that when a file is compiled with @option{-flto}, the generated
>  object file will be larger than a regular object file because it will
>  contain GIMPLE bytecodes and the usual final code.  This means that
> @@ -7603,23 +7627,11 @@ Link time optimization does not play wel
>  information.  Combining @option{-flto} with
>  @option{-g} is experimental.
>
> -If you specify the optional @var{n} the link stage is executed in
> -parallel using @var{n} parallel jobs by utilizing an installed
> -@command{make} program.  The environment variable @env{MAKE} may be
> -used to override the program used.
> -
> -You can also specify @option{-fwhopr=jobserver} to use GNU make's
> -job server mode to determine the number of parallel jobs. This
> -is useful when the Makefile calling GCC is already parallel.
> -The parent Makefile will need a @samp{+} prepended to the command recipe
> -for this to work. This will likely only work if @env{MAKE} is
> -GNU make.
> -
>  This option is disabled by default.
>
>  @item -flto-partition=@var{alg}
>  @opindex flto-partition
> -Specify partitioning algorithm used by @option{-fwhopr} mode.  The value is
> +Specify partitioning algorithm used by link time optimizer.  The value is
>  either @code{1to1} to specify partitioning corresponding to source files
>  or @code{balanced} to specify partitioning into, if possible, equally sized
>  chunks.  Specifying @code{none} as an algorithm disables partitioning

lto1?? This is too developer centric. It is still not clear to the
user what it means by 'corresponding source files' -- any examples?
Also good to reference the partition parameter that is related here.


> @@ -7629,22 +7641,21 @@ The default value is @code{balanced}.
>  @item -fwpa
>  @opindex fwpa
>  This is an internal option used by GCC when compiling with
> -@option{-fwhopr}.  You should never need to use it.
> +@option{-flto} with partitioning enabled.  You should never need to use it.
>
> -This option runs the link-time optimizer in the whole-program-analysis
> -(WPA) mode, which reads in summary information from all inputs and
> -performs a whole-program analysis based on summary information only.
> -It generates object files for subsequent runs of the link-time
> -optimizer where individual object files are optimized using both
> -summary information from the WPA mode and the actual function bodies.
> -It then drives the LTRANS phase.
> +This option runs the serial part of link-time optimizer performing the
> +inter-procedural propagation (WPA mode). Compiler reads in summary information
> +from all inputs and performs an analysis based on summary information only.  It
> +generates object files for subsequent runs of the link-time optimizer where
> +individual object files are optimized using both summary information from the
> +WPA mode and the actual function bodies.  It then drives the LTRANS phase.
>

Would it be better to replace LTRANS to something friendlier to non
compiler savvies ?


Thanks,

David
>  Disabled by default.
>
>  @item -fltrans
>  @opindex fltrans
>  This is an internal option used by GCC when compiling with
> -@option{-fwhopr}.  You should never need to use it.
> +@option{-flto} with partitioning enabled.  You should never need to use it.
>
>  This option runs the link-time optimizer in the local-transformation (LTRANS)
>  mode, which reads in output from a previous run of the LTO in WPA mode.
> @@ -7655,7 +7666,7 @@ Disabled by default.
>  @item -fltrans-output-list=@var{file}
>  @opindex fltrans-output-list
>  This is an internal option used by GCC when compiling with
> -@option{-fwhopr}.  You should never need to use it.
> +@option{-lto}.  You should never need to use it.
>
>  This option specifies a file to which the names of LTRANS output files are
>  written.  This option is only meaningful in conjunction with @option{-fwpa}.
> @@ -7665,7 +7676,7 @@ Disabled by default.
>  @item -flto-compression-level=@var{n}
>  This option specifies the level of compression used for intermediate
>  language written to LTO object files, and is only meaningful in
> -conjunction with LTO mode (@option{-fwhopr}, @option{-flto}).  Valid
> +conjunction with LTO mode (@option{-flto}).  Valid
>  values are 0 (no compression) to 9 (maximum compression).  Values
>  outside this range are clamped to either 0 or 9.  If the option is not
>  given, a default balanced compression setting is used.
> @@ -7674,7 +7685,7 @@ given, a default balanced compression se
>  Prints a report with internal details on the workings of the link-time
>  optimizer.  The contents of this report vary from version to version,
>  it is meant to be useful to GCC developers when processing object
> -files in LTO mode (via @option{-fwhopr} or @option{-flto}).
> +files in LTO mode (via @option{-flto}).
>
>  Disabled by default.
>
>

On Fri, Nov 12, 2010 at 12:43, Xinliang David Li <davidxl@google.com> wrote:

> Would it be better to replace LTRANS to something friendlier to non
> compiler savvies ?

This will all go in the internals manual, so it won't be an issue.


Diego.

> > +If you specify the optional @var{n} the link stage is executed in
> > +parallel using @var{n} parallel jobs by utilizing an installed
> > +@command{make} program.
> 
> link stage or backend optimization+codegen ?

"Optimization and code generation at link time" is probably most precise here.
> 
> What is the default for n?

1
> 
> > The environment variable @env{MAKE} may be
> > +used to override the program used.  See also option
> > +@option{--param lto-partitions=@var{value}}). GCC by default split
> > +program to fixed number of partitions. This parameter can be used
> > +to increase the parallelism and also to reduce memory usage in the
> > +later stage of link time optimization process.
> 
> default partition ?

This is in lto-partitions param docs. I am not sure we want to duplicate it
as we are not really good on keeping all the numbers up to date when defaults
change.
> >  @item -flto-partition=@var{alg}
> >  @opindex flto-partition
> > -Specify partitioning algorithm used by @option{-fwhopr} mode.  The value is
> > +Specify partitioning algorithm used by link time optimizer.  The value is
> >  either @code{1to1} to specify partitioning corresponding to source files
> >  or @code{balanced} to specify partitioning into, if possible, equally sized
> >  chunks.  Specifying @code{none} as an algorithm disables partitioning
> 
> lto1?? This is too developer centric. It is still not clear to the

Well, this is option intended for debugging WHOPR. 1to1 was name of partitioning alg
in the original WHOPR sources.  We can rename it with little of testsuite updating paing.
> user what it means by 'corresponding source files' -- any examples?
When you have a.c and b.c compiled together then the program will be partitioned
again into two object file, first having functions&vars corresponding to those in a.c,
second having functions&vars corresponding to those in b.c

The primary use of this is to write testcases for bugs that needs particular partitioning
to reproduce.

> Also good to reference the partition parameter that is related here.
> 
> 
> > @@ -7629,22 +7641,21 @@ The default value is @code{balanced}.
> >  @item -fwpa
> >  @opindex fwpa
> >  This is an internal option used by GCC when compiling with
> > -@option{-fwhopr}.  You should never need to use it.
> > +@option{-flto} with partitioning enabled.  You should never need to use it.
> >
> > -This option runs the link-time optimizer in the whole-program-analysis
> > -(WPA) mode, which reads in summary information from all inputs and
> > -performs a whole-program analysis based on summary information only.
> > -It generates object files for subsequent runs of the link-time
> > -optimizer where individual object files are optimized using both
> > -summary information from the WPA mode and the actual function bodies.
> > -It then drives the LTRANS phase.
> > +This option runs the serial part of link-time optimizer performing the
> > +inter-procedural propagation (WPA mode). Compiler reads in summary information
> > +from all inputs and performs an analysis based on summary information only.  It
> > +generates object files for subsequent runs of the link-time optimizer where
> > +individual object files are optimized using both summary information from the
> > +WPA mode and the actual function bodies.  It then drives the LTRANS phase.
> >
> 
> Would it be better to replace LTRANS to something friendlier to non
> compiler savvies ?

It is internal option.  I hope diego will nuke those from invoke.texi and we will be all
happy.

Honza

Hello Jan,

some nits in addition to David's:

* Jan Hubicka wrote on Fri, Nov 12, 2010 at 06:18:52PM CET:
> --- invoke.texi	(revision 166652)
> +++ invoke.texi	(working copy)

> @@ -7455,6 +7455,22 @@ file.  When the object files are linked 
>  bodies are read from these ELF sections and instantiated as if they
>  had been part of the same translation unit.
>  
> +If you specify the optional @var{n} the link stage is executed in
> +parallel using @var{n} parallel jobs by utilizing an installed
> +@command{make} program.  The environment variable @env{MAKE} may be
> +used to override the program used.  See also option
> +@option{--param lto-partitions=@var{value}}). GCC by default split
> +program to fixed number of partitions.

splits the.  But how about:

By default, GCC splits the program into a fixed number of ...
partitions.

> This parameter can be used
> +to increase the parallelism and also to reduce memory usage in the
> +later stage of link time optimization process.

stages?

> +You can also specify @option{-flto=jobserver} to use GNU make's 
> +job server mode to determine the number of parallel jobs. This 

Two spaces after period; more instances below.

> +is useful when the Makefile calling GCC is already parallel.

s/parallel/run in &/  ?

> +The parent Makefile will need a @samp{+} prepended to the command recipe
> +for this to work. This will likely only work if @env{MAKE} is 
> +GNU make.
> +
>  To use the link-timer optimizer, @option{-flto} needs to be specified at
>  compile time and during the final link.  For example,
>  
> @@ -7489,6 +7505,14 @@ The only important thing to keep in mind
>  optimizations the @option{-flto} flag needs to be passed to both the
>  compile and the link commands.
>  
> +To make while program optimization effective, it is neccesary to make certain
> +level of whole program assumptions.

I don't understand this sentence.  s/while /whole-/  ?
'certain level' sounds like there's a compiler knob to turn.
How about  s/level of //  ?

>  Compiler needs to know what functions and

The compiler

> +variables can be accessed by libraries and runtime outside of the link time
> +optimized unit.  When supported by linker, the linker plugin (see

the linker

> +@option{-fuse-linker-plugin}) pass to the compiler information about used and

passes information about ... and ... to the compiler

> +externally visible symbols. When linker plugin is not availble,
> +@option{-fwhole-program} should be used to get goo code quality.

s/goo/&d/

>  Note that when a file is compiled with @option{-flto}, the generated
>  object file will be larger than a regular object file because it will
>  contain GIMPLE bytecodes and the usual final code.  This means that
> @@ -7603,23 +7627,11 @@ Link time optimization does not play wel
>  information.  Combining @option{-flto} with
>  @option{-g} is experimental.
>  
> -If you specify the optional @var{n} the link stage is executed in
> -parallel using @var{n} parallel jobs by utilizing an installed
> -@command{make} program.  The environment variable @env{MAKE} may be
> -used to override the program used.
> -
> -You can also specify @option{-fwhopr=jobserver} to use GNU make's 
> -job server mode to determine the number of parallel jobs. This 
> -is useful when the Makefile calling GCC is already parallel.
> -The parent Makefile will need a @samp{+} prepended to the command recipe
> -for this to work. This will likely only work if @env{MAKE} is 
> -GNU make.
> -
>  This option is disabled by default.
>  
>  @item -flto-partition=@var{alg}
>  @opindex flto-partition
> -Specify partitioning algorithm used by @option{-fwhopr} mode.  The value is
> +Specify partitioning algorithm used by link time optimizer.  The value is

the link time optimizer

>  either @code{1to1} to specify partitioning corresponding to source files
>  or @code{balanced} to specify partitioning into, if possible, equally sized
>  chunks.  Specifying @code{none} as an algorithm disables partitioning
> @@ -7629,22 +7641,21 @@ The default value is @code{balanced}.
>  @item -fwpa
>  @opindex fwpa
>  This is an internal option used by GCC when compiling with
> -@option{-fwhopr}.  You should never need to use it.
> +@option{-flto} with partitioning enabled.  You should never need to use it.
>  
> -This option runs the link-time optimizer in the whole-program-analysis
> -(WPA) mode, which reads in summary information from all inputs and
> -performs a whole-program analysis based on summary information only.
> -It generates object files for subsequent runs of the link-time
> -optimizer where individual object files are optimized using both
> -summary information from the WPA mode and the actual function bodies.
> -It then drives the LTRANS phase.
> +This option runs the serial part of link-time optimizer performing the

the link-time optimizer.  Please use one of 'link time' or 'link-time'
only, throughout.

> +inter-procedural propagation (WPA mode). Compiler reads in summary information

The compiler

> +from all inputs and performs an analysis based on summary information only.  It
> +generates object files for subsequent runs of the link-time optimizer where
> +individual object files are optimized using both summary information from the
> +WPA mode and the actual function bodies.  It then drives the LTRANS phase.
>  
>  Disabled by default.
>  
>  @item -fltrans
>  @opindex fltrans
>  This is an internal option used by GCC when compiling with
> -@option{-fwhopr}.  You should never need to use it.
> +@option{-flto} with partitioning enabled.  You should never need to use it.
>  
>  This option runs the link-time optimizer in the local-transformation (LTRANS)
>  mode, which reads in output from a previous run of the LTO in WPA mode.
> @@ -7655,7 +7666,7 @@ Disabled by default.
>  @item -fltrans-output-list=@var{file}
>  @opindex fltrans-output-list
>  This is an internal option used by GCC when compiling with
> -@option{-fwhopr}.  You should never need to use it.
> +@option{-lto}.  You should never need to use it.
>  
>  This option specifies a file to which the names of LTRANS output files are
>  written.  This option is only meaningful in conjunction with @option{-fwpa}.
> @@ -7665,7 +7676,7 @@ Disabled by default.
>  @item -flto-compression-level=@var{n}
>  This option specifies the level of compression used for intermediate
>  language written to LTO object files, and is only meaningful in
> -conjunction with LTO mode (@option{-fwhopr}, @option{-flto}).  Valid
> +conjunction with LTO mode (@option{-flto}).  Valid
>  values are 0 (no compression) to 9 (maximum compression).  Values
>  outside this range are clamped to either 0 or 9.  If the option is not
>  given, a default balanced compression setting is used.
> @@ -7674,7 +7685,7 @@ given, a default balanced compression se
>  Prints a report with internal details on the workings of the link-time
>  optimizer.  The contents of this report vary from version to version,
>  it is meant to be useful to GCC developers when processing object
> -files in LTO mode (via @option{-fwhopr} or @option{-flto}).
> +files in LTO mode (via @option{-flto}).
>  
>  Disabled by default.
>  

Thanks,
Ralf