Add flag to optionally ignore ELF interposition

Hi,
as disucssed some time ago, our assumption that every symbol of shared library can
be interposed at runtime is expensive and prevents a lot of useful optimizations,
including inlining or IPA propagation.

While this is useful feature, it is rather incommon to use it for bigger C++
projects, like firefox and at least clang seems to ignore the ELF interposition
rules and always inline/propagate. This patch adds flag to control the behaviour.
Symbols explicitly delcared WEAK are still considered as interposable.

Bootstrapped/regtested x86_64-linux, will commit it tomorrow if there are
no complains. (Feedback is welcome!)

Honza

	* doc/invoke.texi (-fsemantic-interposition): New flag.
	* common.opt (fsemantic-interposition): Use it.
	* varasm.c (decl_replaceable_p): Use it.

On Tue, 20 May 2014, Jan Hubicka wrote:

> Hi,
> as disucssed some time ago, our assumption that every symbol of shared library can
> be interposed at runtime is expensive and prevents a lot of useful optimizations,
> including inlining or IPA propagation.
> 
> While this is useful feature, it is rather incommon to use it for bigger C++
> projects, like firefox and at least clang seems to ignore the ELF interposition
> rules and always inline/propagate. This patch adds flag to control the behaviour.
> Symbols explicitly delcared WEAK are still considered as interposable.

I could see a use for an option listing the symbols that can be 
interposed.  (For example, glibc supports interposition of a limited 
number of malloc-related symbols, but if it were made to support building 
with LTO in future that it would make sense to be able to optimize calls 
to most other functions.)

> On Tue, 20 May 2014, Jan Hubicka wrote:
> 
> > Hi,
> > as disucssed some time ago, our assumption that every symbol of shared library can
> > be interposed at runtime is expensive and prevents a lot of useful optimizations,
> > including inlining or IPA propagation.
> > 
> > While this is useful feature, it is rather incommon to use it for bigger C++
> > projects, like firefox and at least clang seems to ignore the ELF interposition
> > rules and always inline/propagate. This patch adds flag to control the behaviour.
> > Symbols explicitly delcared WEAK are still considered as interposable.
> 
> I could see a use for an option listing the symbols that can be 
> interposed.  (For example, glibc supports interposition of a limited 
> number of malloc-related symbols, but if it were made to support building 
> with LTO in future that it would make sense to be able to optimize calls 
> to most other functions.)

I tought glibc uses handcoded local aliases for all its exported symbols except
for those where interposition is allowed.
But yes, this mechanism does kind of similar thing - in addition to enabling optimizations
ipa.c knows how to create local aliases for those symbols. Compiling:
$ cat ~/t.c
__attribute__
((noinline))
t()
{
  printf ("test\n");
}
q()
{
  t();
}
$ ./xgcc -B ./ -O2 ~/t.c -fno-semantic-interposition -S -fPIC

will get you:
        .set    t.localalias.0,t
        .globl  q
        .type   q, @function
q:
        xorl    %eax, %eax
        jmp     t.localalias.0

which is sort of what many libraries does by hand.

I wonder if this can't be best modeled as yet another visibility setting.  I.e. having visibility=default with
interposition assumed and visibility=default-no-semantic-interposition with visibility not allowed.
This can be both for -fvisibility flag and for attributes?

Honza
> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com

On Wed, 21 May 2014, Jan Hubicka wrote:

> I tought glibc uses handcoded local aliases for all its exported symbols except
> for those where interposition is allowed.

It does that for exported symbols.  But there are lots of non-exported __* 
functions used internally - and while some are declared hidden (or use the 
facility for hidden aliases), I suspect lots are not, so it's only the 
linker scripts that default-hide symbols that result in them not being 
interposable (and I'm guessing this effect of a linker script is too late 
for LTO to optimize based on that information).

> On Wed, 21 May 2014, Jan Hubicka wrote:
> 
> > I tought glibc uses handcoded local aliases for all its exported symbols except
> > for those where interposition is allowed.
> 
> It does that for exported symbols.  But there are lots of non-exported __* 
> functions used internally - and while some are declared hidden (or use the 
> facility for hidden aliases), I suspect lots are not, so it's only the 
> linker scripts that default-hide symbols that result in them not being 
> interposable (and I'm guessing this effect of a linker script is too late 
> for LTO to optimize based on that information).

This is actually good question, do you think you can check if such symbol get
LDPR_PREVAILING_DEF_IRONLY instead of LDPR_PREVAILING_DEF_IRONLY_EXP?
It would make sense for linker to do so and GCC will then privatize the symbols
(turn them to static).
If GNU LD or gold will declare them LDPR_PREVAILING_DEF_IRONLY_EXP, we probably could
fill in enhancement request.

The problem would be symbols that are used by non-LTO objects but still turned
hidden by the script - those would get LDPR_PREVAILING_DEF and since we do not
have LDPR_PREVAILING_DEF_EXP, we don't have the information that they are
hidden...
Introducing LDPR_PREVAILING_DEF_EXP is possible with another revision of plugin API,
but I am not sure it is worth the effort.

But yes, having way to tell this to non-LTO compiler would be nice. I wonder if
visibility attribute is right mechanism or the proposed -fsemantic-interposition=<list>

Honza
> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com

Em ter 20 maio 2014 22:04:32 você escreveu:
> Hi,
> as disucssed some time ago, our assumption that every symbol of shared
> library can be interposed at runtime is expensive and prevents a lot of
> useful optimizations, including inlining or IPA propagation.
> 
> While this is useful feature, it is rather incommon to use it for bigger C++
> projects, like firefox and at least clang seems to ignore the ELF
> interposition rules and always inline/propagate. This patch adds flag to
> control the behaviour. Symbols explicitly delcared WEAK are still
> considered as interposable.
> 
> Bootstrapped/regtested x86_64-linux, will commit it tomorrow if there are
> no complains. (Feedback is welcome!)

This patch seems to be controlling the behaviour of the compiler when seeing 
functions global that may or may not be interposed in the future. There 
doesn't seem to be a correct way to declare "no, this can't be interposed". Is 
there any plan of doing that?

In other words, if a function is not declared inline and GCC decides not to 
inline it anyway, then it and the linker will generate an indirect call 
through the PLT, with an associated (lazy) relocation. It's worse for 
variables: all non-hidden global variables must always be accessed indirectly 
via the GOT.

We have a couple of solutions for forbidding interposing (-Bsymbolic and 
protected visibility) and thus improve code generation, but they seem to 
always cause trouble in some places. The most common one for variables is copy 
relocation, whereas for functions it's the official function pointer address 
(will resolve to the PLT in the main executable).

Large projects would really appreciate being able to declare some symbols not 
interposable at all.

On Tue, May 20, 2014 at 1:04 PM, Jan Hubicka <hubicka@ucw.cz> wrote:
> Hi,
> as disucssed some time ago, our assumption that every symbol of shared library can
> be interposed at runtime is expensive and prevents a lot of useful optimizations,
> including inlining or IPA propagation.
>
> While this is useful feature, it is rather incommon to use it for bigger C++
> projects, like firefox and at least clang seems to ignore the ELF interposition
> rules and always inline/propagate. This patch adds flag to control the behaviour.
> Symbols explicitly delcared WEAK are still considered as interposable.
>
> Bootstrapped/regtested x86_64-linux, will commit it tomorrow if there are
> no complains. (Feedback is welcome!)


Did this patch ever go in?  I am running in a case where I would like
to use this option to improve some C++ library speed.

Thanks,
Andrew

>
> Honza
>
>         * doc/invoke.texi (-fsemantic-interposition): New flag.
>         * common.opt (fsemantic-interposition): Use it.
>         * varasm.c (decl_replaceable_p): Use it.
> Index: doc/invoke.texi
> ===================================================================
> --- doc/invoke.texi     (revision 210653)
> +++ doc/invoke.texi     (working copy)
> @@ -411,6 +411,7 @@ Objective-C and Objective-C++ Dialects}.
>  -fschedule-insns -fschedule-insns2 -fsection-anchors @gol
>  -fselective-scheduling -fselective-scheduling2 @gol
>  -fsel-sched-pipelining -fsel-sched-pipelining-outer-loops @gol
> +-fsemantic-interposition @gol
>  -fshrink-wrap -fsignaling-nans -fsingle-precision-constant @gol
>  -fsplit-ivs-in-unroller -fsplit-wide-types -fstack-protector @gol
>  -fstack-protector-all -fstack-protector-strong -fstrict-aliasing @gol
> @@ -7709,6 +7710,22 @@ This option has no effect unless one of
>  When pipelining loops during selective scheduling, also pipeline outer loops.
>  This option has no effect unless @option{-fsel-sched-pipelining} is turned on.
>
> +@item -fsemantic-interposition
> +@opindex fsemantic-interposition
> +Some object formats, like ELF, allow interposing of symbols by dynamic linker.
> +This means that for symbols exported from the DSO compiler can not perform
> +inter-procedural propagation, inlining and other optimizations in anticipation
> +that the function or variable in question may change. While this feature is
> +useful, for example, to rewrite memory allocation functions by a debugging
> +implementation, it is expensive in the terms of code quality.
> +With @option{-fno-semantic-inteposition} compiler assumest that if interposition
> +happens for functions the overwritting function will have
> +precisely same semantics (and side effects). Similarly if interposition happens
> +for variables, the constructor of the variable will be the same. The flag
> +has no effect for functions explicitly declared inline, where
> +interposition changing semantic is never allowed and for symbols explicitly
> +declared weak.
> +
>  @item -fshrink-wrap
>  @opindex fshrink-wrap
>  Emit function prologues only before parts of the function that need it,
> Index: common.opt
> ===================================================================
> --- common.opt  (revision 210653)
> +++ common.opt  (working copy)
> @@ -1854,6 +1854,10 @@ fsel-sched-reschedule-pipelined
>  Common Report Var(flag_sel_sched_reschedule_pipelined) Init(0) Optimization
>  Reschedule pipelined regions without pipelining
>
> +fsemantic-interposition
> +Common Report Var(flag_semantic_interposition) Init(1)
> +Allow interposing function (or variables) by ones with different semantics (or initializer) respectively by dynamic linker
> +
>  ; sched_stalled_insns means that insns can be moved prematurely from the queue
>  ; of stalled insns into the ready list.
>  fsched-stalled-insns
> Index: varasm.c
> ===================================================================
> --- varasm.c    (revision 210654)
> +++ varasm.c    (working copy)
> @@ -6870,6 +6870,9 @@ decl_replaceable_p (tree decl)
>    gcc_assert (DECL_P (decl));
>    if (!TREE_PUBLIC (decl) || DECL_COMDAT (decl))
>      return false;
> +  if (!flag_semantic_interposition
> +      && !DECL_WEAK (decl))
> +    return false;
>    return !decl_binds_to_current_def_p (decl);
>  }
>

> On Tue, May 20, 2014 at 1:04 PM, Jan Hubicka <hubicka@ucw.cz> wrote:
> > Hi,
> > as disucssed some time ago, our assumption that every symbol of shared library can
> > be interposed at runtime is expensive and prevents a lot of useful optimizations,
> > including inlining or IPA propagation.
> >
> > While this is useful feature, it is rather incommon to use it for bigger C++
> > projects, like firefox and at least clang seems to ignore the ELF interposition
> > rules and always inline/propagate. This patch adds flag to control the behaviour.
> > Symbols explicitly delcared WEAK are still considered as interposable.
> >
> > Bootstrapped/regtested x86_64-linux, will commit it tomorrow if there are
> > no complains. (Feedback is welcome!)
> 
> 
> Did this patch ever go in?  I am running in a case where I would like
> to use this option to improve some C++ library speed.

Not yet, with the feedback about possibly turning this into an attribute I was
thinking if perhaps fake visibility (default with no semantic interposition) would
do the job better.  But because i did not get any YES replies to that proposal,
I do not know of good name of those and it seems that mixing the concepts may be actually
a trouble, I will go with this patch and the attribute incrementally.

Honza
> 
> Thanks,
> Andrew
> 
> >
> > Honza
> >
> >         * doc/invoke.texi (-fsemantic-interposition): New flag.
> >         * common.opt (fsemantic-interposition): Use it.
> >         * varasm.c (decl_replaceable_p): Use it.
> > Index: doc/invoke.texi
> > ===================================================================
> > --- doc/invoke.texi     (revision 210653)
> > +++ doc/invoke.texi     (working copy)
> > @@ -411,6 +411,7 @@ Objective-C and Objective-C++ Dialects}.
> >  -fschedule-insns -fschedule-insns2 -fsection-anchors @gol
> >  -fselective-scheduling -fselective-scheduling2 @gol
> >  -fsel-sched-pipelining -fsel-sched-pipelining-outer-loops @gol
> > +-fsemantic-interposition @gol
> >  -fshrink-wrap -fsignaling-nans -fsingle-precision-constant @gol
> >  -fsplit-ivs-in-unroller -fsplit-wide-types -fstack-protector @gol
> >  -fstack-protector-all -fstack-protector-strong -fstrict-aliasing @gol
> > @@ -7709,6 +7710,22 @@ This option has no effect unless one of
> >  When pipelining loops during selective scheduling, also pipeline outer loops.
> >  This option has no effect unless @option{-fsel-sched-pipelining} is turned on.
> >
> > +@item -fsemantic-interposition
> > +@opindex fsemantic-interposition
> > +Some object formats, like ELF, allow interposing of symbols by dynamic linker.
> > +This means that for symbols exported from the DSO compiler can not perform
> > +inter-procedural propagation, inlining and other optimizations in anticipation
> > +that the function or variable in question may change. While this feature is
> > +useful, for example, to rewrite memory allocation functions by a debugging
> > +implementation, it is expensive in the terms of code quality.
> > +With @option{-fno-semantic-inteposition} compiler assumest that if interposition
> > +happens for functions the overwritting function will have
> > +precisely same semantics (and side effects). Similarly if interposition happens
> > +for variables, the constructor of the variable will be the same. The flag
> > +has no effect for functions explicitly declared inline, where
> > +interposition changing semantic is never allowed and for symbols explicitly
> > +declared weak.
> > +
> >  @item -fshrink-wrap
> >  @opindex fshrink-wrap
> >  Emit function prologues only before parts of the function that need it,
> > Index: common.opt
> > ===================================================================
> > --- common.opt  (revision 210653)
> > +++ common.opt  (working copy)
> > @@ -1854,6 +1854,10 @@ fsel-sched-reschedule-pipelined
> >  Common Report Var(flag_sel_sched_reschedule_pipelined) Init(0) Optimization
> >  Reschedule pipelined regions without pipelining
> >
> > +fsemantic-interposition
> > +Common Report Var(flag_semantic_interposition) Init(1)
> > +Allow interposing function (or variables) by ones with different semantics (or initializer) respectively by dynamic linker
> > +
> >  ; sched_stalled_insns means that insns can be moved prematurely from the queue
> >  ; of stalled insns into the ready list.
> >  fsched-stalled-insns
> > Index: varasm.c
> > ===================================================================
> > --- varasm.c    (revision 210654)
> > +++ varasm.c    (working copy)
> > @@ -6870,6 +6870,9 @@ decl_replaceable_p (tree decl)
> >    gcc_assert (DECL_P (decl));
> >    if (!TREE_PUBLIC (decl) || DECL_COMDAT (decl))
> >      return false;
> > +  if (!flag_semantic_interposition
> > +      && !DECL_WEAK (decl))
> > +    return false;
> >    return !decl_binds_to_current_def_p (decl);
> >  }
> >