Add GLIBC_PTHREAD_ELISION_ENABLE tunable

Some related discussions below:

https://sourceware.org/ml/libc-alpha/2015-09/msg00128.html
https://sourceware.org/ml/libc-alpha/2015-09/msg00065.html
https://sourceware.org/ml/libc-alpha/2015-09/msg00130.html

---8<---
This tunable controls the usage of transactional lock elision
within the nptl.  This also defines a new manual section to
describe the tunable, and any future tunables.

powerpc now disables TLE by default, and uses the tunable
to optionally enable the feature if desired.

2015-09-09  Paul E. Murphy  <murphyp@linux.vnet.ibm.com>

	* manual/Makefile (chapters): Add tunables.texi.
	* manual/into.texi: Add reference to Tunables.
	* manual/probes.texi: Enforce tunables.texi chapter location.
	* manual/tunables.texi: New file.
	* sysdeps/unix/sysv/linux/powerpc/elision-conf.c (elision-init):
	Add tunable to enable elision.
	* sysdeps/unix/sysv/linux/390/elision-conf.c (elision-init):
	Add tunable to disable Likewise.
	* sysdeps/unix/sysv/linux/x86/elision-conf.c (elision-init):
	Likewise.
---
 manual/Makefile                                |    3 +-
 manual/intro.texi                              |    3 ++
 manual/probes.texi                             |    2 +-
 manual/tunables.texi                           |   36 ++++++++++++++++++++++++
 sysdeps/unix/sysv/linux/powerpc/elision-conf.c |    6 +++-
 sysdeps/unix/sysv/linux/s390/elision-conf.c    |    5 +++
 sysdeps/unix/sysv/linux/x86/elision-conf.c     |    5 +++
 7 files changed, 57 insertions(+), 3 deletions(-)
 create mode 100644 manual/tunables.texi

"Paul E. Murphy" <murphyp@linux.vnet.ibm.com> writes:

>    int elision_available = (GLRO (dl_hwcap) & HWCAP_S390_TE) ? 1 : 0;
>  
> +  /* Check environment to see if we should disable LE.  */
> +  for (; *environ != NULL; environ++)
> +    if (strcmp (*environ, "GLIBC_PTHREAD_ELISION_ENABLE=no") == 0)
> +      __pthread_force_elision = 0;
> +
>    __pthread_force_elision = __libc_enable_secure ? 0 : elision_available;

This does not make sense, any value assigned to
__pthread_force_elision in the loop will be overwritten without
reading after the loop.

On 11 Sep 2015 14:57, Paul E. Murphy wrote:
> +  /* Check environment to see if we should enable LE.  */
> +  for (; *environ != NULL; environ++)
> +    if (strcmp (*environ, "GLIBC_PTHREAD_ELISION_ENABLE=yes") == 0)
> +      __pthread_force_elision = __libc_enable_secure ? 0 : elision_available;

there should be a new internal header for checking the values of tunables so
that we don't end up with lots of duplication (like this patch already does).
for the functions that cannot call getenv, there should be a static inline
function that they can call (which i imagine would look like this).
-mike

On 09/16/2015 11:37 PM, Mike Frysinger wrote:
> On 11 Sep 2015 14:57, Paul E. Murphy wrote:
>> +  /* Check environment to see if we should enable LE.  */
>> +  for (; *environ != NULL; environ++)
>> +    if (strcmp (*environ, "GLIBC_PTHREAD_ELISION_ENABLE=yes") == 0)
>> +      __pthread_force_elision = __libc_enable_secure ? 0 : elision_available;
> 
> there should be a new internal header for checking the values of tunables so
> that we don't end up with lots of duplication (like this patch already does).
> for the functions that cannot call getenv, there should be a static inline
> function that they can call (which i imagine would look like this).

I don't disagree with your idea. However, such work impinges on the tunable
work by Siddhesh. My intention is the simplest scaffolding while we wait out
the tunables framework.

This patch attempts to answer the following outstanding community questions:

1. Should we be using a single envvar vs multiple?

2. Should we be using GLIBC_ vs. GNU_ prefix?

3. Does the manual page sufficiently document tunables?

AFAIK, we still lack consensus and input on 1 and 2.

BR,
Paul

There should be no explicit code using new environment variables at all.
It should all go through the tunables framework so policy and knowledge
about use of environment variables is centralized.

On Thu, 2015-09-17 at 10:51 -0700, Roland McGrath wrote:
> There should be no explicit code using new environment variables at
> all.
> It should all go through the tunables framework so policy and
> knowledge
> about use of environment variables is centralized.

I pushed some code to the siddhesh/tunables branch last week that reads
just one magic environment variable and parses the string to initialize
all tunables at once.  This gives a definite path to deprecate the
existing arbitrary environment variables in future as well since it
clearly separates those bits from the core framework.  So now y'all
have both approaches to try out while I cool my heels off for a bit - I
hope to be back hacking soon.

We have talked about both approaches for a bit in different threads and
I continue to have a slight preference for the single environment
variable for the sheer ease of initialization and the possibility of
doing it really early once I figure out how to get the code included
into the dynamic linker without dropping it into the elf directory. 
 All other differences seem cosmetic to me.

Siddhesh

On Thu, 2015-09-17 at 10:51 -0700, Roland McGrath wrote:
> There should be no explicit code using new environment variables at all.
> It should all go through the tunables framework so policy and knowledge
> about use of environment variables is centralized.
> 
So how do we actually move that along?

Because it is not moving?

> On Thu, 2015-09-17 at 10:51 -0700, Roland McGrath wrote:
> > There should be no explicit code using new environment variables at all.
> > It should all go through the tunables framework so policy and knowledge
> > about use of environment variables is centralized.
> > 
> So how do we actually move that along?
> 
> Because it is not moving? 

What we agreed at the Cauldron was that the first step should be the
internal API for tunables, with a stub implementation that does not
implement new external mechanisms like new environment variables.  With
that in place, we can experiment concretely with specific "back ends" and
start achieving consensus.  But people seem to be be doing things in a
different order than what was agreed.

On Thu, 2015-09-17 at 11:19 -0700, Roland McGrath wrote:
> What we agreed at the Cauldron was that the first step should be the
> internal API for tunables, with a stub implementation that does not
> implement new external mechanisms like new environment variables. 
>  With
> that in place, we can experiment concretely with specific "back ends"
> and
> start achieving consensus.  But people seem to be be doing things in
> a
> different order than what was agreed.

I may have misunderstood what you meant; my understanding of your
objection was that you did not want an envvar for each tunable and
wanted a single envvar or config file, which seems to be different from
what you've mentioned now, which seems to be to not provide any way to
externally specify tunables and only have the internal scaffolding in
place.  I believe Carlos may have misunderstood as well since we
discussed this in depth later and seemed to be talking the same thing.

If what I understand now is correct, the first cut should be easier. 
 I'll try to get a proper patch up for review after a week or so.

Siddhesh

> I may have misunderstood what you meant; my understanding of your
> objection was that you did not want an envvar for each tunable and
> wanted a single envvar or config file, which seems to be different from
> what you've mentioned now, which seems to be to not provide any way to
> externally specify tunables and only have the internal scaffolding in
> place.  I believe Carlos may have misunderstood as well since we
> discussed this in depth later and seemed to be talking the same thing.
> 
> If what I understand now is correct, the first cut should be easier. 
>  I'll try to get a proper patch up for review after a week or so.

The main point is to separate the internal API infrastructure that will be
used pervasively across the source tree and set the terms for how we
describe a tunable (i.e. naming, types, et al) from the "back end"
implementation of any particular scheme for getting values into the system.
What we agreed on was that we could relatively easily reach consensus on
this internal API.  We explicitly deferred specific discussion on what I'm
calling "back end" issues until after the API is in place.

After the BoF was over, you, Carlos, and I then went on to start to
minimally discuss back end issues despite the stated plan to differ that
discussion.  In that context, I briefly reiterated my general concern about
all environment variables (which I will continue to argue when we actually
get to that).  The notion of a single environment variable was an example
of a way to mitigate some of those concerns if I were to capitulate on the
firm stance about environment variables.  It was not a proposal, except to
suggest that you might use it as a straw-man starting proposal when we
began that discussion after landing the internal API.  (We also discussed
much more forward-looking ideas like a shared memory block manipulated by
an external utility.  All of that was in the realm of ideas we might
consider in the future, once we have established the context of the
internal API for tunables.)

I stand by my core positions.  We have not recently discussed my positions
about back end issues.  I don't really want to get into any of that in
earnest until we have reached consensus on the internal API and it is close
to being landed.  Once that's done, then people can go off and implement
working prototypes of particular ideas (be they environment variables or
whatever else) and we can play with those as part of reaching consensus on
what we actually want to have in the code.  (Multiple such implementations
with configure switches to choose among them might be something we choose
to do.)

Roland McGrath <roland@hack.frob.com> writes:

Roland,
>
> The main point is to separate the internal API infrastructure that will be
> used pervasively across the source tree and set the terms for how we
> describe a tunable (i.e. naming, types, et al) from the "back end"
> implementation of any particular scheme for getting values into the
> system.

I don't see how such a complicated procedure is needed to add simple
tunables. It seems just an elaborate way to say "I can't make up my mind"?

> What we agreed on was that we could relatively easily reach consensus on
> this internal API.  We explicitly deferred specific discussion on what I'm
> calling "back end" issues until after the API is in place.

Please make a decision. This whole train wreck is going on for far too
long now. And tunables are badly needed.

An "internal API" doesn't help any user. If some plugin scheme is done
the first usable version of it will be just the defacto implementation.

I thought we had a consensus earlier when I was told to implement
environment variables with opt-in config file. Is that now already
forgotten?

> I stand by my core positions.

Nobody can figure out what your core positions are, they don't
actually describe any way to solve the problem.

-Andi

> I don't see how such a complicated procedure is needed to add simple
> tunables. It seems just an elaborate way to say "I can't make up my mind"?

It's called modularity.

> Please make a decision. This whole train wreck is going on for far too
> long now. And tunables are badly needed.

Decision here are made by consensus.  It often takes a while.  Technical
preparations that make it easier for more people to deliberate effectively
can help achieve consensus.

> An "internal API" doesn't help any user. 

It is the only reasonable way forward, since one-off hacks are not feasible
to maintain or audit.

> I thought we had a consensus earlier when I was told to implement
> environment variables with opt-in config file. Is that now already
> forgotten?

That describes a back end implementation that I would not object to.  I
think it likely that when we do reach consensus on something, it will be
something like that.  There has never been consensus on the notion of
implementing any one-off hacks without a coherent internal API
infrastructure for all tunables.

> > I stand by my core positions.
> 
> Nobody can figure out what your core positions are, they don't
> actually describe any way to solve the problem.

I'm not responding to this ad hominem statement that bears no relation to
any technical subject (and is not accurate in its characterization anyway).

On 09/17/2015 01:59 PM, Siddhesh Poyarekar wrote:
> On Thu, 2015-09-17 at 10:51 -0700, Roland McGrath wrote:
>> There should be no explicit code using new environment variables at
>> all.
>> It should all go through the tunables framework so policy and
>> knowledge
>> about use of environment variables is centralized.
> 
> I pushed some code to the siddhesh/tunables branch last week that reads
> just one magic environment variable and parses the string to initialize
> all tunables at once.  This gives a definite path to deprecate the
> existing arbitrary environment variables in future as well since it
> clearly separates those bits from the core framework.  So now y'all
> have both approaches to try out while I cool my heels off for a bit - I
> hope to be back hacking soon.
> 
> We have talked about both approaches for a bit in different threads and
> I continue to have a slight preference for the single environment
> variable for the sheer ease of initialization and the possibility of
> doing it really early once I figure out how to get the code included
> into the dynamic linker without dropping it into the elf directory. 
>  All other differences seem cosmetic to me.

Given that we can change our mind on a release-by-release basis I'm also
happy with just one env var... given that you implemented it :-)

c.

On 17 Sep 2015 13:32, Andi Kleen wrote:
> Roland McGrath <roland@hack.frob.com> writes:
> > The main point is to separate the internal API infrastructure that will be
> > used pervasively across the source tree and set the terms for how we
> > describe a tunable (i.e. naming, types, et al) from the "back end"
> > implementation of any particular scheme for getting values into the
> > system.
> 
> I don't see how such a complicated procedure is needed to add simple
> tunables. It seems just an elaborate way to say "I can't make up my mind"?

glibc is known for its stability and strong guarantees of not breaking
ABIs.  describing something as simple seems to brush that off -- we don't
want to introduce knobs hastily that we are going to regret and be stuck
with forever.  perhaps this particular discussion has been dragging on a
bit, but glibc isn't really the place for experimentation (which is then
released directly to users) and for seeing what sticks purely by throwing
things against the wall.  i'd rather we be overly conservative.
-mike

On Fri, Sep 18, 2015 at 02:32:51AM -0400, Mike Frysinger wrote:
> On 17 Sep 2015 13:32, Andi Kleen wrote:
> > Roland McGrath <roland@hack.frob.com> writes:
> > > The main point is to separate the internal API infrastructure that will be
> > > used pervasively across the source tree and set the terms for how we
> > > describe a tunable (i.e. naming, types, et al) from the "back end"
> > > implementation of any particular scheme for getting values into the
> > > system.
> > 
> > I don't see how such a complicated procedure is needed to add simple
> > tunables. It seems just an elaborate way to say "I can't make up my mind"?
> 
> glibc is known for its stability and strong guarantees of not breaking
> ABIs.  describing something as simple seems to brush that off -- we don't
> want to introduce knobs hastily that we are going to regret and be stuck

This has been discussed since at least two years now.
There's nothing hasty about the existing procedure. In fact "glacially
slow" would be a more appropiate description.

> with forever.  perhaps this particular discussion has been dragging on a
> bit, but glibc isn't really the place for experimentation (which is then
> released directly to users) and for seeing what sticks purely by throwing
> things against the wall.  i'd rather we be overly conservative.

Does that argue for never changing anything?

I don't see how you can even test anything without "releasing it to
users". 

-Andi

On Fri, Sep 18, 2015 at 1:32 AM, Mike Frysinger <vapier@gentoo.org> wrote:
> On 17 Sep 2015 13:32, Andi Kleen wrote:
>> Roland McGrath <roland@hack.frob.com> writes:
>> > The main point is to separate the internal API infrastructure that will be
>> > used pervasively across the source tree and set the terms for how we
>> > describe a tunable (i.e. naming, types, et al) from the "back end"
>> > implementation of any particular scheme for getting values into the
>> > system.
>>
>> I don't see how such a complicated procedure is needed to add simple
>> tunables. It seems just an elaborate way to say "I can't make up my mind"?
>
> glibc is known for its stability and strong guarantees of not breaking
> ABIs.  describing something as simple seems to brush that off -- we don't
> want to introduce knobs hastily that we are going to regret and be stuck
> with forever.  perhaps this particular discussion has been dragging on a
> bit, but glibc isn't really the place for experimentation (which is then
> released directly to users) and for seeing what sticks purely by throwing
> things against the wall.  i'd rather we be overly conservative.
> -mike

Empirically, progress in free software projects has been due more to
throwing and observing stickiness than to cautious approaches, and I
know the specific examples have already come to mind as you're reading
this, so I don't need to recite them yet again. :-)

But part of the point of using a branch-supporting source control
system is exactly to support experiments.  Is the concern that if a
tunable branch becomes popular, there will be too much pressure to
merge it wholesale, good and bad parts together?

Stan

On Fri, 2015-09-18 at 18:06 +0200, Andi Kleen wrote:
> On Fri, Sep 18, 2015 at 02:32:51AM -0400, Mike Frysinger wrote:
> > On 17 Sep 2015 13:32, Andi Kleen wrote:
> > > Roland McGrath <roland@hack.frob.com> writes:
> > > > The main point is to separate the internal API infrastructure that will be
> > > > used pervasively across the source tree and set the terms for how we
> > > > describe a tunable (i.e. naming, types, et al) from the "back end"
> > > > implementation of any particular scheme for getting values into the
> > > > system.
> > > 
> > > I don't see how such a complicated procedure is needed to add simple
> > > tunables. It seems just an elaborate way to say "I can't make up my mind"?
> > 
> > glibc is known for its stability and strong guarantees of not breaking
> > ABIs.  describing something as simple seems to brush that off -- we don't
> > want to introduce knobs hastily that we are going to regret and be stuck
> 
> This has been discussed since at least two years now.
> There's nothing hasty about the existing procedure. In fact "glacially
> slow" would be a more appropiate description.
> 
> > with forever.  perhaps this particular discussion has been dragging on a
> > bit, but glibc isn't really the place for experimentation (which is then
> > released directly to users) and for seeing what sticks purely by throwing
> > things against the wall.  i'd rather we be overly conservative.
> 
> Does that argue for never changing anything?
> 
> I don't see how you can even test anything without "releasing it to
> users". 
> 

I have to agree with Andi on this. What happened to release early,
release often, get stakeholder feedback, adjust, repeat? How do we get
stake holder feed back without releasing code?

In the this specific case we have at least 3 hardware platforms that
support Transactional Memory and TLE. That is at least 7 platform
targets.

GLIBC has released the TLE code. That anyone can enable.

Real hardward, software in a supported community release, real
customers.

Now we are at the stake holder feed back and adjust stage.

So we have a modest proposal to tune (enable / disable) TLE to help
customers deal with the software already gave them.

I agree that we should have a more systematic approach to "tunables",
and we are willing to pitch in to that effort.

But I do not agree that the current situation is so broken and untenable
that we have to stop all progress to design an as yet to be defined
perfect solution.

Carlos has suggested a incremental approach based on systematizing the
environment variable name-space. Which is simple and pragmatic while
moving things in a direction where additional management tooling and
internal frameworks can be developed in parallel.

If Roland still strongly disagrees with this then he should explain in
detail what is problems are with the current situation. And why Carlos'
modest proposal should not be pursued as a starting point

On 18 Sep 2015 18:06, Andi Kleen wrote:
> On Fri, Sep 18, 2015 at 02:32:51AM -0400, Mike Frysinger wrote:
> > On 17 Sep 2015 13:32, Andi Kleen wrote:
> > > Roland McGrath <roland@hack.frob.com> writes:
> > > > The main point is to separate the internal API infrastructure that will be
> > > > used pervasively across the source tree and set the terms for how we
> > > > describe a tunable (i.e. naming, types, et al) from the "back end"
> > > > implementation of any particular scheme for getting values into the
> > > > system.
> > > 
> > > I don't see how such a complicated procedure is needed to add simple
> > > tunables. It seems just an elaborate way to say "I can't make up my mind"?
> > 
> > glibc is known for its stability and strong guarantees of not breaking
> > ABIs.  describing something as simple seems to brush that off -- we don't
> > want to introduce knobs hastily that we are going to regret and be stuck
> 
> This has been discussed since at least two years now.
> There's nothing hasty about the existing procedure. In fact "glacially
> slow" would be a more appropiate description.

i'm aware this particular discussion has been around for quite some time,
but i don't think the result should be "f-it, let's do it live!".

> > with forever.  perhaps this particular discussion has been dragging on a
> > bit, but glibc isn't really the place for experimentation (which is then
> > released directly to users) and for seeing what sticks purely by throwing
> > things against the wall.  i'd rather we be overly conservative.
> 
> Does that argue for never changing anything?
> 
> I don't see how you can even test anything without "releasing it to
> users". 

it argues for a solution we're confident isn't going to screw us over
in the long run and looks like it will scale.
-mike

On 18 September 2015 9:36:48 pm IST, Andi Kleen <andi@firstfloor.org> wrote:
>This has been discussed since at least two years now.
>There's nothing hasty about the existing procedure. In fact "glacially
>slow" would be a more appropiate description.

To be fair, tunables has only been discussed once a year at every Cauldron for the last years and never otherwise until this year. I think you're free to base your patches off my branch and then once we have consensus, it is just a matter of merging patches in. I have started working on this only during this release and I am optimistic about having a useful framework for 2.23.

Siddhesh