| Message ID | 20160116185545.GA17772@devel.intra.reserved-bit.com |
|---|---|
| State | New |
| Headers | show |
On 01/16/2016 12:55 PM, Siddhesh Poyarekar wrote: > The tunables framework allows us to uniformly manage and expose global > variables inside glibc as switches to users. tunables/README has > instructions for glibc developers to add new tunables. > > Tunables support can be enabled by passing the --enable-tunables > configure flag to the configure script. This patch only adds a > framework and does not pose any limitations on how tunable values are > read from the user. It also adds environment variables used in malloc > behaviour tweaking to the tunables framework as a PoC of the > compatibility interface. This looks like a good start to me. Tested on ppc64le with the second patch.
On Mon, Jan 18, 2016 at 03:58:54PM -0600, Paul E. Murphy wrote: > On 01/16/2016 12:55 PM, Siddhesh Poyarekar wrote: > > The tunables framework allows us to uniformly manage and expose global > > variables inside glibc as switches to users. tunables/README has > > instructions for glibc developers to add new tunables. > > > > Tunables support can be enabled by passing the --enable-tunables > > configure flag to the configure script. This patch only adds a > > framework and does not pose any limitations on how tunable values are > > read from the user. It also adds environment variables used in malloc > > behaviour tweaking to the tunables framework as a PoC of the > > compatibility interface. > > This looks like a good start to me. Tested on ppc64le with the second patch. > Thanks for testing. Can someone please review this patch so that I can (fix up if necessary and) push this in 2.23? Thanks, Siddhesh
Ping! On Sun, Jan 17, 2016 at 12:25:50AM +0530, Siddhesh Poyarekar wrote: > The tunables framework allows us to uniformly manage and expose global > variables inside glibc as switches to users. tunables/README has > instructions for glibc developers to add new tunables. > > Tunables support can be enabled by passing the --enable-tunables > configure flag to the configure script. This patch only adds a > framework and does not pose any limitations on how tunable values are > read from the user. It also adds environment variables used in malloc > behaviour tweaking to the tunables framework as a PoC of the > compatibility interface. > > * manual/install.texi: Add --enable-tunables option. > * INSTALL: Regenerate. > * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE. > (all-subdirs): Add tunables. > * config.h.in: Add BUILD_TUNABLES. > * config.make.in: Add build-tunables. > * configure.ac: Add --enable-tunables option. > * configure: Regenerate. > * csu/init-first.c: Include tunables.h. > (_init): Initialize tunables. > * nptl/nptl-init.c [BUILD_TUNABLES]: Include tunables.h. > (__pthread_initialize_minimal_internal): Accept argc, argv and > envp. Initialize tunables. > * csu/libc-start.c (LIBC_START_MAIN): Adjust. > * malloc/arena.c [BUILD_TUNABLES]: Include tunables.h. Define > TUNABLE_NAMESPACE. > (tunable_set_mallopt_top_pad, tunable_set_mallopt_perturb, > tunable_set_mallopt_mmap_max, tunable_set_mallopt_arena_max, > tunable_set_mallopt_arena_test, > tunable_set_mallopt_trim_threshold, > tunable_set_mallopt_mmap_threshold, > tunable_set_mallopt_check): New functions. > (ptmalloc_init): Use them and register tunables. > * scripts/gen-tunables.awk: New file. > * tunables/Makefile: New file. > * tunables/README: New file. > * tunables/Versions: New file. > * tunables/tunable-list.h: New auto-generated file. > * tunables/tunables.c: New file. > * tunables/tunables.h: New file. > * tunables/tunables.list: New file. > --- > INSTALL | 6 ++ > Makeconfig | 9 +++ > config.h.in | 3 + > config.make.in | 1 + > configure | 16 +++++ > configure.ac | 10 +++ > csu/init-first.c | 7 ++ > csu/libc-start.c | 4 +- > malloc/arena.c | 86 +++++++++++++++++++++++++ > manual/install.texi | 5 ++ > nptl/nptl-init.c | 9 ++- > scripts/gen-tunables.awk | 84 ++++++++++++++++++++++++ > tunables/Makefile | 33 ++++++++++ > tunables/README | 93 +++++++++++++++++++++++++++ > tunables/Versions | 8 +++ > tunables/tunable-list.h | 32 ++++++++++ > tunables/tunables.c | 163 +++++++++++++++++++++++++++++++++++++++++++++++ > tunables/tunables.h | 110 ++++++++++++++++++++++++++++++++ > tunables/tunables.list | 12 ++++ > 19 files changed, 688 insertions(+), 3 deletions(-) > create mode 100644 scripts/gen-tunables.awk > create mode 100644 tunables/Makefile > create mode 100644 tunables/README > create mode 100644 tunables/Versions > create mode 100644 tunables/tunable-list.h > create mode 100644 tunables/tunables.c > create mode 100644 tunables/tunables.h > create mode 100644 tunables/tunables.list > > diff --git a/INSTALL b/INSTALL > index c70ea9f..dd06e2e 100644 > --- a/INSTALL > +++ b/INSTALL > @@ -164,6 +164,12 @@ will be used, and CFLAGS sets optimization options for the compiler. > By default for x86_64, the GNU C Library is built with vector math > library. Use this option to disable vector math library. > > +'--enable-tunables' > + Tunables support allows additional library parameters to be > + customized at runtime for each application. This is an > + experimental feature and affects startup time and is thus disabled > + by default. > + > '--build=BUILD-SYSTEM' > '--host=HOST-SYSTEM' > These options are for cross-compiling. If you specify both options > diff --git a/Makeconfig b/Makeconfig > index 87a22e8..8ac199b 100644 > --- a/Makeconfig > +++ b/Makeconfig > @@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \ > $(foreach lib,$(libof-$(basename $(@F))) \ > $(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \ > $(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F))) > + > +ifeq (yes,$(build-tunables)) > +CPPFLAGS += -DTOP_NAMESPACE=glibc > +endif > + > override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \ > $(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \ > $(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \ > @@ -1098,6 +1103,10 @@ all-subdirs = csu assert ctype locale intl catgets math setjmp signal \ > crypt localedata timezone rt conform debug mathvec \ > $(add-on-subdirs) dlfcn elf > > +ifeq (yes,$(build-tunables)) > +all-subdirs += tunables > +endif > + > ifndef avoid-generated > # sysd-sorted itself will contain rules making the sysd-sorted target > # depend on Depend files. But if you just added a Depend file to an > diff --git a/config.h.in b/config.h.in > index ec9c8bc..9b07580 100644 > --- a/config.h.in > +++ b/config.h.in > @@ -231,4 +231,7 @@ > /* PowerPC32 uses fctidz for floating point to long long conversions. */ > #define HAVE_PPC_FCTIDZ 0 > > +/* Build glibc with tunables support. */ > +#define BUILD_TUNABLES 0 > + > #endif > diff --git a/config.make.in b/config.make.in > index 05ed6ec..b158f51 100644 > --- a/config.make.in > +++ b/config.make.in > @@ -90,6 +90,7 @@ use-nscd = @use_nscd@ > build-hardcoded-path-in-tests= @hardcoded_path_in_tests@ > build-pt-chown = @build_pt_chown@ > enable-lock-elision = @enable_lock_elision@ > +build-tunables = @build_tunables@ > > # Build tools. > CC = @CC@ > diff --git a/configure b/configure > index aa05d49..6fef02c 100755 > --- a/configure > +++ b/configure > @@ -659,6 +659,7 @@ multi_arch > base_machine > add_on_subdirs > add_ons > +build_tunables > build_pt_chown > build_nscd > link_obsolete_rpc > @@ -773,6 +774,7 @@ enable_systemtap > enable_build_nscd > enable_nscd > enable_pt_chown > +enable_tunables > enable_mathvec > with_cpu > ' > @@ -1440,6 +1442,7 @@ Optional Features: > --disable-build-nscd disable building and installing the nscd daemon > --disable-nscd library functions will not contact the nscd daemon > --enable-pt_chown Enable building and installing pt_chown > + --enable-tunables Enable tunables support > --enable-mathvec Enable building and installing mathvec [default > depends on architecture] > > @@ -3642,6 +3645,19 @@ if test "$build_pt_chown" = yes; then > > fi > > +# Check whether --enable-tunables was given. > +if test "${enable_tunables+set}" = set; then : > + enableval=$enable_tunables; build_tunables=$enableval > +else > + build_tunables=no > +fi > + > + > +if test "$build_tunables" = yes; then > + $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h > + > +fi > + > # The abi-tags file uses a fairly simplistic model for name recognition that > # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a > # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell. > diff --git a/configure.ac b/configure.ac > index ee7a3f1..41507ac 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -392,6 +392,16 @@ if test "$build_pt_chown" = yes; then > AC_DEFINE(HAVE_PT_CHOWN) > fi > > +AC_ARG_ENABLE([tunables], > + [AS_HELP_STRING([--enable-tunables], > + [Enable tunables support])], > + [build_tunables=$enableval], > + [build_tunables=no]) > +AC_SUBST(build_tunables) > +if test "$build_tunables" = yes; then > + AC_DEFINE(BUILD_TUNABLES) > +fi > + > # The abi-tags file uses a fairly simplistic model for name recognition that > # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a > # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell. > diff --git a/csu/init-first.c b/csu/init-first.c > index 77c6e1c..a4e0c54 100644 > --- a/csu/init-first.c > +++ b/csu/init-first.c > @@ -26,6 +26,9 @@ > #include <sys/param.h> > #include <sys/types.h> > #include <libc-internal.h> > +#if BUILD_TUNABLES > +# include <tunables/tunables.h> > +#endif > > #include <ldsodefs.h> > > @@ -96,6 +99,10 @@ _init (int argc, char **argv, char **envp) > #if defined SHARED && !defined NO_CTORS_DTORS_SECTIONS > __libc_global_ctors (); > #endif > + > +#if BUILD_TUNABLES > + __tunables_init (__environ); > +#endif > } > > /* This function is defined here so that if this file ever gets into > diff --git a/csu/libc-start.c b/csu/libc-start.c > index f4aa01a..7629fec 100644 > --- a/csu/libc-start.c > +++ b/csu/libc-start.c > @@ -32,7 +32,7 @@ extern int __libc_multiple_libcs; > #include <tls.h> > #ifndef SHARED > # include <dl-osinfo.h> > -extern void __pthread_initialize_minimal (void); > +extern void __pthread_initialize_minimal (int, char **, char **); > # ifndef THREAD_SET_STACK_GUARD > /* Only exported for architectures that don't store the stack guard canary > in thread local area. */ > @@ -193,7 +193,7 @@ LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL), > /* Initialize the thread library at least a bit since the libgcc > functions are using thread functions if these are available and > we need to setup errno. */ > - __pthread_initialize_minimal (); > + __pthread_initialize_minimal (argc, argv, __environ); > > /* Set up the stack checker's canary. */ > uintptr_t stack_chk_guard = _dl_setup_stack_chk_guard (_dl_random); > diff --git a/malloc/arena.c b/malloc/arena.c > index 1edb4d4..f773b10 100644 > --- a/malloc/arena.c > +++ b/malloc/arena.c > @@ -19,6 +19,11 @@ > > #include <stdbool.h> > > +#if BUILD_TUNABLES > +# define TUNABLE_NAMESPACE malloc > +# include <tunables/tunables.h> > +#endif > + > /* Compile-time constants. */ > > #define HEAP_MIN_SIZE (32 * 1024) > @@ -332,6 +337,61 @@ ptmalloc_unlock_all2 (void) > # endif > #endif /* !NO_THREADS */ > > +#if BUILD_TUNABLES > +static void > +tunable_set_mallopt_top_pad (const char *val) > +{ > + __libc_mallopt (M_TOP_PAD, atoi (val)); > +} > + > +static void > +tunable_set_mallopt_perturb (const char *val) > +{ > + __libc_mallopt (M_PERTURB, atoi (val)); > +} > + > +static void > +tunable_set_mallopt_mmap_max (const char *val) > +{ > + __libc_mallopt (M_MMAP_MAX, atoi (val)); > +} > + > +static void > +tunable_set_mallopt_arena_max (const char *val) > +{ > + __libc_mallopt (M_ARENA_MAX, atoi (val)); > +} > + > +static void > +tunable_set_mallopt_arena_test (const char *val) > +{ > + __libc_mallopt (M_ARENA_TEST, atoi (val)); > +} > + > +static void > +tunable_set_mallopt_trim_threshold (const char *val) > +{ > + __libc_mallopt (M_TRIM_THRESHOLD, atoi (val)); > +} > + > +static void > +tunable_set_mallopt_mmap_threshold (const char *val) > +{ > + __libc_mallopt (M_MMAP_THRESHOLD, atoi (val)); > +} > + > +static void > +tunable_set_mallopt_check (const char *val) > +{ > + if (val[0]) > + { > + __libc_mallopt (M_CHECK_ACTION, (int) (val[0] - '0')); > + if (check_action != 0) > + __malloc_check_init (); > + } > +} > + > +#else > /* Initialization routine. */ > #include <string.h> > extern char **_environ; > @@ -366,6 +426,7 @@ next_env_entry (char ***position) > > return result; > } > +#endif > > > #ifdef SHARED > @@ -401,6 +462,29 @@ ptmalloc_init (void) > > thread_arena = &main_arena; > thread_atfork (ptmalloc_lock_all, ptmalloc_unlock_all, ptmalloc_unlock_all2); > + > +#if BUILD_TUNABLES > + COMPAT_TUNABLES_NAMESPACE_BEGIN (8); > + > + COMPAT_TUNABLE_REGISTER_SECURE (check, "MALLOC_CHECK_", > + tunable_set_mallopt_check); > + COMPAT_TUNABLE_REGISTER (top_pad, "MALLOC_TOP_PAD_", > + tunable_set_mallopt_top_pad); > + COMPAT_TUNABLE_REGISTER (perturb, "MALLOC_PERTURB_", > + tunable_set_mallopt_perturb); > + COMPAT_TUNABLE_REGISTER (mmap_threshold, "MALLOC_MMAP_THRESHOLD_", > + tunable_set_mallopt_mmap_threshold); > + COMPAT_TUNABLE_REGISTER (trim_threshold, "MALLOC_TRIM_THRESHOLD_", > + tunable_set_mallopt_trim_threshold); > + COMPAT_TUNABLE_REGISTER (mmap_max, "MALLOC_MMAP_MAX_", > + tunable_set_mallopt_mmap_max); > + COMPAT_TUNABLE_REGISTER (arena_max, "MALLOC_ARENA_MAX", > + tunable_set_mallopt_arena_max); > + COMPAT_TUNABLE_REGISTER (arena_test, "MALLOC_ARENA_TEST", > + tunable_set_mallopt_arena_test); > + > + COMPAT_TUNABLES_NAMESPACE_INIT (); > +#else > const char *s = NULL; > if (__glibc_likely (_environ != NULL)) > { > @@ -469,6 +553,8 @@ ptmalloc_init (void) > if (check_action != 0) > __malloc_check_init (); > } > +#endif > + > void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook); > if (hook != NULL) > (*hook)(); > diff --git a/manual/install.texi b/manual/install.texi > index de9d270..95392ea 100644 > --- a/manual/install.texi > +++ b/manual/install.texi > @@ -195,6 +195,11 @@ configure with @option{--disable-werror}. > By default for x86_64, @theglibc{} is built with vector math library. > Use this option to disable vector math library. > > +@item --enable-tunables > +Tunables support allows additional library parameters to be customized at > +runtime for each application. This is an experimental feature and affects > +startup time and is thus disabled by default. > + > @item --build=@var{build-system} > @itemx --host=@var{host-system} > These options are for cross-compiling. If you specify both options and > diff --git a/nptl/nptl-init.c b/nptl/nptl-init.c > index bdbdfed..6667801 100644 > --- a/nptl/nptl-init.c > +++ b/nptl/nptl-init.c > @@ -38,6 +38,9 @@ > #include <kernel-features.h> > #include <libc-internal.h> > #include <pthread-pids.h> > +#if BUILD_TUNABLES > +# include <tunables/tunables.h> > +#endif > > #ifndef TLS_MULTIPLE_THREADS_IN_TCB > /* Pointer to the corresponding variable in libc. */ > @@ -297,7 +300,7 @@ extern void **__libc_dl_error_tsd (void) __attribute__ ((const)); > static bool __nptl_initial_report_events __attribute_used__; > > void > -__pthread_initialize_minimal_internal (void) > +__pthread_initialize_minimal_internal (int argc, char **argv, char **envp) > { > #ifndef SHARED > /* Unlike in the dynamically linked case the dynamic linker has not > @@ -311,6 +314,10 @@ __pthread_initialize_minimal_internal (void) > __asm __volatile (""); > #endif > > +#if BUILD_TUNABLES > + __tunables_init (envp); > +#endif > + > /* Minimal initialization of the thread descriptor. */ > struct pthread *pd = THREAD_SELF; > __pthread_initialize_pids (pd); > diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk > new file mode 100644 > index 0000000..ccbf789 > --- /dev/null > +++ b/scripts/gen-tunables.awk > @@ -0,0 +1,84 @@ > +# Generate tunable-list.h from tunables.list > + > +BEGIN { > + ns="" > + top_ns="" > +} > + > +$2 == "{" { > + if (top_ns == "") { > + top_ns = $1 > + } > + else if (ns == "") { > + ns = $1 > + count = 0 > + } > + else { > + printf ("Unexpected occurrence of '{' inside a namespace: %s:%d\n", > + FILENAME, FNR) > + exit 1 > + } > + > + next > +} > + > +$1 == "}" { > + if (ns != "") { > + ns = "" > + } > + else if (top_ns != "") { > + top_ns = "" > + } > + else { > + printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR) > + exit 1 > + } > + next > +} > + > +{ > + if (ns == "") { > + print "Invalid tunable outside a namespace" > + exit 1 > + } > + val[top_ns][ns][count] = $1 > + count = count + 1 > +} > + > +END { > + if (ns != "") { > + print "Unterminated namespace. Is a closing brace missing?" > + exit 1 > + } > + > + print "/* Print a full tunable enum name. */" > + print "#include <stddef.h>" > + print "#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)" > + print "#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id\n" > + > + print "/* Full name for a tunable is top_ns.tunable_ns.id. */" > + print "#define TUNABLE_NAME_S(top,ns,id) #top \".\" #ns \".\" #id\n" > + > + print "typedef enum" > + print "{" > + for (t in val) { > + for (n in val[t]) { > + for (c in val[t][n]) { > + printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, val[t][n][c]); > + } > + } > + } > + print "} tunable_id_t;\n" > + print "#ifdef TUNABLES_INTERNAL" > + print "static tunable_t tunable_list[] = {" > + for (t in val) { > + for (n in val[t]) { > + for (c in val[t][n]) { > + printf (" {TUNABLE_NAME_S(%s, %s, %s), NULL, NULL, false},\n", > + t, n, val[t][n][c], t, n, val[t][n][c]); > + } > + } > + } > + print "};" > + print "#endif" > +} > diff --git a/tunables/Makefile b/tunables/Makefile > new file mode 100644 > index 0000000..44439f6 > --- /dev/null > +++ b/tunables/Makefile > @@ -0,0 +1,33 @@ > +# Copyright (C) 2016 Free Software Foundation, Inc. > +# This file is part of the GNU C Library. > + > +# The GNU C Library is free software; you can redistribute it and/or > +# modify it under the terms of the GNU Lesser General Public > +# License as published by the Free Software Foundation; either > +# version 2.1 of the License, or (at your option) any later version. > + > +# The GNU C Library is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > +# Lesser General Public License for more details. > + > +# You should have received a copy of the GNU Lesser General Public > +# License along with the GNU C Library; if not, see > +# <http://www.gnu.org/licenses/>. > + > +# > +# Makefile for tunables. > +# > +subdir := tunables > + > +include ../Makeconfig > + > +routines = tunables > + > +$(objpfx)tunables.os: tunable-list.h > + > +tunable-list.h: $(..)scripts/gen-tunables.awk tunables.list > + $(AWK) -f $^ > $@.tmp > + mv $@{.tmp,} > + > +include ../Rules > diff --git a/tunables/README b/tunables/README > new file mode 100644 > index 0000000..02a3f7a > --- /dev/null > +++ b/tunables/README > @@ -0,0 +1,93 @@ > + TUNABLE FRAMEWORK > + ================= > + > +The tunable framework allows modules within glibc to register variables that > +may be tweaked through an environment variable or an API call. It aims to > +enforce a strict namespace rule to bring consistency to naming of these tunable > +environment variables across the project. > + > +ADDING A NEW TUNABLE > +-------------------- > + > +The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in > +distributions for specific tunables if they want to add their own tunables. > +Downstream implementations are discouraged from using the 'glibc' namespace for > +tunables they don't already have consensus to push upstream. > + > +There are two steps to adding a tunable: > + > +1. Add a tunable ID: > + > +Modules that wish to use the tunables interface must define the > +TUNABLE_NAMESPACE macro. Following this, for each tunable you want to > +register, call the TUNABLE_REGISTER macro. > + > +The TUNABLE_REGISTER macro takes the following arguments: > + > +- id: The short name of the tunable. It will be concatenated with > + the TOP_NAMESPACE and TUNABLE_NAMESPACE to build the full ID of > + the tunable. > + > +- setter: A function that accepts a string input and initializes the > + tunable variable. The prototype of the function should be as > + follows: > + > + void setter_func (const char *) > + > +2. Add to tunables.list: > + > +This file nests tunable ids inside their tunable namespace within curly braces > +and multiple such namespaces are in turn nested inside their top namespaces in > +a similar manner. The malloc and resolv snippets for example could look like > +this: > + > +glibc { > + malloc { > + check > + top_pad > + } > + resolv { > + secure_dns > + } > +} > + > +Add your tunable in this hierarchy using the proper nesting for your namespace > +and top namespace. If the top namespace is not glibc, create a new snippet > +with its own hierarchy, separate from the glibc hierarchy. > + > +ADDING A LEGACY TUNABLE > +----------------------- > + > +One may add tunables for variables that are currently controlled by users with > +environment variables. Multiple such variables should be defined together for > +better performance by using the COMPAT_TUNABLES_NAMESPACE_BEGIN macro, which > +accepts a numeric parameter with the number of tunables you intend to register > +in the block. > + > +Individual tunables are then registered using the COMPAT_TUNABLE_REGISTER and > +COMPAT_TUNABLE_REGISTER_SECURE macros. They take the following parameters: > + > +- id: The tunable ID, similar to the normal way of adding a tunable > + > +- env: The name of the legacy environment variable so that one may > + read its value and use it. > + > +- setter: The setter function, similar to the normal way of adding a > + tunable. > + > +COMPAT_TUNABLE_REGISTER reads the environment variable only for non-setuid > +binaries, whereas COMPAT_TUNABLE_REGISTER_SECURE always attempts to read the > +environment variable. Use COMPAT_TUNABLE_REGISTER_SECURE with care. > + > +You will also have to add the tunable id to tunables/tunables.list. > + > +DO NOT ADD NEW TUNABLES USING THIS MECHANISM! > + > +FUTURE WORK > +----------- > + > +The framework currently only allows a one-time initialization of variables > +through environment variables and in some cases, modification of variables via > +an API call. A future goal for this project is to allow tweaking of some > +values in a running process, possibly through some kind of shared memory > +mechanism. > diff --git a/tunables/Versions b/tunables/Versions > new file mode 100644 > index 0000000..4c68b0e > --- /dev/null > +++ b/tunables/Versions > @@ -0,0 +1,8 @@ > +# Exports from tunables should only be in the GLIBC_PRIVATE namespace. > +libc { > + GLIBC_PRIVATE { > + __compat_tunables_init_envvars; > + __tunables_init; > + __tunable_register; > + } > +} > diff --git a/tunables/tunable-list.h b/tunables/tunable-list.h > new file mode 100644 > index 0000000..2b1caa4 > --- /dev/null > +++ b/tunables/tunable-list.h > @@ -0,0 +1,32 @@ > +/* Print a full tunable enum name. */ > +#include <stddef.h> > +#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id) > +#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id > + > +/* Full name for a tunable is top_ns.tunable_ns.id. */ > +#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id > + > +typedef enum > +{ > + TUNABLE_ENUM_NAME(glibc, malloc, check), > + TUNABLE_ENUM_NAME(glibc, malloc, top_pad), > + TUNABLE_ENUM_NAME(glibc, malloc, perturb), > + TUNABLE_ENUM_NAME(glibc, malloc, mmap_threshold), > + TUNABLE_ENUM_NAME(glibc, malloc, trim_threshold), > + TUNABLE_ENUM_NAME(glibc, malloc, mmap_max), > + TUNABLE_ENUM_NAME(glibc, malloc, arena_max), > + TUNABLE_ENUM_NAME(glibc, malloc, arena_test), > +} tunable_id_t; > + > +#ifdef TUNABLES_INTERNAL > +static tunable_t tunable_list[] = { > + {TUNABLE_NAME_S(glibc, malloc, check), NULL, NULL, false}, > + {TUNABLE_NAME_S(glibc, malloc, top_pad), NULL, NULL, false}, > + {TUNABLE_NAME_S(glibc, malloc, perturb), NULL, NULL, false}, > + {TUNABLE_NAME_S(glibc, malloc, mmap_threshold), NULL, NULL, false}, > + {TUNABLE_NAME_S(glibc, malloc, trim_threshold), NULL, NULL, false}, > + {TUNABLE_NAME_S(glibc, malloc, mmap_max), NULL, NULL, false}, > + {TUNABLE_NAME_S(glibc, malloc, arena_max), NULL, NULL, false}, > + {TUNABLE_NAME_S(glibc, malloc, arena_test), NULL, NULL, false}, > +}; > +#endif > diff --git a/tunables/tunables.c b/tunables/tunables.c > new file mode 100644 > index 0000000..8834aff > --- /dev/null > +++ b/tunables/tunables.c > @@ -0,0 +1,163 @@ > +/* The tunable framework. See the README to know how to use the tunable in > + a glibc module. > + > + Copyright (C) 2016 Free Software Foundation, Inc. > + This file is part of the GNU C Library. > + > + The GNU C Library is free software; you can redistribute it and/or > + modify it under the terms of the GNU Lesser General Public > + License as published by the Free Software Foundation; either > + version 2.1 of the License, or (at your option) any later version. > + > + The GNU C Library is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + Lesser General Public License for more details. > + > + You should have received a copy of the GNU Lesser General Public > + License along with the GNU C Library; if not, see > + <http://www.gnu.org/licenses/>. */ > + > +#include <stddef.h> > +#include <stdlib.h> > +#include <string.h> > +#include <stdbool.h> > +#include <unistd.h> > +#include <sys/param.h> > + > +extern char **__environ; > + > +#define TUNABLES_INTERNAL 1 > +#include "tunables.h" > + > +/* We avoid calling into the C library as much as we can, especially functions > + that we know could use tunables in future for some reason or the other. For > + example, we call mmap directly instead of malloc because malloc uses > + tunables. Similarly, the string functions may use tunables in future to > + select optimal routines and we want to make sure we stay out of the way and > + not invoke their initializers before the tunable list has been > + initialized. */ > + > +static void > +t_memcpy (char *dest, const char *src, size_t len) > +{ > + for (size_t i = 0; i < len; i++) > + dest[i] = src[i]; > +} > + > +static int > +t_strcmp (const char *a, const char *b) > +{ > + size_t i = 0; > + > + for (i = 0; *a != '\0' && *b != '\0'; i++, a++, b++) > + if (*a != *b) > + return *a - *b; > + > + return *a - *b; > +} > + > +static bool > +get_next_env (char ***envp, char **name, size_t *namelen, char **val) > +{ > + char **ev = *envp; > + > + while (ev != NULL && *ev != '\0') > + { > + char *envline = *ev; > + int len = 0; > + > + while (envline[len] != '\0' && envline[len] != '=') > + len++; > + > + /* Just the name and no value, go to the next one. */ > + if (envline[len] == '\0') > + continue; > + > + *name = envline; > + *namelen = len; > + *val = &envline[len + 1]; > + *envp = ++ev; > + > + return true; > + } > + > + return false; > +} > + > +/* Initialize tunables from the GLIBC_TUNABLES environment variable. The > + variable is set as colon separated name=value pairs. This routine may be > + called from the constructor of either libpthread.so or lic.so. Constructor > + calls for libraries are protected by the dl_load_lock, so this routine won't > + be called concurrently from diferent threads. */ > +void > +__tunables_init (char **envp) > +{ > + /* Empty for now. */ > +} > +libc_hidden_def (__tunables_init) > + > +/* Initialize all tunables using its legacy environment variable values whose > + names are passed in ENVVARS. */ > +void > +__compat_tunables_init_envvars (struct compat_tunable_env *envvars, int count) > +{ > + /* Traverse through the environment to find environment variables we may need > + to set. */ > + char **envp = __environ; > + char *envname = NULL; > + char *envval = NULL; > + size_t len = 0; > + > + while (get_next_env (&envp, &envname, &len, &envval)) > + { > + int init_count = 0; > + for (int i = 0; i < count; i++) > + { > + tunable_id_t t = envvars[i].id; > + tunable_t *cur = &tunable_list[t]; > + > + /* Skip over tunables that have already been initialized. */ > + if (cur->initialized) > + { > + init_count++; > + continue; > + } > + > + const char *name = envvars[i].env; > + char *ename = alloca (len + 1); > + > + t_memcpy (ename, envname, len); > + ename[len] = '\0'; > + > + /* We have a match. Initialize and move on to the next line. */ > + if (t_strcmp (ename, name) == 0) > + { > + cur->val = envval; > + cur->set (cur->val); > + cur->initialized = true; > + break; > + } > + } > + > + /* All of the tunable envvars have been initialized. */ > + if (count == init_count) > + break; > + } > +} > +libc_hidden_def (__compat_tunables_init_envvars) > + > +/* Initialize a tunable and set its value. */ > +void > +__tunable_register (tunable_id_t id, tunable_setter_t set_func) > +{ > + tunable_t *cur = &tunable_list[id]; > + > + cur->set = set_func; > + if (cur->val != NULL) > + { > + set_func (cur->val); > + cur->initialized = true; > + } > +} > +libc_hidden_def (__tunable_register) > diff --git a/tunables/tunables.h b/tunables/tunables.h > new file mode 100644 > index 0000000..6aef30e > --- /dev/null > +++ b/tunables/tunables.h > @@ -0,0 +1,110 @@ > +/* The tunable framework. See the README to know how to use the tunable in > + a glibc module. > + > + Copyright (C) 2016 Free Software Foundation, Inc. > + This file is part of the GNU C Library. > + > + The GNU C Library is free software; you can redistribute it and/or > + modify it under the terms of the GNU Lesser General Public > + License as published by the Free Software Foundation; either > + version 2.1 of the License, or (at your option) any later version. > + > + The GNU C Library is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + Lesser General Public License for more details. > + > + You should have received a copy of the GNU Lesser General Public > + License along with the GNU C Library; if not, see > + <http://www.gnu.org/licenses/>. */ > + > +#include <stdbool.h> > + > +typedef void (*tunable_setter_t) (const char *); > + > +/* A tunable. */ > +struct _tunable > +{ > + const char *name; > + const char *val; > + tunable_setter_t set; > + bool initialized; > +}; > + > +typedef struct _tunable tunable_t; > + > +#include "tunable-list.h" > + > +struct compat_tunable_env > +{ > + tunable_id_t id; > + const char *env; > +}; > + > + > +extern void __compat_tunables_init_envvars (struct compat_tunable_env *, int); > +extern void __tunable_register (tunable_id_t, tunable_setter_t); > +extern void __tunables_init (char **); > + > +/* Avoid a PLT reference when called from within libc.so. */ > +libc_hidden_proto (__compat_tunables_init_envvars) > +libc_hidden_proto (__tunable_register) > +libc_hidden_proto (__tunables_init) > + > +/* Register a tunable ID in namespace NS inside the glibc top namespace with > + SET as the setter function. */ > +#define TUNABLE_REGISTER(ns,id,set) \ > + TUNABLE_REGISTER_FULL (TOP_NAMESPACE, ns, id, set) > + > +/* Register a tunable ID in namespace NS inside the TNS top namespace with > + SET as the setter function. */ > +#define TUNABLE_REGISTER_FULL(tns,ns,id,set) \ > + __tunable_register (TUNABLE_ENUM_NAME (tns, ns, id), (set)) > + > + > + > +/* COMPATIBILITY INTERFACE. These macros should be used to define tunables for > + variables that are currently being controlled using environment variables. > + New tunables must not use this interface. */ > + > +#define _ADD_COMPAT_TUNABLE_ENV(__id,__env) \ > +({ \ > + envvars[envvars_cnt].id = TUNABLE_ENUM_NAME (TOP_NAMESPACE, \ > + TUNABLE_NAMESPACE, __id); \ > + envvars[envvars_cnt++].env = (__env); \ > +}) > + > +/* Start registering tunables in the current namespace. */ > +#define COMPAT_TUNABLES_NAMESPACE_BEGIN(size) \ > + { \ > + struct compat_tunable_env envvars[size]; \ > + int envvars_cnt = 0; > + > +/* Register a tunable. This macro validates that the call is OK and then calls > + tunable_init to do the real work of adding the tunable and setting its value > + based on its environment variable(s). */ > +#define COMPAT_TUNABLE_REGISTER(id,env,set) \ > +({ \ > + assert (envvars_cnt < (sizeof (envvars) \ > + / sizeof (struct compat_tunable_env))); \ > + if (!__libc_enable_secure) \ > + _ADD_COMPAT_TUNABLE_ENV (id, env); \ > + TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \ > +}) > + > +/* Does exactly the same thing as TUNABLE_REGISTER, except that it allows the > + tunable to look for environment variable values even for setuid binaries. > + This is a separate macro and not just another parameter in TUNABLE_REGISTER > + to avoid accidentally setting a secure flag where it is not required. */ > +#define COMPAT_TUNABLE_REGISTER_SECURE(id,env,set) \ > +({ \ > + assert (envvars_cnt < (sizeof (envvars) \ > + / sizeof (struct compat_tunable_env))); \ > + _ADD_COMPAT_TUNABLE_ENV(id, env); \ > + TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \ > +}) > + > +/* Initialize tunables in the namespace. */ > +#define COMPAT_TUNABLES_NAMESPACE_INIT() \ > + __compat_tunables_init_envvars (envvars, envvars_cnt); \ > + } > diff --git a/tunables/tunables.list b/tunables/tunables.list > new file mode 100644 > index 0000000..f1335cc > --- /dev/null > +++ b/tunables/tunables.list > @@ -0,0 +1,12 @@ > +glibc { > + malloc { > + check > + top_pad > + perturb > + mmap_threshold > + trim_threshold > + mmap_max > + arena_max > + arena_test > + } > +} > -- > 2.5.0 >
Ping! On Mon, Jan 25, 2016 at 08:55:06AM +0530, Siddhesh Poyarekar wrote: > Ping! > > On Sun, Jan 17, 2016 at 12:25:50AM +0530, Siddhesh Poyarekar wrote: > > The tunables framework allows us to uniformly manage and expose global > > variables inside glibc as switches to users. tunables/README has > > instructions for glibc developers to add new tunables. > > > > Tunables support can be enabled by passing the --enable-tunables > > configure flag to the configure script. This patch only adds a > > framework and does not pose any limitations on how tunable values are > > read from the user. It also adds environment variables used in malloc > > behaviour tweaking to the tunables framework as a PoC of the > > compatibility interface. > > > > * manual/install.texi: Add --enable-tunables option. > > * INSTALL: Regenerate. > > * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE. > > (all-subdirs): Add tunables. > > * config.h.in: Add BUILD_TUNABLES. > > * config.make.in: Add build-tunables. > > * configure.ac: Add --enable-tunables option. > > * configure: Regenerate. > > * csu/init-first.c: Include tunables.h. > > (_init): Initialize tunables. > > * nptl/nptl-init.c [BUILD_TUNABLES]: Include tunables.h. > > (__pthread_initialize_minimal_internal): Accept argc, argv and > > envp. Initialize tunables. > > * csu/libc-start.c (LIBC_START_MAIN): Adjust. > > * malloc/arena.c [BUILD_TUNABLES]: Include tunables.h. Define > > TUNABLE_NAMESPACE. > > (tunable_set_mallopt_top_pad, tunable_set_mallopt_perturb, > > tunable_set_mallopt_mmap_max, tunable_set_mallopt_arena_max, > > tunable_set_mallopt_arena_test, > > tunable_set_mallopt_trim_threshold, > > tunable_set_mallopt_mmap_threshold, > > tunable_set_mallopt_check): New functions. > > (ptmalloc_init): Use them and register tunables. > > * scripts/gen-tunables.awk: New file. > > * tunables/Makefile: New file. > > * tunables/README: New file. > > * tunables/Versions: New file. > > * tunables/tunable-list.h: New auto-generated file. > > * tunables/tunables.c: New file. > > * tunables/tunables.h: New file. > > * tunables/tunables.list: New file. > > --- > > INSTALL | 6 ++ > > Makeconfig | 9 +++ > > config.h.in | 3 + > > config.make.in | 1 + > > configure | 16 +++++ > > configure.ac | 10 +++ > > csu/init-first.c | 7 ++ > > csu/libc-start.c | 4 +- > > malloc/arena.c | 86 +++++++++++++++++++++++++ > > manual/install.texi | 5 ++ > > nptl/nptl-init.c | 9 ++- > > scripts/gen-tunables.awk | 84 ++++++++++++++++++++++++ > > tunables/Makefile | 33 ++++++++++ > > tunables/README | 93 +++++++++++++++++++++++++++ > > tunables/Versions | 8 +++ > > tunables/tunable-list.h | 32 ++++++++++ > > tunables/tunables.c | 163 +++++++++++++++++++++++++++++++++++++++++++++++ > > tunables/tunables.h | 110 ++++++++++++++++++++++++++++++++ > > tunables/tunables.list | 12 ++++ > > 19 files changed, 688 insertions(+), 3 deletions(-) > > create mode 100644 scripts/gen-tunables.awk > > create mode 100644 tunables/Makefile > > create mode 100644 tunables/README > > create mode 100644 tunables/Versions > > create mode 100644 tunables/tunable-list.h > > create mode 100644 tunables/tunables.c > > create mode 100644 tunables/tunables.h > > create mode 100644 tunables/tunables.list > > > > diff --git a/INSTALL b/INSTALL > > index c70ea9f..dd06e2e 100644 > > --- a/INSTALL > > +++ b/INSTALL > > @@ -164,6 +164,12 @@ will be used, and CFLAGS sets optimization options for the compiler. > > By default for x86_64, the GNU C Library is built with vector math > > library. Use this option to disable vector math library. > > > > +'--enable-tunables' > > + Tunables support allows additional library parameters to be > > + customized at runtime for each application. This is an > > + experimental feature and affects startup time and is thus disabled > > + by default. > > + > > '--build=BUILD-SYSTEM' > > '--host=HOST-SYSTEM' > > These options are for cross-compiling. If you specify both options > > diff --git a/Makeconfig b/Makeconfig > > index 87a22e8..8ac199b 100644 > > --- a/Makeconfig > > +++ b/Makeconfig > > @@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \ > > $(foreach lib,$(libof-$(basename $(@F))) \ > > $(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \ > > $(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F))) > > + > > +ifeq (yes,$(build-tunables)) > > +CPPFLAGS += -DTOP_NAMESPACE=glibc > > +endif > > + > > override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \ > > $(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \ > > $(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \ > > @@ -1098,6 +1103,10 @@ all-subdirs = csu assert ctype locale intl catgets math setjmp signal \ > > crypt localedata timezone rt conform debug mathvec \ > > $(add-on-subdirs) dlfcn elf > > > > +ifeq (yes,$(build-tunables)) > > +all-subdirs += tunables > > +endif > > + > > ifndef avoid-generated > > # sysd-sorted itself will contain rules making the sysd-sorted target > > # depend on Depend files. But if you just added a Depend file to an > > diff --git a/config.h.in b/config.h.in > > index ec9c8bc..9b07580 100644 > > --- a/config.h.in > > +++ b/config.h.in > > @@ -231,4 +231,7 @@ > > /* PowerPC32 uses fctidz for floating point to long long conversions. */ > > #define HAVE_PPC_FCTIDZ 0 > > > > +/* Build glibc with tunables support. */ > > +#define BUILD_TUNABLES 0 > > + > > #endif > > diff --git a/config.make.in b/config.make.in > > index 05ed6ec..b158f51 100644 > > --- a/config.make.in > > +++ b/config.make.in > > @@ -90,6 +90,7 @@ use-nscd = @use_nscd@ > > build-hardcoded-path-in-tests= @hardcoded_path_in_tests@ > > build-pt-chown = @build_pt_chown@ > > enable-lock-elision = @enable_lock_elision@ > > +build-tunables = @build_tunables@ > > > > # Build tools. > > CC = @CC@ > > diff --git a/configure b/configure > > index aa05d49..6fef02c 100755 > > --- a/configure > > +++ b/configure > > @@ -659,6 +659,7 @@ multi_arch > > base_machine > > add_on_subdirs > > add_ons > > +build_tunables > > build_pt_chown > > build_nscd > > link_obsolete_rpc > > @@ -773,6 +774,7 @@ enable_systemtap > > enable_build_nscd > > enable_nscd > > enable_pt_chown > > +enable_tunables > > enable_mathvec > > with_cpu > > ' > > @@ -1440,6 +1442,7 @@ Optional Features: > > --disable-build-nscd disable building and installing the nscd daemon > > --disable-nscd library functions will not contact the nscd daemon > > --enable-pt_chown Enable building and installing pt_chown > > + --enable-tunables Enable tunables support > > --enable-mathvec Enable building and installing mathvec [default > > depends on architecture] > > > > @@ -3642,6 +3645,19 @@ if test "$build_pt_chown" = yes; then > > > > fi > > > > +# Check whether --enable-tunables was given. > > +if test "${enable_tunables+set}" = set; then : > > + enableval=$enable_tunables; build_tunables=$enableval > > +else > > + build_tunables=no > > +fi > > + > > + > > +if test "$build_tunables" = yes; then > > + $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h > > + > > +fi > > + > > # The abi-tags file uses a fairly simplistic model for name recognition that > > # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a > > # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell. > > diff --git a/configure.ac b/configure.ac > > index ee7a3f1..41507ac 100644 > > --- a/configure.ac > > +++ b/configure.ac > > @@ -392,6 +392,16 @@ if test "$build_pt_chown" = yes; then > > AC_DEFINE(HAVE_PT_CHOWN) > > fi > > > > +AC_ARG_ENABLE([tunables], > > + [AS_HELP_STRING([--enable-tunables], > > + [Enable tunables support])], > > + [build_tunables=$enableval], > > + [build_tunables=no]) > > +AC_SUBST(build_tunables) > > +if test "$build_tunables" = yes; then > > + AC_DEFINE(BUILD_TUNABLES) > > +fi > > + > > # The abi-tags file uses a fairly simplistic model for name recognition that > > # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a > > # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell. > > diff --git a/csu/init-first.c b/csu/init-first.c > > index 77c6e1c..a4e0c54 100644 > > --- a/csu/init-first.c > > +++ b/csu/init-first.c > > @@ -26,6 +26,9 @@ > > #include <sys/param.h> > > #include <sys/types.h> > > #include <libc-internal.h> > > +#if BUILD_TUNABLES > > +# include <tunables/tunables.h> > > +#endif > > > > #include <ldsodefs.h> > > > > @@ -96,6 +99,10 @@ _init (int argc, char **argv, char **envp) > > #if defined SHARED && !defined NO_CTORS_DTORS_SECTIONS > > __libc_global_ctors (); > > #endif > > + > > +#if BUILD_TUNABLES > > + __tunables_init (__environ); > > +#endif > > } > > > > /* This function is defined here so that if this file ever gets into > > diff --git a/csu/libc-start.c b/csu/libc-start.c > > index f4aa01a..7629fec 100644 > > --- a/csu/libc-start.c > > +++ b/csu/libc-start.c > > @@ -32,7 +32,7 @@ extern int __libc_multiple_libcs; > > #include <tls.h> > > #ifndef SHARED > > # include <dl-osinfo.h> > > -extern void __pthread_initialize_minimal (void); > > +extern void __pthread_initialize_minimal (int, char **, char **); > > # ifndef THREAD_SET_STACK_GUARD > > /* Only exported for architectures that don't store the stack guard canary > > in thread local area. */ > > @@ -193,7 +193,7 @@ LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL), > > /* Initialize the thread library at least a bit since the libgcc > > functions are using thread functions if these are available and > > we need to setup errno. */ > > - __pthread_initialize_minimal (); > > + __pthread_initialize_minimal (argc, argv, __environ); > > > > /* Set up the stack checker's canary. */ > > uintptr_t stack_chk_guard = _dl_setup_stack_chk_guard (_dl_random); > > diff --git a/malloc/arena.c b/malloc/arena.c > > index 1edb4d4..f773b10 100644 > > --- a/malloc/arena.c > > +++ b/malloc/arena.c > > @@ -19,6 +19,11 @@ > > > > #include <stdbool.h> > > > > +#if BUILD_TUNABLES > > +# define TUNABLE_NAMESPACE malloc > > +# include <tunables/tunables.h> > > +#endif > > + > > /* Compile-time constants. */ > > > > #define HEAP_MIN_SIZE (32 * 1024) > > @@ -332,6 +337,61 @@ ptmalloc_unlock_all2 (void) > > # endif > > #endif /* !NO_THREADS */ > > > > +#if BUILD_TUNABLES > > +static void > > +tunable_set_mallopt_top_pad (const char *val) > > +{ > > + __libc_mallopt (M_TOP_PAD, atoi (val)); > > +} > > + > > +static void > > +tunable_set_mallopt_perturb (const char *val) > > +{ > > + __libc_mallopt (M_PERTURB, atoi (val)); > > +} > > + > > +static void > > +tunable_set_mallopt_mmap_max (const char *val) > > +{ > > + __libc_mallopt (M_MMAP_MAX, atoi (val)); > > +} > > + > > +static void > > +tunable_set_mallopt_arena_max (const char *val) > > +{ > > + __libc_mallopt (M_ARENA_MAX, atoi (val)); > > +} > > + > > +static void > > +tunable_set_mallopt_arena_test (const char *val) > > +{ > > + __libc_mallopt (M_ARENA_TEST, atoi (val)); > > +} > > + > > +static void > > +tunable_set_mallopt_trim_threshold (const char *val) > > +{ > > + __libc_mallopt (M_TRIM_THRESHOLD, atoi (val)); > > +} > > + > > +static void > > +tunable_set_mallopt_mmap_threshold (const char *val) > > +{ > > + __libc_mallopt (M_MMAP_THRESHOLD, atoi (val)); > > +} > > + > > +static void > > +tunable_set_mallopt_check (const char *val) > > +{ > > + if (val[0]) > > + { > > + __libc_mallopt (M_CHECK_ACTION, (int) (val[0] - '0')); > > + if (check_action != 0) > > + __malloc_check_init (); > > + } > > +} > > + > > +#else > > /* Initialization routine. */ > > #include <string.h> > > extern char **_environ; > > @@ -366,6 +426,7 @@ next_env_entry (char ***position) > > > > return result; > > } > > +#endif > > > > > > #ifdef SHARED > > @@ -401,6 +462,29 @@ ptmalloc_init (void) > > > > thread_arena = &main_arena; > > thread_atfork (ptmalloc_lock_all, ptmalloc_unlock_all, ptmalloc_unlock_all2); > > + > > +#if BUILD_TUNABLES > > + COMPAT_TUNABLES_NAMESPACE_BEGIN (8); > > + > > + COMPAT_TUNABLE_REGISTER_SECURE (check, "MALLOC_CHECK_", > > + tunable_set_mallopt_check); > > + COMPAT_TUNABLE_REGISTER (top_pad, "MALLOC_TOP_PAD_", > > + tunable_set_mallopt_top_pad); > > + COMPAT_TUNABLE_REGISTER (perturb, "MALLOC_PERTURB_", > > + tunable_set_mallopt_perturb); > > + COMPAT_TUNABLE_REGISTER (mmap_threshold, "MALLOC_MMAP_THRESHOLD_", > > + tunable_set_mallopt_mmap_threshold); > > + COMPAT_TUNABLE_REGISTER (trim_threshold, "MALLOC_TRIM_THRESHOLD_", > > + tunable_set_mallopt_trim_threshold); > > + COMPAT_TUNABLE_REGISTER (mmap_max, "MALLOC_MMAP_MAX_", > > + tunable_set_mallopt_mmap_max); > > + COMPAT_TUNABLE_REGISTER (arena_max, "MALLOC_ARENA_MAX", > > + tunable_set_mallopt_arena_max); > > + COMPAT_TUNABLE_REGISTER (arena_test, "MALLOC_ARENA_TEST", > > + tunable_set_mallopt_arena_test); > > + > > + COMPAT_TUNABLES_NAMESPACE_INIT (); > > +#else > > const char *s = NULL; > > if (__glibc_likely (_environ != NULL)) > > { > > @@ -469,6 +553,8 @@ ptmalloc_init (void) > > if (check_action != 0) > > __malloc_check_init (); > > } > > +#endif > > + > > void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook); > > if (hook != NULL) > > (*hook)(); > > diff --git a/manual/install.texi b/manual/install.texi > > index de9d270..95392ea 100644 > > --- a/manual/install.texi > > +++ b/manual/install.texi > > @@ -195,6 +195,11 @@ configure with @option{--disable-werror}. > > By default for x86_64, @theglibc{} is built with vector math library. > > Use this option to disable vector math library. > > > > +@item --enable-tunables > > +Tunables support allows additional library parameters to be customized at > > +runtime for each application. This is an experimental feature and affects > > +startup time and is thus disabled by default. > > + > > @item --build=@var{build-system} > > @itemx --host=@var{host-system} > > These options are for cross-compiling. If you specify both options and > > diff --git a/nptl/nptl-init.c b/nptl/nptl-init.c > > index bdbdfed..6667801 100644 > > --- a/nptl/nptl-init.c > > +++ b/nptl/nptl-init.c > > @@ -38,6 +38,9 @@ > > #include <kernel-features.h> > > #include <libc-internal.h> > > #include <pthread-pids.h> > > +#if BUILD_TUNABLES > > +# include <tunables/tunables.h> > > +#endif > > > > #ifndef TLS_MULTIPLE_THREADS_IN_TCB > > /* Pointer to the corresponding variable in libc. */ > > @@ -297,7 +300,7 @@ extern void **__libc_dl_error_tsd (void) __attribute__ ((const)); > > static bool __nptl_initial_report_events __attribute_used__; > > > > void > > -__pthread_initialize_minimal_internal (void) > > +__pthread_initialize_minimal_internal (int argc, char **argv, char **envp) > > { > > #ifndef SHARED > > /* Unlike in the dynamically linked case the dynamic linker has not > > @@ -311,6 +314,10 @@ __pthread_initialize_minimal_internal (void) > > __asm __volatile (""); > > #endif > > > > +#if BUILD_TUNABLES > > + __tunables_init (envp); > > +#endif > > + > > /* Minimal initialization of the thread descriptor. */ > > struct pthread *pd = THREAD_SELF; > > __pthread_initialize_pids (pd); > > diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk > > new file mode 100644 > > index 0000000..ccbf789 > > --- /dev/null > > +++ b/scripts/gen-tunables.awk > > @@ -0,0 +1,84 @@ > > +# Generate tunable-list.h from tunables.list > > + > > +BEGIN { > > + ns="" > > + top_ns="" > > +} > > + > > +$2 == "{" { > > + if (top_ns == "") { > > + top_ns = $1 > > + } > > + else if (ns == "") { > > + ns = $1 > > + count = 0 > > + } > > + else { > > + printf ("Unexpected occurrence of '{' inside a namespace: %s:%d\n", > > + FILENAME, FNR) > > + exit 1 > > + } > > + > > + next > > +} > > + > > +$1 == "}" { > > + if (ns != "") { > > + ns = "" > > + } > > + else if (top_ns != "") { > > + top_ns = "" > > + } > > + else { > > + printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR) > > + exit 1 > > + } > > + next > > +} > > + > > +{ > > + if (ns == "") { > > + print "Invalid tunable outside a namespace" > > + exit 1 > > + } > > + val[top_ns][ns][count] = $1 > > + count = count + 1 > > +} > > + > > +END { > > + if (ns != "") { > > + print "Unterminated namespace. Is a closing brace missing?" > > + exit 1 > > + } > > + > > + print "/* Print a full tunable enum name. */" > > + print "#include <stddef.h>" > > + print "#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)" > > + print "#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id\n" > > + > > + print "/* Full name for a tunable is top_ns.tunable_ns.id. */" > > + print "#define TUNABLE_NAME_S(top,ns,id) #top \".\" #ns \".\" #id\n" > > + > > + print "typedef enum" > > + print "{" > > + for (t in val) { > > + for (n in val[t]) { > > + for (c in val[t][n]) { > > + printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, val[t][n][c]); > > + } > > + } > > + } > > + print "} tunable_id_t;\n" > > + print "#ifdef TUNABLES_INTERNAL" > > + print "static tunable_t tunable_list[] = {" > > + for (t in val) { > > + for (n in val[t]) { > > + for (c in val[t][n]) { > > + printf (" {TUNABLE_NAME_S(%s, %s, %s), NULL, NULL, false},\n", > > + t, n, val[t][n][c], t, n, val[t][n][c]); > > + } > > + } > > + } > > + print "};" > > + print "#endif" > > +} > > diff --git a/tunables/Makefile b/tunables/Makefile > > new file mode 100644 > > index 0000000..44439f6 > > --- /dev/null > > +++ b/tunables/Makefile > > @@ -0,0 +1,33 @@ > > +# Copyright (C) 2016 Free Software Foundation, Inc. > > +# This file is part of the GNU C Library. > > + > > +# The GNU C Library is free software; you can redistribute it and/or > > +# modify it under the terms of the GNU Lesser General Public > > +# License as published by the Free Software Foundation; either > > +# version 2.1 of the License, or (at your option) any later version. > > + > > +# The GNU C Library is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > +# Lesser General Public License for more details. > > + > > +# You should have received a copy of the GNU Lesser General Public > > +# License along with the GNU C Library; if not, see > > +# <http://www.gnu.org/licenses/>. > > + > > +# > > +# Makefile for tunables. > > +# > > +subdir := tunables > > + > > +include ../Makeconfig > > + > > +routines = tunables > > + > > +$(objpfx)tunables.os: tunable-list.h > > + > > +tunable-list.h: $(..)scripts/gen-tunables.awk tunables.list > > + $(AWK) -f $^ > $@.tmp > > + mv $@{.tmp,} > > + > > +include ../Rules > > diff --git a/tunables/README b/tunables/README > > new file mode 100644 > > index 0000000..02a3f7a > > --- /dev/null > > +++ b/tunables/README > > @@ -0,0 +1,93 @@ > > + TUNABLE FRAMEWORK > > + ================= > > + > > +The tunable framework allows modules within glibc to register variables that > > +may be tweaked through an environment variable or an API call. It aims to > > +enforce a strict namespace rule to bring consistency to naming of these tunable > > +environment variables across the project. > > + > > +ADDING A NEW TUNABLE > > +-------------------- > > + > > +The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in > > +distributions for specific tunables if they want to add their own tunables. > > +Downstream implementations are discouraged from using the 'glibc' namespace for > > +tunables they don't already have consensus to push upstream. > > + > > +There are two steps to adding a tunable: > > + > > +1. Add a tunable ID: > > + > > +Modules that wish to use the tunables interface must define the > > +TUNABLE_NAMESPACE macro. Following this, for each tunable you want to > > +register, call the TUNABLE_REGISTER macro. > > + > > +The TUNABLE_REGISTER macro takes the following arguments: > > + > > +- id: The short name of the tunable. It will be concatenated with > > + the TOP_NAMESPACE and TUNABLE_NAMESPACE to build the full ID of > > + the tunable. > > + > > +- setter: A function that accepts a string input and initializes the > > + tunable variable. The prototype of the function should be as > > + follows: > > + > > + void setter_func (const char *) > > + > > +2. Add to tunables.list: > > + > > +This file nests tunable ids inside their tunable namespace within curly braces > > +and multiple such namespaces are in turn nested inside their top namespaces in > > +a similar manner. The malloc and resolv snippets for example could look like > > +this: > > + > > +glibc { > > + malloc { > > + check > > + top_pad > > + } > > + resolv { > > + secure_dns > > + } > > +} > > + > > +Add your tunable in this hierarchy using the proper nesting for your namespace > > +and top namespace. If the top namespace is not glibc, create a new snippet > > +with its own hierarchy, separate from the glibc hierarchy. > > + > > +ADDING A LEGACY TUNABLE > > +----------------------- > > + > > +One may add tunables for variables that are currently controlled by users with > > +environment variables. Multiple such variables should be defined together for > > +better performance by using the COMPAT_TUNABLES_NAMESPACE_BEGIN macro, which > > +accepts a numeric parameter with the number of tunables you intend to register > > +in the block. > > + > > +Individual tunables are then registered using the COMPAT_TUNABLE_REGISTER and > > +COMPAT_TUNABLE_REGISTER_SECURE macros. They take the following parameters: > > + > > +- id: The tunable ID, similar to the normal way of adding a tunable > > + > > +- env: The name of the legacy environment variable so that one may > > + read its value and use it. > > + > > +- setter: The setter function, similar to the normal way of adding a > > + tunable. > > + > > +COMPAT_TUNABLE_REGISTER reads the environment variable only for non-setuid > > +binaries, whereas COMPAT_TUNABLE_REGISTER_SECURE always attempts to read the > > +environment variable. Use COMPAT_TUNABLE_REGISTER_SECURE with care. > > + > > +You will also have to add the tunable id to tunables/tunables.list. > > + > > +DO NOT ADD NEW TUNABLES USING THIS MECHANISM! > > + > > +FUTURE WORK > > +----------- > > + > > +The framework currently only allows a one-time initialization of variables > > +through environment variables and in some cases, modification of variables via > > +an API call. A future goal for this project is to allow tweaking of some > > +values in a running process, possibly through some kind of shared memory > > +mechanism. > > diff --git a/tunables/Versions b/tunables/Versions > > new file mode 100644 > > index 0000000..4c68b0e > > --- /dev/null > > +++ b/tunables/Versions > > @@ -0,0 +1,8 @@ > > +# Exports from tunables should only be in the GLIBC_PRIVATE namespace. > > +libc { > > + GLIBC_PRIVATE { > > + __compat_tunables_init_envvars; > > + __tunables_init; > > + __tunable_register; > > + } > > +} > > diff --git a/tunables/tunable-list.h b/tunables/tunable-list.h > > new file mode 100644 > > index 0000000..2b1caa4 > > --- /dev/null > > +++ b/tunables/tunable-list.h > > @@ -0,0 +1,32 @@ > > +/* Print a full tunable enum name. */ > > +#include <stddef.h> > > +#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id) > > +#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id > > + > > +/* Full name for a tunable is top_ns.tunable_ns.id. */ > > +#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id > > + > > +typedef enum > > +{ > > + TUNABLE_ENUM_NAME(glibc, malloc, check), > > + TUNABLE_ENUM_NAME(glibc, malloc, top_pad), > > + TUNABLE_ENUM_NAME(glibc, malloc, perturb), > > + TUNABLE_ENUM_NAME(glibc, malloc, mmap_threshold), > > + TUNABLE_ENUM_NAME(glibc, malloc, trim_threshold), > > + TUNABLE_ENUM_NAME(glibc, malloc, mmap_max), > > + TUNABLE_ENUM_NAME(glibc, malloc, arena_max), > > + TUNABLE_ENUM_NAME(glibc, malloc, arena_test), > > +} tunable_id_t; > > + > > +#ifdef TUNABLES_INTERNAL > > +static tunable_t tunable_list[] = { > > + {TUNABLE_NAME_S(glibc, malloc, check), NULL, NULL, false}, > > + {TUNABLE_NAME_S(glibc, malloc, top_pad), NULL, NULL, false}, > > + {TUNABLE_NAME_S(glibc, malloc, perturb), NULL, NULL, false}, > > + {TUNABLE_NAME_S(glibc, malloc, mmap_threshold), NULL, NULL, false}, > > + {TUNABLE_NAME_S(glibc, malloc, trim_threshold), NULL, NULL, false}, > > + {TUNABLE_NAME_S(glibc, malloc, mmap_max), NULL, NULL, false}, > > + {TUNABLE_NAME_S(glibc, malloc, arena_max), NULL, NULL, false}, > > + {TUNABLE_NAME_S(glibc, malloc, arena_test), NULL, NULL, false}, > > +}; > > +#endif > > diff --git a/tunables/tunables.c b/tunables/tunables.c > > new file mode 100644 > > index 0000000..8834aff > > --- /dev/null > > +++ b/tunables/tunables.c > > @@ -0,0 +1,163 @@ > > +/* The tunable framework. See the README to know how to use the tunable in > > + a glibc module. > > + > > + Copyright (C) 2016 Free Software Foundation, Inc. > > + This file is part of the GNU C Library. > > + > > + The GNU C Library is free software; you can redistribute it and/or > > + modify it under the terms of the GNU Lesser General Public > > + License as published by the Free Software Foundation; either > > + version 2.1 of the License, or (at your option) any later version. > > + > > + The GNU C Library is distributed in the hope that it will be useful, > > + but WITHOUT ANY WARRANTY; without even the implied warranty of > > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > + Lesser General Public License for more details. > > + > > + You should have received a copy of the GNU Lesser General Public > > + License along with the GNU C Library; if not, see > > + <http://www.gnu.org/licenses/>. */ > > + > > +#include <stddef.h> > > +#include <stdlib.h> > > +#include <string.h> > > +#include <stdbool.h> > > +#include <unistd.h> > > +#include <sys/param.h> > > + > > +extern char **__environ; > > + > > +#define TUNABLES_INTERNAL 1 > > +#include "tunables.h" > > + > > +/* We avoid calling into the C library as much as we can, especially functions > > + that we know could use tunables in future for some reason or the other. For > > + example, we call mmap directly instead of malloc because malloc uses > > + tunables. Similarly, the string functions may use tunables in future to > > + select optimal routines and we want to make sure we stay out of the way and > > + not invoke their initializers before the tunable list has been > > + initialized. */ > > + > > +static void > > +t_memcpy (char *dest, const char *src, size_t len) > > +{ > > + for (size_t i = 0; i < len; i++) > > + dest[i] = src[i]; > > +} > > + > > +static int > > +t_strcmp (const char *a, const char *b) > > +{ > > + size_t i = 0; > > + > > + for (i = 0; *a != '\0' && *b != '\0'; i++, a++, b++) > > + if (*a != *b) > > + return *a - *b; > > + > > + return *a - *b; > > +} > > + > > +static bool > > +get_next_env (char ***envp, char **name, size_t *namelen, char **val) > > +{ > > + char **ev = *envp; > > + > > + while (ev != NULL && *ev != '\0') > > + { > > + char *envline = *ev; > > + int len = 0; > > + > > + while (envline[len] != '\0' && envline[len] != '=') > > + len++; > > + > > + /* Just the name and no value, go to the next one. */ > > + if (envline[len] == '\0') > > + continue; > > + > > + *name = envline; > > + *namelen = len; > > + *val = &envline[len + 1]; > > + *envp = ++ev; > > + > > + return true; > > + } > > + > > + return false; > > +} > > + > > +/* Initialize tunables from the GLIBC_TUNABLES environment variable. The > > + variable is set as colon separated name=value pairs. This routine may be > > + called from the constructor of either libpthread.so or lic.so. Constructor > > + calls for libraries are protected by the dl_load_lock, so this routine won't > > + be called concurrently from diferent threads. */ > > +void > > +__tunables_init (char **envp) > > +{ > > + /* Empty for now. */ > > +} > > +libc_hidden_def (__tunables_init) > > + > > +/* Initialize all tunables using its legacy environment variable values whose > > + names are passed in ENVVARS. */ > > +void > > +__compat_tunables_init_envvars (struct compat_tunable_env *envvars, int count) > > +{ > > + /* Traverse through the environment to find environment variables we may need > > + to set. */ > > + char **envp = __environ; > > + char *envname = NULL; > > + char *envval = NULL; > > + size_t len = 0; > > + > > + while (get_next_env (&envp, &envname, &len, &envval)) > > + { > > + int init_count = 0; > > + for (int i = 0; i < count; i++) > > + { > > + tunable_id_t t = envvars[i].id; > > + tunable_t *cur = &tunable_list[t]; > > + > > + /* Skip over tunables that have already been initialized. */ > > + if (cur->initialized) > > + { > > + init_count++; > > + continue; > > + } > > + > > + const char *name = envvars[i].env; > > + char *ename = alloca (len + 1); > > + > > + t_memcpy (ename, envname, len); > > + ename[len] = '\0'; > > + > > + /* We have a match. Initialize and move on to the next line. */ > > + if (t_strcmp (ename, name) == 0) > > + { > > + cur->val = envval; > > + cur->set (cur->val); > > + cur->initialized = true; > > + break; > > + } > > + } > > + > > + /* All of the tunable envvars have been initialized. */ > > + if (count == init_count) > > + break; > > + } > > +} > > +libc_hidden_def (__compat_tunables_init_envvars) > > + > > +/* Initialize a tunable and set its value. */ > > +void > > +__tunable_register (tunable_id_t id, tunable_setter_t set_func) > > +{ > > + tunable_t *cur = &tunable_list[id]; > > + > > + cur->set = set_func; > > + if (cur->val != NULL) > > + { > > + set_func (cur->val); > > + cur->initialized = true; > > + } > > +} > > +libc_hidden_def (__tunable_register) > > diff --git a/tunables/tunables.h b/tunables/tunables.h > > new file mode 100644 > > index 0000000..6aef30e > > --- /dev/null > > +++ b/tunables/tunables.h > > @@ -0,0 +1,110 @@ > > +/* The tunable framework. See the README to know how to use the tunable in > > + a glibc module. > > + > > + Copyright (C) 2016 Free Software Foundation, Inc. > > + This file is part of the GNU C Library. > > + > > + The GNU C Library is free software; you can redistribute it and/or > > + modify it under the terms of the GNU Lesser General Public > > + License as published by the Free Software Foundation; either > > + version 2.1 of the License, or (at your option) any later version. > > + > > + The GNU C Library is distributed in the hope that it will be useful, > > + but WITHOUT ANY WARRANTY; without even the implied warranty of > > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > + Lesser General Public License for more details. > > + > > + You should have received a copy of the GNU Lesser General Public > > + License along with the GNU C Library; if not, see > > + <http://www.gnu.org/licenses/>. */ > > + > > +#include <stdbool.h> > > + > > +typedef void (*tunable_setter_t) (const char *); > > + > > +/* A tunable. */ > > +struct _tunable > > +{ > > + const char *name; > > + const char *val; > > + tunable_setter_t set; > > + bool initialized; > > +}; > > + > > +typedef struct _tunable tunable_t; > > + > > +#include "tunable-list.h" > > + > > +struct compat_tunable_env > > +{ > > + tunable_id_t id; > > + const char *env; > > +}; > > + > > + > > +extern void __compat_tunables_init_envvars (struct compat_tunable_env *, int); > > +extern void __tunable_register (tunable_id_t, tunable_setter_t); > > +extern void __tunables_init (char **); > > + > > +/* Avoid a PLT reference when called from within libc.so. */ > > +libc_hidden_proto (__compat_tunables_init_envvars) > > +libc_hidden_proto (__tunable_register) > > +libc_hidden_proto (__tunables_init) > > + > > +/* Register a tunable ID in namespace NS inside the glibc top namespace with > > + SET as the setter function. */ > > +#define TUNABLE_REGISTER(ns,id,set) \ > > + TUNABLE_REGISTER_FULL (TOP_NAMESPACE, ns, id, set) > > + > > +/* Register a tunable ID in namespace NS inside the TNS top namespace with > > + SET as the setter function. */ > > +#define TUNABLE_REGISTER_FULL(tns,ns,id,set) \ > > + __tunable_register (TUNABLE_ENUM_NAME (tns, ns, id), (set)) > > + > > + > > + > > +/* COMPATIBILITY INTERFACE. These macros should be used to define tunables for > > + variables that are currently being controlled using environment variables. > > + New tunables must not use this interface. */ > > + > > +#define _ADD_COMPAT_TUNABLE_ENV(__id,__env) \ > > +({ \ > > + envvars[envvars_cnt].id = TUNABLE_ENUM_NAME (TOP_NAMESPACE, \ > > + TUNABLE_NAMESPACE, __id); \ > > + envvars[envvars_cnt++].env = (__env); \ > > +}) > > + > > +/* Start registering tunables in the current namespace. */ > > +#define COMPAT_TUNABLES_NAMESPACE_BEGIN(size) \ > > + { \ > > + struct compat_tunable_env envvars[size]; \ > > + int envvars_cnt = 0; > > + > > +/* Register a tunable. This macro validates that the call is OK and then calls > > + tunable_init to do the real work of adding the tunable and setting its value > > + based on its environment variable(s). */ > > +#define COMPAT_TUNABLE_REGISTER(id,env,set) \ > > +({ \ > > + assert (envvars_cnt < (sizeof (envvars) \ > > + / sizeof (struct compat_tunable_env))); \ > > + if (!__libc_enable_secure) \ > > + _ADD_COMPAT_TUNABLE_ENV (id, env); \ > > + TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \ > > +}) > > + > > +/* Does exactly the same thing as TUNABLE_REGISTER, except that it allows the > > + tunable to look for environment variable values even for setuid binaries. > > + This is a separate macro and not just another parameter in TUNABLE_REGISTER > > + to avoid accidentally setting a secure flag where it is not required. */ > > +#define COMPAT_TUNABLE_REGISTER_SECURE(id,env,set) \ > > +({ \ > > + assert (envvars_cnt < (sizeof (envvars) \ > > + / sizeof (struct compat_tunable_env))); \ > > + _ADD_COMPAT_TUNABLE_ENV(id, env); \ > > + TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \ > > +}) > > + > > +/* Initialize tunables in the namespace. */ > > +#define COMPAT_TUNABLES_NAMESPACE_INIT() \ > > + __compat_tunables_init_envvars (envvars, envvars_cnt); \ > > + } > > diff --git a/tunables/tunables.list b/tunables/tunables.list > > new file mode 100644 > > index 0000000..f1335cc > > --- /dev/null > > +++ b/tunables/tunables.list > > @@ -0,0 +1,12 @@ > > +glibc { > > + malloc { > > + check > > + top_pad > > + perturb > > + mmap_threshold > > + trim_threshold > > + mmap_max > > + arena_max > > + arena_test > > + } > > +} > > -- > > 2.5.0 > >
Ping? Restating my interest in the patch. Would it be more palatable if the malloc changes are backed out into a separate patch? This is a critical foot in the door for those wanting to use TLE effectively on ppc. On 02/01/2016 03:40 AM, Siddhesh Poyarekar wrote: > Ping! > > On Mon, Jan 25, 2016 at 08:55:06AM +0530, Siddhesh Poyarekar wrote: >> Ping!
Siddhesh Poyarekar <sid@reserved-bit.com> writes: > The tunables framework allows us to uniformly manage and expose global > variables inside glibc as switches to users. tunables/README has > instructions for glibc developers to add new tunables. > > Tunables support can be enabled by passing the --enable-tunables > configure flag to the configure script. This patch only adds a > framework and does not pose any limitations on how tunable values are > read from the user. It also adds environment variables used in malloc > behaviour tweaking to the tunables framework as a PoC of the > compatibility interface. LGTM, but I have some cosmetic comments. I tested this on powerpc, powerpc64 and powerpc64le. > diff --git a/tunables/README b/tunables/README > new file mode 100644 > index 0000000..02a3f7a > --- /dev/null > +++ b/tunables/README > @@ -0,0 +1,93 @@ > + TUNABLE FRAMEWORK > + ================= > + > +The tunable framework allows modules within glibc to register variables that Tunables instead of tunable? Both in the subject and in the previous line. > diff --git a/tunables/tunables.c b/tunables/tunables.c > new file mode 100644 > index 0000000..8834aff > --- /dev/null > +++ b/tunables/tunables.c > @@ -0,0 +1,163 @@ > ... > + > +/* Initialize tunables from the GLIBC_TUNABLES environment variable. The > + variable is set as colon separated name=value pairs. This routine may be > + called from the constructor of either libpthread.so or lic.so. Constructor s/lic.so/libc.so/ > diff --git a/tunables/tunables.h b/tunables/tunables.h > new file mode 100644 > index 0000000..6aef30e > --- /dev/null > +++ b/tunables/tunables.h > @@ -0,0 +1,110 @@ > ... > +/* Register a tunable ID in namespace NS inside the glibc top namespace with > + SET as the setter function. */ > +#define TUNABLE_REGISTER(ns,id,set) \ > + TUNABLE_REGISTER_FULL (TOP_NAMESPACE, ns, id, set) > + > +/* Register a tunable ID in namespace NS inside the TNS top namespace with > + SET as the setter function. */ > +#define TUNABLE_REGISTER_FULL(tns,ns,id,set) \ > + __tunable_register (TUNABLE_ENUM_NAME (tns, ns, id), (set)) > + > + ^ Is this character intentional?
On Wed, Feb 03, 2016 at 11:47:43AM -0200, Tulio Magno Quites Machado Filho wrote: > LGTM, but I have some cosmetic comments. > I tested this on powerpc, powerpc64 and powerpc64le. Thanks for your comments and the testing. > > > diff --git a/tunables/README b/tunables/README > > new file mode 100644 > > index 0000000..02a3f7a > > --- /dev/null > > +++ b/tunables/README > > @@ -0,0 +1,93 @@ > > + TUNABLE FRAMEWORK > > + ================= > > + > > +The tunable framework allows modules within glibc to register variables that > > Tunables instead of tunable? > Both in the subject and in the previous line. > > > diff --git a/tunables/tunables.c b/tunables/tunables.c > > new file mode 100644 > > index 0000000..8834aff > > --- /dev/null > > +++ b/tunables/tunables.c > > @@ -0,0 +1,163 @@ > > ... > > + > > +/* Initialize tunables from the GLIBC_TUNABLES environment variable. The > > + variable is set as colon separated name=value pairs. This routine may be > > + called from the constructor of either libpthread.so or lic.so. Constructor > > s/lic.so/libc.so/ > > > diff --git a/tunables/tunables.h b/tunables/tunables.h > > new file mode 100644 > > index 0000000..6aef30e > > --- /dev/null > > +++ b/tunables/tunables.h > > @@ -0,0 +1,110 @@ > > ... > > +/* Register a tunable ID in namespace NS inside the glibc top namespace with > > + SET as the setter function. */ > > +#define TUNABLE_REGISTER(ns,id,set) \ > > + TUNABLE_REGISTER_FULL (TOP_NAMESPACE, ns, id, set) > > + > > +/* Register a tunable ID in namespace NS inside the TNS top namespace with > > + SET as the setter function. */ > > +#define TUNABLE_REGISTER_FULL(tns,ns,id,set) \ > > + __tunable_register (TUNABLE_ENUM_NAME (tns, ns, id), (set)) > > + > > + > ^ > Is this character intentional? Yes, it is the page break character to separate the API calls from the internal macros below. It doesn't have any functional benefit, just a useful marker that we've used in other places in the code. Siddhesh
Ping! On Mon, Feb 01, 2016 at 03:10:32PM +0530, Siddhesh Poyarekar wrote: > Ping! > > On Mon, Jan 25, 2016 at 08:55:06AM +0530, Siddhesh Poyarekar wrote: > > Ping! > > > > On Sun, Jan 17, 2016 at 12:25:50AM +0530, Siddhesh Poyarekar wrote: > > > The tunables framework allows us to uniformly manage and expose global > > > variables inside glibc as switches to users. tunables/README has > > > instructions for glibc developers to add new tunables. > > > > > > Tunables support can be enabled by passing the --enable-tunables > > > configure flag to the configure script. This patch only adds a > > > framework and does not pose any limitations on how tunable values are > > > read from the user. It also adds environment variables used in malloc > > > behaviour tweaking to the tunables framework as a PoC of the > > > compatibility interface. > > > > > > * manual/install.texi: Add --enable-tunables option. > > > * INSTALL: Regenerate. > > > * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE. > > > (all-subdirs): Add tunables. > > > * config.h.in: Add BUILD_TUNABLES. > > > * config.make.in: Add build-tunables. > > > * configure.ac: Add --enable-tunables option. > > > * configure: Regenerate. > > > * csu/init-first.c: Include tunables.h. > > > (_init): Initialize tunables. > > > * nptl/nptl-init.c [BUILD_TUNABLES]: Include tunables.h. > > > (__pthread_initialize_minimal_internal): Accept argc, argv and > > > envp. Initialize tunables. > > > * csu/libc-start.c (LIBC_START_MAIN): Adjust. > > > * malloc/arena.c [BUILD_TUNABLES]: Include tunables.h. Define > > > TUNABLE_NAMESPACE. > > > (tunable_set_mallopt_top_pad, tunable_set_mallopt_perturb, > > > tunable_set_mallopt_mmap_max, tunable_set_mallopt_arena_max, > > > tunable_set_mallopt_arena_test, > > > tunable_set_mallopt_trim_threshold, > > > tunable_set_mallopt_mmap_threshold, > > > tunable_set_mallopt_check): New functions. > > > (ptmalloc_init): Use them and register tunables. > > > * scripts/gen-tunables.awk: New file. > > > * tunables/Makefile: New file. > > > * tunables/README: New file. > > > * tunables/Versions: New file. > > > * tunables/tunable-list.h: New auto-generated file. > > > * tunables/tunables.c: New file. > > > * tunables/tunables.h: New file. > > > * tunables/tunables.list: New file. > > > --- > > > INSTALL | 6 ++ > > > Makeconfig | 9 +++ > > > config.h.in | 3 + > > > config.make.in | 1 + > > > configure | 16 +++++ > > > configure.ac | 10 +++ > > > csu/init-first.c | 7 ++ > > > csu/libc-start.c | 4 +- > > > malloc/arena.c | 86 +++++++++++++++++++++++++ > > > manual/install.texi | 5 ++ > > > nptl/nptl-init.c | 9 ++- > > > scripts/gen-tunables.awk | 84 ++++++++++++++++++++++++ > > > tunables/Makefile | 33 ++++++++++ > > > tunables/README | 93 +++++++++++++++++++++++++++ > > > tunables/Versions | 8 +++ > > > tunables/tunable-list.h | 32 ++++++++++ > > > tunables/tunables.c | 163 +++++++++++++++++++++++++++++++++++++++++++++++ > > > tunables/tunables.h | 110 ++++++++++++++++++++++++++++++++ > > > tunables/tunables.list | 12 ++++ > > > 19 files changed, 688 insertions(+), 3 deletions(-) > > > create mode 100644 scripts/gen-tunables.awk > > > create mode 100644 tunables/Makefile > > > create mode 100644 tunables/README > > > create mode 100644 tunables/Versions > > > create mode 100644 tunables/tunable-list.h > > > create mode 100644 tunables/tunables.c > > > create mode 100644 tunables/tunables.h > > > create mode 100644 tunables/tunables.list > > > > > > diff --git a/INSTALL b/INSTALL > > > index c70ea9f..dd06e2e 100644 > > > --- a/INSTALL > > > +++ b/INSTALL > > > @@ -164,6 +164,12 @@ will be used, and CFLAGS sets optimization options for the compiler. > > > By default for x86_64, the GNU C Library is built with vector math > > > library. Use this option to disable vector math library. > > > > > > +'--enable-tunables' > > > + Tunables support allows additional library parameters to be > > > + customized at runtime for each application. This is an > > > + experimental feature and affects startup time and is thus disabled > > > + by default. > > > + > > > '--build=BUILD-SYSTEM' > > > '--host=HOST-SYSTEM' > > > These options are for cross-compiling. If you specify both options > > > diff --git a/Makeconfig b/Makeconfig > > > index 87a22e8..8ac199b 100644 > > > --- a/Makeconfig > > > +++ b/Makeconfig > > > @@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \ > > > $(foreach lib,$(libof-$(basename $(@F))) \ > > > $(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \ > > > $(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F))) > > > + > > > +ifeq (yes,$(build-tunables)) > > > +CPPFLAGS += -DTOP_NAMESPACE=glibc > > > +endif > > > + > > > override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \ > > > $(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \ > > > $(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \ > > > @@ -1098,6 +1103,10 @@ all-subdirs = csu assert ctype locale intl catgets math setjmp signal \ > > > crypt localedata timezone rt conform debug mathvec \ > > > $(add-on-subdirs) dlfcn elf > > > > > > +ifeq (yes,$(build-tunables)) > > > +all-subdirs += tunables > > > +endif > > > + > > > ifndef avoid-generated > > > # sysd-sorted itself will contain rules making the sysd-sorted target > > > # depend on Depend files. But if you just added a Depend file to an > > > diff --git a/config.h.in b/config.h.in > > > index ec9c8bc..9b07580 100644 > > > --- a/config.h.in > > > +++ b/config.h.in > > > @@ -231,4 +231,7 @@ > > > /* PowerPC32 uses fctidz for floating point to long long conversions. */ > > > #define HAVE_PPC_FCTIDZ 0 > > > > > > +/* Build glibc with tunables support. */ > > > +#define BUILD_TUNABLES 0 > > > + > > > #endif > > > diff --git a/config.make.in b/config.make.in > > > index 05ed6ec..b158f51 100644 > > > --- a/config.make.in > > > +++ b/config.make.in > > > @@ -90,6 +90,7 @@ use-nscd = @use_nscd@ > > > build-hardcoded-path-in-tests= @hardcoded_path_in_tests@ > > > build-pt-chown = @build_pt_chown@ > > > enable-lock-elision = @enable_lock_elision@ > > > +build-tunables = @build_tunables@ > > > > > > # Build tools. > > > CC = @CC@ > > > diff --git a/configure b/configure > > > index aa05d49..6fef02c 100755 > > > --- a/configure > > > +++ b/configure > > > @@ -659,6 +659,7 @@ multi_arch > > > base_machine > > > add_on_subdirs > > > add_ons > > > +build_tunables > > > build_pt_chown > > > build_nscd > > > link_obsolete_rpc > > > @@ -773,6 +774,7 @@ enable_systemtap > > > enable_build_nscd > > > enable_nscd > > > enable_pt_chown > > > +enable_tunables > > > enable_mathvec > > > with_cpu > > > ' > > > @@ -1440,6 +1442,7 @@ Optional Features: > > > --disable-build-nscd disable building and installing the nscd daemon > > > --disable-nscd library functions will not contact the nscd daemon > > > --enable-pt_chown Enable building and installing pt_chown > > > + --enable-tunables Enable tunables support > > > --enable-mathvec Enable building and installing mathvec [default > > > depends on architecture] > > > > > > @@ -3642,6 +3645,19 @@ if test "$build_pt_chown" = yes; then > > > > > > fi > > > > > > +# Check whether --enable-tunables was given. > > > +if test "${enable_tunables+set}" = set; then : > > > + enableval=$enable_tunables; build_tunables=$enableval > > > +else > > > + build_tunables=no > > > +fi > > > + > > > + > > > +if test "$build_tunables" = yes; then > > > + $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h > > > + > > > +fi > > > + > > > # The abi-tags file uses a fairly simplistic model for name recognition that > > > # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a > > > # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell. > > > diff --git a/configure.ac b/configure.ac > > > index ee7a3f1..41507ac 100644 > > > --- a/configure.ac > > > +++ b/configure.ac > > > @@ -392,6 +392,16 @@ if test "$build_pt_chown" = yes; then > > > AC_DEFINE(HAVE_PT_CHOWN) > > > fi > > > > > > +AC_ARG_ENABLE([tunables], > > > + [AS_HELP_STRING([--enable-tunables], > > > + [Enable tunables support])], > > > + [build_tunables=$enableval], > > > + [build_tunables=no]) > > > +AC_SUBST(build_tunables) > > > +if test "$build_tunables" = yes; then > > > + AC_DEFINE(BUILD_TUNABLES) > > > +fi > > > + > > > # The abi-tags file uses a fairly simplistic model for name recognition that > > > # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a > > > # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell. > > > diff --git a/csu/init-first.c b/csu/init-first.c > > > index 77c6e1c..a4e0c54 100644 > > > --- a/csu/init-first.c > > > +++ b/csu/init-first.c > > > @@ -26,6 +26,9 @@ > > > #include <sys/param.h> > > > #include <sys/types.h> > > > #include <libc-internal.h> > > > +#if BUILD_TUNABLES > > > +# include <tunables/tunables.h> > > > +#endif > > > > > > #include <ldsodefs.h> > > > > > > @@ -96,6 +99,10 @@ _init (int argc, char **argv, char **envp) > > > #if defined SHARED && !defined NO_CTORS_DTORS_SECTIONS > > > __libc_global_ctors (); > > > #endif > > > + > > > +#if BUILD_TUNABLES > > > + __tunables_init (__environ); > > > +#endif > > > } > > > > > > /* This function is defined here so that if this file ever gets into > > > diff --git a/csu/libc-start.c b/csu/libc-start.c > > > index f4aa01a..7629fec 100644 > > > --- a/csu/libc-start.c > > > +++ b/csu/libc-start.c > > > @@ -32,7 +32,7 @@ extern int __libc_multiple_libcs; > > > #include <tls.h> > > > #ifndef SHARED > > > # include <dl-osinfo.h> > > > -extern void __pthread_initialize_minimal (void); > > > +extern void __pthread_initialize_minimal (int, char **, char **); > > > # ifndef THREAD_SET_STACK_GUARD > > > /* Only exported for architectures that don't store the stack guard canary > > > in thread local area. */ > > > @@ -193,7 +193,7 @@ LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL), > > > /* Initialize the thread library at least a bit since the libgcc > > > functions are using thread functions if these are available and > > > we need to setup errno. */ > > > - __pthread_initialize_minimal (); > > > + __pthread_initialize_minimal (argc, argv, __environ); > > > > > > /* Set up the stack checker's canary. */ > > > uintptr_t stack_chk_guard = _dl_setup_stack_chk_guard (_dl_random); > > > diff --git a/malloc/arena.c b/malloc/arena.c > > > index 1edb4d4..f773b10 100644 > > > --- a/malloc/arena.c > > > +++ b/malloc/arena.c > > > @@ -19,6 +19,11 @@ > > > > > > #include <stdbool.h> > > > > > > +#if BUILD_TUNABLES > > > +# define TUNABLE_NAMESPACE malloc > > > +# include <tunables/tunables.h> > > > +#endif > > > + > > > /* Compile-time constants. */ > > > > > > #define HEAP_MIN_SIZE (32 * 1024) > > > @@ -332,6 +337,61 @@ ptmalloc_unlock_all2 (void) > > > # endif > > > #endif /* !NO_THREADS */ > > > > > > +#if BUILD_TUNABLES > > > +static void > > > +tunable_set_mallopt_top_pad (const char *val) > > > +{ > > > + __libc_mallopt (M_TOP_PAD, atoi (val)); > > > +} > > > + > > > +static void > > > +tunable_set_mallopt_perturb (const char *val) > > > +{ > > > + __libc_mallopt (M_PERTURB, atoi (val)); > > > +} > > > + > > > +static void > > > +tunable_set_mallopt_mmap_max (const char *val) > > > +{ > > > + __libc_mallopt (M_MMAP_MAX, atoi (val)); > > > +} > > > + > > > +static void > > > +tunable_set_mallopt_arena_max (const char *val) > > > +{ > > > + __libc_mallopt (M_ARENA_MAX, atoi (val)); > > > +} > > > + > > > +static void > > > +tunable_set_mallopt_arena_test (const char *val) > > > +{ > > > + __libc_mallopt (M_ARENA_TEST, atoi (val)); > > > +} > > > + > > > +static void > > > +tunable_set_mallopt_trim_threshold (const char *val) > > > +{ > > > + __libc_mallopt (M_TRIM_THRESHOLD, atoi (val)); > > > +} > > > + > > > +static void > > > +tunable_set_mallopt_mmap_threshold (const char *val) > > > +{ > > > + __libc_mallopt (M_MMAP_THRESHOLD, atoi (val)); > > > +} > > > + > > > +static void > > > +tunable_set_mallopt_check (const char *val) > > > +{ > > > + if (val[0]) > > > + { > > > + __libc_mallopt (M_CHECK_ACTION, (int) (val[0] - '0')); > > > + if (check_action != 0) > > > + __malloc_check_init (); > > > + } > > > +} > > > + > > > +#else > > > /* Initialization routine. */ > > > #include <string.h> > > > extern char **_environ; > > > @@ -366,6 +426,7 @@ next_env_entry (char ***position) > > > > > > return result; > > > } > > > +#endif > > > > > > > > > #ifdef SHARED > > > @@ -401,6 +462,29 @@ ptmalloc_init (void) > > > > > > thread_arena = &main_arena; > > > thread_atfork (ptmalloc_lock_all, ptmalloc_unlock_all, ptmalloc_unlock_all2); > > > + > > > +#if BUILD_TUNABLES > > > + COMPAT_TUNABLES_NAMESPACE_BEGIN (8); > > > + > > > + COMPAT_TUNABLE_REGISTER_SECURE (check, "MALLOC_CHECK_", > > > + tunable_set_mallopt_check); > > > + COMPAT_TUNABLE_REGISTER (top_pad, "MALLOC_TOP_PAD_", > > > + tunable_set_mallopt_top_pad); > > > + COMPAT_TUNABLE_REGISTER (perturb, "MALLOC_PERTURB_", > > > + tunable_set_mallopt_perturb); > > > + COMPAT_TUNABLE_REGISTER (mmap_threshold, "MALLOC_MMAP_THRESHOLD_", > > > + tunable_set_mallopt_mmap_threshold); > > > + COMPAT_TUNABLE_REGISTER (trim_threshold, "MALLOC_TRIM_THRESHOLD_", > > > + tunable_set_mallopt_trim_threshold); > > > + COMPAT_TUNABLE_REGISTER (mmap_max, "MALLOC_MMAP_MAX_", > > > + tunable_set_mallopt_mmap_max); > > > + COMPAT_TUNABLE_REGISTER (arena_max, "MALLOC_ARENA_MAX", > > > + tunable_set_mallopt_arena_max); > > > + COMPAT_TUNABLE_REGISTER (arena_test, "MALLOC_ARENA_TEST", > > > + tunable_set_mallopt_arena_test); > > > + > > > + COMPAT_TUNABLES_NAMESPACE_INIT (); > > > +#else > > > const char *s = NULL; > > > if (__glibc_likely (_environ != NULL)) > > > { > > > @@ -469,6 +553,8 @@ ptmalloc_init (void) > > > if (check_action != 0) > > > __malloc_check_init (); > > > } > > > +#endif > > > + > > > void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook); > > > if (hook != NULL) > > > (*hook)(); > > > diff --git a/manual/install.texi b/manual/install.texi > > > index de9d270..95392ea 100644 > > > --- a/manual/install.texi > > > +++ b/manual/install.texi > > > @@ -195,6 +195,11 @@ configure with @option{--disable-werror}. > > > By default for x86_64, @theglibc{} is built with vector math library. > > > Use this option to disable vector math library. > > > > > > +@item --enable-tunables > > > +Tunables support allows additional library parameters to be customized at > > > +runtime for each application. This is an experimental feature and affects > > > +startup time and is thus disabled by default. > > > + > > > @item --build=@var{build-system} > > > @itemx --host=@var{host-system} > > > These options are for cross-compiling. If you specify both options and > > > diff --git a/nptl/nptl-init.c b/nptl/nptl-init.c > > > index bdbdfed..6667801 100644 > > > --- a/nptl/nptl-init.c > > > +++ b/nptl/nptl-init.c > > > @@ -38,6 +38,9 @@ > > > #include <kernel-features.h> > > > #include <libc-internal.h> > > > #include <pthread-pids.h> > > > +#if BUILD_TUNABLES > > > +# include <tunables/tunables.h> > > > +#endif > > > > > > #ifndef TLS_MULTIPLE_THREADS_IN_TCB > > > /* Pointer to the corresponding variable in libc. */ > > > @@ -297,7 +300,7 @@ extern void **__libc_dl_error_tsd (void) __attribute__ ((const)); > > > static bool __nptl_initial_report_events __attribute_used__; > > > > > > void > > > -__pthread_initialize_minimal_internal (void) > > > +__pthread_initialize_minimal_internal (int argc, char **argv, char **envp) > > > { > > > #ifndef SHARED > > > /* Unlike in the dynamically linked case the dynamic linker has not > > > @@ -311,6 +314,10 @@ __pthread_initialize_minimal_internal (void) > > > __asm __volatile (""); > > > #endif > > > > > > +#if BUILD_TUNABLES > > > + __tunables_init (envp); > > > +#endif > > > + > > > /* Minimal initialization of the thread descriptor. */ > > > struct pthread *pd = THREAD_SELF; > > > __pthread_initialize_pids (pd); > > > diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk > > > new file mode 100644 > > > index 0000000..ccbf789 > > > --- /dev/null > > > +++ b/scripts/gen-tunables.awk > > > @@ -0,0 +1,84 @@ > > > +# Generate tunable-list.h from tunables.list > > > + > > > +BEGIN { > > > + ns="" > > > + top_ns="" > > > +} > > > + > > > +$2 == "{" { > > > + if (top_ns == "") { > > > + top_ns = $1 > > > + } > > > + else if (ns == "") { > > > + ns = $1 > > > + count = 0 > > > + } > > > + else { > > > + printf ("Unexpected occurrence of '{' inside a namespace: %s:%d\n", > > > + FILENAME, FNR) > > > + exit 1 > > > + } > > > + > > > + next > > > +} > > > + > > > +$1 == "}" { > > > + if (ns != "") { > > > + ns = "" > > > + } > > > + else if (top_ns != "") { > > > + top_ns = "" > > > + } > > > + else { > > > + printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR) > > > + exit 1 > > > + } > > > + next > > > +} > > > + > > > +{ > > > + if (ns == "") { > > > + print "Invalid tunable outside a namespace" > > > + exit 1 > > > + } > > > + val[top_ns][ns][count] = $1 > > > + count = count + 1 > > > +} > > > + > > > +END { > > > + if (ns != "") { > > > + print "Unterminated namespace. Is a closing brace missing?" > > > + exit 1 > > > + } > > > + > > > + print "/* Print a full tunable enum name. */" > > > + print "#include <stddef.h>" > > > + print "#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)" > > > + print "#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id\n" > > > + > > > + print "/* Full name for a tunable is top_ns.tunable_ns.id. */" > > > + print "#define TUNABLE_NAME_S(top,ns,id) #top \".\" #ns \".\" #id\n" > > > + > > > + print "typedef enum" > > > + print "{" > > > + for (t in val) { > > > + for (n in val[t]) { > > > + for (c in val[t][n]) { > > > + printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, val[t][n][c]); > > > + } > > > + } > > > + } > > > + print "} tunable_id_t;\n" > > > + print "#ifdef TUNABLES_INTERNAL" > > > + print "static tunable_t tunable_list[] = {" > > > + for (t in val) { > > > + for (n in val[t]) { > > > + for (c in val[t][n]) { > > > + printf (" {TUNABLE_NAME_S(%s, %s, %s), NULL, NULL, false},\n", > > > + t, n, val[t][n][c], t, n, val[t][n][c]); > > > + } > > > + } > > > + } > > > + print "};" > > > + print "#endif" > > > +} > > > diff --git a/tunables/Makefile b/tunables/Makefile > > > new file mode 100644 > > > index 0000000..44439f6 > > > --- /dev/null > > > +++ b/tunables/Makefile > > > @@ -0,0 +1,33 @@ > > > +# Copyright (C) 2016 Free Software Foundation, Inc. > > > +# This file is part of the GNU C Library. > > > + > > > +# The GNU C Library is free software; you can redistribute it and/or > > > +# modify it under the terms of the GNU Lesser General Public > > > +# License as published by the Free Software Foundation; either > > > +# version 2.1 of the License, or (at your option) any later version. > > > + > > > +# The GNU C Library is distributed in the hope that it will be useful, > > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > > +# Lesser General Public License for more details. > > > + > > > +# You should have received a copy of the GNU Lesser General Public > > > +# License along with the GNU C Library; if not, see > > > +# <http://www.gnu.org/licenses/>. > > > + > > > +# > > > +# Makefile for tunables. > > > +# > > > +subdir := tunables > > > + > > > +include ../Makeconfig > > > + > > > +routines = tunables > > > + > > > +$(objpfx)tunables.os: tunable-list.h > > > + > > > +tunable-list.h: $(..)scripts/gen-tunables.awk tunables.list > > > + $(AWK) -f $^ > $@.tmp > > > + mv $@{.tmp,} > > > + > > > +include ../Rules > > > diff --git a/tunables/README b/tunables/README > > > new file mode 100644 > > > index 0000000..02a3f7a > > > --- /dev/null > > > +++ b/tunables/README > > > @@ -0,0 +1,93 @@ > > > + TUNABLE FRAMEWORK > > > + ================= > > > + > > > +The tunable framework allows modules within glibc to register variables that > > > +may be tweaked through an environment variable or an API call. It aims to > > > +enforce a strict namespace rule to bring consistency to naming of these tunable > > > +environment variables across the project. > > > + > > > +ADDING A NEW TUNABLE > > > +-------------------- > > > + > > > +The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in > > > +distributions for specific tunables if they want to add their own tunables. > > > +Downstream implementations are discouraged from using the 'glibc' namespace for > > > +tunables they don't already have consensus to push upstream. > > > + > > > +There are two steps to adding a tunable: > > > + > > > +1. Add a tunable ID: > > > + > > > +Modules that wish to use the tunables interface must define the > > > +TUNABLE_NAMESPACE macro. Following this, for each tunable you want to > > > +register, call the TUNABLE_REGISTER macro. > > > + > > > +The TUNABLE_REGISTER macro takes the following arguments: > > > + > > > +- id: The short name of the tunable. It will be concatenated with > > > + the TOP_NAMESPACE and TUNABLE_NAMESPACE to build the full ID of > > > + the tunable. > > > + > > > +- setter: A function that accepts a string input and initializes the > > > + tunable variable. The prototype of the function should be as > > > + follows: > > > + > > > + void setter_func (const char *) > > > + > > > +2. Add to tunables.list: > > > + > > > +This file nests tunable ids inside their tunable namespace within curly braces > > > +and multiple such namespaces are in turn nested inside their top namespaces in > > > +a similar manner. The malloc and resolv snippets for example could look like > > > +this: > > > + > > > +glibc { > > > + malloc { > > > + check > > > + top_pad > > > + } > > > + resolv { > > > + secure_dns > > > + } > > > +} > > > + > > > +Add your tunable in this hierarchy using the proper nesting for your namespace > > > +and top namespace. If the top namespace is not glibc, create a new snippet > > > +with its own hierarchy, separate from the glibc hierarchy. > > > + > > > +ADDING A LEGACY TUNABLE > > > +----------------------- > > > + > > > +One may add tunables for variables that are currently controlled by users with > > > +environment variables. Multiple such variables should be defined together for > > > +better performance by using the COMPAT_TUNABLES_NAMESPACE_BEGIN macro, which > > > +accepts a numeric parameter with the number of tunables you intend to register > > > +in the block. > > > + > > > +Individual tunables are then registered using the COMPAT_TUNABLE_REGISTER and > > > +COMPAT_TUNABLE_REGISTER_SECURE macros. They take the following parameters: > > > + > > > +- id: The tunable ID, similar to the normal way of adding a tunable > > > + > > > +- env: The name of the legacy environment variable so that one may > > > + read its value and use it. > > > + > > > +- setter: The setter function, similar to the normal way of adding a > > > + tunable. > > > + > > > +COMPAT_TUNABLE_REGISTER reads the environment variable only for non-setuid > > > +binaries, whereas COMPAT_TUNABLE_REGISTER_SECURE always attempts to read the > > > +environment variable. Use COMPAT_TUNABLE_REGISTER_SECURE with care. > > > + > > > +You will also have to add the tunable id to tunables/tunables.list. > > > + > > > +DO NOT ADD NEW TUNABLES USING THIS MECHANISM! > > > + > > > +FUTURE WORK > > > +----------- > > > + > > > +The framework currently only allows a one-time initialization of variables > > > +through environment variables and in some cases, modification of variables via > > > +an API call. A future goal for this project is to allow tweaking of some > > > +values in a running process, possibly through some kind of shared memory > > > +mechanism. > > > diff --git a/tunables/Versions b/tunables/Versions > > > new file mode 100644 > > > index 0000000..4c68b0e > > > --- /dev/null > > > +++ b/tunables/Versions > > > @@ -0,0 +1,8 @@ > > > +# Exports from tunables should only be in the GLIBC_PRIVATE namespace. > > > +libc { > > > + GLIBC_PRIVATE { > > > + __compat_tunables_init_envvars; > > > + __tunables_init; > > > + __tunable_register; > > > + } > > > +} > > > diff --git a/tunables/tunable-list.h b/tunables/tunable-list.h > > > new file mode 100644 > > > index 0000000..2b1caa4 > > > --- /dev/null > > > +++ b/tunables/tunable-list.h > > > @@ -0,0 +1,32 @@ > > > +/* Print a full tunable enum name. */ > > > +#include <stddef.h> > > > +#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id) > > > +#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id > > > + > > > +/* Full name for a tunable is top_ns.tunable_ns.id. */ > > > +#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id > > > + > > > +typedef enum > > > +{ > > > + TUNABLE_ENUM_NAME(glibc, malloc, check), > > > + TUNABLE_ENUM_NAME(glibc, malloc, top_pad), > > > + TUNABLE_ENUM_NAME(glibc, malloc, perturb), > > > + TUNABLE_ENUM_NAME(glibc, malloc, mmap_threshold), > > > + TUNABLE_ENUM_NAME(glibc, malloc, trim_threshold), > > > + TUNABLE_ENUM_NAME(glibc, malloc, mmap_max), > > > + TUNABLE_ENUM_NAME(glibc, malloc, arena_max), > > > + TUNABLE_ENUM_NAME(glibc, malloc, arena_test), > > > +} tunable_id_t; > > > + > > > +#ifdef TUNABLES_INTERNAL > > > +static tunable_t tunable_list[] = { > > > + {TUNABLE_NAME_S(glibc, malloc, check), NULL, NULL, false}, > > > + {TUNABLE_NAME_S(glibc, malloc, top_pad), NULL, NULL, false}, > > > + {TUNABLE_NAME_S(glibc, malloc, perturb), NULL, NULL, false}, > > > + {TUNABLE_NAME_S(glibc, malloc, mmap_threshold), NULL, NULL, false}, > > > + {TUNABLE_NAME_S(glibc, malloc, trim_threshold), NULL, NULL, false}, > > > + {TUNABLE_NAME_S(glibc, malloc, mmap_max), NULL, NULL, false}, > > > + {TUNABLE_NAME_S(glibc, malloc, arena_max), NULL, NULL, false}, > > > + {TUNABLE_NAME_S(glibc, malloc, arena_test), NULL, NULL, false}, > > > +}; > > > +#endif > > > diff --git a/tunables/tunables.c b/tunables/tunables.c > > > new file mode 100644 > > > index 0000000..8834aff > > > --- /dev/null > > > +++ b/tunables/tunables.c > > > @@ -0,0 +1,163 @@ > > > +/* The tunable framework. See the README to know how to use the tunable in > > > + a glibc module. > > > + > > > + Copyright (C) 2016 Free Software Foundation, Inc. > > > + This file is part of the GNU C Library. > > > + > > > + The GNU C Library is free software; you can redistribute it and/or > > > + modify it under the terms of the GNU Lesser General Public > > > + License as published by the Free Software Foundation; either > > > + version 2.1 of the License, or (at your option) any later version. > > > + > > > + The GNU C Library is distributed in the hope that it will be useful, > > > + but WITHOUT ANY WARRANTY; without even the implied warranty of > > > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > > + Lesser General Public License for more details. > > > + > > > + You should have received a copy of the GNU Lesser General Public > > > + License along with the GNU C Library; if not, see > > > + <http://www.gnu.org/licenses/>. */ > > > + > > > +#include <stddef.h> > > > +#include <stdlib.h> > > > +#include <string.h> > > > +#include <stdbool.h> > > > +#include <unistd.h> > > > +#include <sys/param.h> > > > + > > > +extern char **__environ; > > > + > > > +#define TUNABLES_INTERNAL 1 > > > +#include "tunables.h" > > > + > > > +/* We avoid calling into the C library as much as we can, especially functions > > > + that we know could use tunables in future for some reason or the other. For > > > + example, we call mmap directly instead of malloc because malloc uses > > > + tunables. Similarly, the string functions may use tunables in future to > > > + select optimal routines and we want to make sure we stay out of the way and > > > + not invoke their initializers before the tunable list has been > > > + initialized. */ > > > + > > > +static void > > > +t_memcpy (char *dest, const char *src, size_t len) > > > +{ > > > + for (size_t i = 0; i < len; i++) > > > + dest[i] = src[i]; > > > +} > > > + > > > +static int > > > +t_strcmp (const char *a, const char *b) > > > +{ > > > + size_t i = 0; > > > + > > > + for (i = 0; *a != '\0' && *b != '\0'; i++, a++, b++) > > > + if (*a != *b) > > > + return *a - *b; > > > + > > > + return *a - *b; > > > +} > > > + > > > +static bool > > > +get_next_env (char ***envp, char **name, size_t *namelen, char **val) > > > +{ > > > + char **ev = *envp; > > > + > > > + while (ev != NULL && *ev != '\0') > > > + { > > > + char *envline = *ev; > > > + int len = 0; > > > + > > > + while (envline[len] != '\0' && envline[len] != '=') > > > + len++; > > > + > > > + /* Just the name and no value, go to the next one. */ > > > + if (envline[len] == '\0') > > > + continue; > > > + > > > + *name = envline; > > > + *namelen = len; > > > + *val = &envline[len + 1]; > > > + *envp = ++ev; > > > + > > > + return true; > > > + } > > > + > > > + return false; > > > +} > > > + > > > +/* Initialize tunables from the GLIBC_TUNABLES environment variable. The > > > + variable is set as colon separated name=value pairs. This routine may be > > > + called from the constructor of either libpthread.so or lic.so. Constructor > > > + calls for libraries are protected by the dl_load_lock, so this routine won't > > > + be called concurrently from diferent threads. */ > > > +void > > > +__tunables_init (char **envp) > > > +{ > > > + /* Empty for now. */ > > > +} > > > +libc_hidden_def (__tunables_init) > > > + > > > +/* Initialize all tunables using its legacy environment variable values whose > > > + names are passed in ENVVARS. */ > > > +void > > > +__compat_tunables_init_envvars (struct compat_tunable_env *envvars, int count) > > > +{ > > > + /* Traverse through the environment to find environment variables we may need > > > + to set. */ > > > + char **envp = __environ; > > > + char *envname = NULL; > > > + char *envval = NULL; > > > + size_t len = 0; > > > + > > > + while (get_next_env (&envp, &envname, &len, &envval)) > > > + { > > > + int init_count = 0; > > > + for (int i = 0; i < count; i++) > > > + { > > > + tunable_id_t t = envvars[i].id; > > > + tunable_t *cur = &tunable_list[t]; > > > + > > > + /* Skip over tunables that have already been initialized. */ > > > + if (cur->initialized) > > > + { > > > + init_count++; > > > + continue; > > > + } > > > + > > > + const char *name = envvars[i].env; > > > + char *ename = alloca (len + 1); > > > + > > > + t_memcpy (ename, envname, len); > > > + ename[len] = '\0'; > > > + > > > + /* We have a match. Initialize and move on to the next line. */ > > > + if (t_strcmp (ename, name) == 0) > > > + { > > > + cur->val = envval; > > > + cur->set (cur->val); > > > + cur->initialized = true; > > > + break; > > > + } > > > + } > > > + > > > + /* All of the tunable envvars have been initialized. */ > > > + if (count == init_count) > > > + break; > > > + } > > > +} > > > +libc_hidden_def (__compat_tunables_init_envvars) > > > + > > > +/* Initialize a tunable and set its value. */ > > > +void > > > +__tunable_register (tunable_id_t id, tunable_setter_t set_func) > > > +{ > > > + tunable_t *cur = &tunable_list[id]; > > > + > > > + cur->set = set_func; > > > + if (cur->val != NULL) > > > + { > > > + set_func (cur->val); > > > + cur->initialized = true; > > > + } > > > +} > > > +libc_hidden_def (__tunable_register) > > > diff --git a/tunables/tunables.h b/tunables/tunables.h > > > new file mode 100644 > > > index 0000000..6aef30e > > > --- /dev/null > > > +++ b/tunables/tunables.h > > > @@ -0,0 +1,110 @@ > > > +/* The tunable framework. See the README to know how to use the tunable in > > > + a glibc module. > > > + > > > + Copyright (C) 2016 Free Software Foundation, Inc. > > > + This file is part of the GNU C Library. > > > + > > > + The GNU C Library is free software; you can redistribute it and/or > > > + modify it under the terms of the GNU Lesser General Public > > > + License as published by the Free Software Foundation; either > > > + version 2.1 of the License, or (at your option) any later version. > > > + > > > + The GNU C Library is distributed in the hope that it will be useful, > > > + but WITHOUT ANY WARRANTY; without even the implied warranty of > > > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > > + Lesser General Public License for more details. > > > + > > > + You should have received a copy of the GNU Lesser General Public > > > + License along with the GNU C Library; if not, see > > > + <http://www.gnu.org/licenses/>. */ > > > + > > > +#include <stdbool.h> > > > + > > > +typedef void (*tunable_setter_t) (const char *); > > > + > > > +/* A tunable. */ > > > +struct _tunable > > > +{ > > > + const char *name; > > > + const char *val; > > > + tunable_setter_t set; > > > + bool initialized; > > > +}; > > > + > > > +typedef struct _tunable tunable_t; > > > + > > > +#include "tunable-list.h" > > > + > > > +struct compat_tunable_env > > > +{ > > > + tunable_id_t id; > > > + const char *env; > > > +}; > > > + > > > + > > > +extern void __compat_tunables_init_envvars (struct compat_tunable_env *, int); > > > +extern void __tunable_register (tunable_id_t, tunable_setter_t); > > > +extern void __tunables_init (char **); > > > + > > > +/* Avoid a PLT reference when called from within libc.so. */ > > > +libc_hidden_proto (__compat_tunables_init_envvars) > > > +libc_hidden_proto (__tunable_register) > > > +libc_hidden_proto (__tunables_init) > > > + > > > +/* Register a tunable ID in namespace NS inside the glibc top namespace with > > > + SET as the setter function. */ > > > +#define TUNABLE_REGISTER(ns,id,set) \ > > > + TUNABLE_REGISTER_FULL (TOP_NAMESPACE, ns, id, set) > > > + > > > +/* Register a tunable ID in namespace NS inside the TNS top namespace with > > > + SET as the setter function. */ > > > +#define TUNABLE_REGISTER_FULL(tns,ns,id,set) \ > > > + __tunable_register (TUNABLE_ENUM_NAME (tns, ns, id), (set)) > > > + > > > + > > > + > > > +/* COMPATIBILITY INTERFACE. These macros should be used to define tunables for > > > + variables that are currently being controlled using environment variables. > > > + New tunables must not use this interface. */ > > > + > > > +#define _ADD_COMPAT_TUNABLE_ENV(__id,__env) \ > > > +({ \ > > > + envvars[envvars_cnt].id = TUNABLE_ENUM_NAME (TOP_NAMESPACE, \ > > > + TUNABLE_NAMESPACE, __id); \ > > > + envvars[envvars_cnt++].env = (__env); \ > > > +}) > > > + > > > +/* Start registering tunables in the current namespace. */ > > > +#define COMPAT_TUNABLES_NAMESPACE_BEGIN(size) \ > > > + { \ > > > + struct compat_tunable_env envvars[size]; \ > > > + int envvars_cnt = 0; > > > + > > > +/* Register a tunable. This macro validates that the call is OK and then calls > > > + tunable_init to do the real work of adding the tunable and setting its value > > > + based on its environment variable(s). */ > > > +#define COMPAT_TUNABLE_REGISTER(id,env,set) \ > > > +({ \ > > > + assert (envvars_cnt < (sizeof (envvars) \ > > > + / sizeof (struct compat_tunable_env))); \ > > > + if (!__libc_enable_secure) \ > > > + _ADD_COMPAT_TUNABLE_ENV (id, env); \ > > > + TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \ > > > +}) > > > + > > > +/* Does exactly the same thing as TUNABLE_REGISTER, except that it allows the > > > + tunable to look for environment variable values even for setuid binaries. > > > + This is a separate macro and not just another parameter in TUNABLE_REGISTER > > > + to avoid accidentally setting a secure flag where it is not required. */ > > > +#define COMPAT_TUNABLE_REGISTER_SECURE(id,env,set) \ > > > +({ \ > > > + assert (envvars_cnt < (sizeof (envvars) \ > > > + / sizeof (struct compat_tunable_env))); \ > > > + _ADD_COMPAT_TUNABLE_ENV(id, env); \ > > > + TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \ > > > +}) > > > + > > > +/* Initialize tunables in the namespace. */ > > > +#define COMPAT_TUNABLES_NAMESPACE_INIT() \ > > > + __compat_tunables_init_envvars (envvars, envvars_cnt); \ > > > + } > > > diff --git a/tunables/tunables.list b/tunables/tunables.list > > > new file mode 100644 > > > index 0000000..f1335cc > > > --- /dev/null > > > +++ b/tunables/tunables.list > > > @@ -0,0 +1,12 @@ > > > +glibc { > > > + malloc { > > > + check > > > + top_pad > > > + perturb > > > + mmap_threshold > > > + trim_threshold > > > + mmap_max > > > + arena_max > > > + arena_test > > > + } > > > +} > > > -- > > > 2.5.0 > > > >
On 01/16/2016 07:55 PM, Siddhesh Poyarekar wrote: > The tunables framework allows us to uniformly manage and expose global > variables inside glibc as switches to users. tunables/README has > instructions for glibc developers to add new tunables. > > Tunables support can be enabled by passing the --enable-tunables > configure flag to the configure script. This patch only adds a > framework and does not pose any limitations on how tunable values are > read from the user. It also adds environment variables used in malloc > behaviour tweaking to the tunables framework as a PoC of the > compatibility interface. I think it is a mid-term goal to support re-tuning of running processes (from the outside). It seems difficult to get there from what you have posted, mainly for two reasons: Tunable arguments are strings. It is difficult patch strings into running processes because of memory management issues (when is it safe to free the old version?), and you'd have to call the registered callback to parse. Tunables are registered programmatically. This means that there is a time where the tunables descriptor information is incomplete and only partially initialized. The introspecting process would have to compensate for this rare corner case. Furthermore, the tunables in glibc should be self-describing, but they currently are not because there is no type information associated with them, and it is not possible to determine the valid number ranges. > +/* We avoid calling into the C library as much as we can, especially functions > + that we know could use tunables in future for some reason or the other. For > + example, we call mmap directly instead of malloc because malloc uses > + tunables. Similarly, the string functions may use tunables in future to > + select optimal routines and we want to make sure we stay out of the way and > + not invoke their initializers before the tunable list has been > + initialized. */ This needs special compiler flags, otherwise it won't work as intended. The registered callbacks would have to be written in the same style (atoi might not be safe there). This problem will go away if you switch to a full declarative approach (which does not need any initialization). A declarative approach would also avoid a discussion whether we have to mangle function pointers (because there wouldn't be any). > + const char *name = envvars[i].env; > + char *ename = alloca (len + 1); Looks like an unbounded alloca, and it's in a loop even. Florian
On Wed, Feb 10, 2016 at 02:39:37PM +0100, Florian Weimer wrote: > I think it is a mid-term goal to support re-tuning of running processes > (from the outside). It seems difficult to get there from what you have > posted, mainly for two reasons: In general getting to re-tuning running processes will require significant changes anyway and I don't want to over-engineer the current solution based on that target. Of course, if you have an idea that doesn't need too much additional infrastructure then I am very interested in hearing that. > Tunable arguments are strings. It is difficult patch strings into > running processes because of memory management issues (when is it safe > to free the old version?), and you'd have to call the registered > callback to parse. <see comment about self-description> > Tunables are registered programmatically. This means that there is a > time where the tunables descriptor information is incomplete and only > partially initialized. The introspecting process would have to > compensate for this rare corner case. The introspecting process will have to synchronize with the target in some other way. That would be a necessary enhancement regardless of the way the data is represented. > Furthermore, the tunables in glibc should be self-describing, but they > currently are not because there is no type information associated with > them, and it is not possible to determine the valid number ranges. I had done an initial mockup with self-describing tunables but I don't remember why I had dropped it. Maybe I could try it again and see if I come across the problem again. That should also fix the string patching issue. > A declarative approach would also avoid a discussion whether we have to > mangle function pointers (because there wouldn't be any). > > > + const char *name = envvars[i].env; > > + char *ename = alloca (len + 1); > > Looks like an unbounded alloca, and it's in a loop even. It is bounded by the length of the tunable name, which is specified in the glibc source. Siddhesh
diff --git a/INSTALL b/INSTALL index c70ea9f..dd06e2e 100644 --- a/INSTALL +++ b/INSTALL @@ -164,6 +164,12 @@ will be used, and CFLAGS sets optimization options for the compiler. By default for x86_64, the GNU C Library is built with vector math library. Use this option to disable vector math library. +'--enable-tunables' + Tunables support allows additional library parameters to be + customized at runtime for each application. This is an + experimental feature and affects startup time and is thus disabled + by default. + '--build=BUILD-SYSTEM' '--host=HOST-SYSTEM' These options are for cross-compiling. If you specify both options diff --git a/Makeconfig b/Makeconfig index 87a22e8..8ac199b 100644 --- a/Makeconfig +++ b/Makeconfig @@ -883,6 +883,11 @@ CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \ $(foreach lib,$(libof-$(basename $(@F))) \ $(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \ $(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F))) + +ifeq (yes,$(build-tunables)) +CPPFLAGS += -DTOP_NAMESPACE=glibc +endif + override CFLAGS = -std=gnu11 -fgnu89-inline $(config-extra-cflags) \ $(filter-out %frame-pointer,$(+cflags)) $(+gccwarn-c) \ $(sysdep-CFLAGS) $(CFLAGS-$(suffix $@)) $(CFLAGS-$(<F)) \ @@ -1098,6 +1103,10 @@ all-subdirs = csu assert ctype locale intl catgets math setjmp signal \ crypt localedata timezone rt conform debug mathvec \ $(add-on-subdirs) dlfcn elf +ifeq (yes,$(build-tunables)) +all-subdirs += tunables +endif + ifndef avoid-generated # sysd-sorted itself will contain rules making the sysd-sorted target # depend on Depend files. But if you just added a Depend file to an diff --git a/config.h.in b/config.h.in index ec9c8bc..9b07580 100644 --- a/config.h.in +++ b/config.h.in @@ -231,4 +231,7 @@ /* PowerPC32 uses fctidz for floating point to long long conversions. */ #define HAVE_PPC_FCTIDZ 0 +/* Build glibc with tunables support. */ +#define BUILD_TUNABLES 0 + #endif diff --git a/config.make.in b/config.make.in index 05ed6ec..b158f51 100644 --- a/config.make.in +++ b/config.make.in @@ -90,6 +90,7 @@ use-nscd = @use_nscd@ build-hardcoded-path-in-tests= @hardcoded_path_in_tests@ build-pt-chown = @build_pt_chown@ enable-lock-elision = @enable_lock_elision@ +build-tunables = @build_tunables@ # Build tools. CC = @CC@ diff --git a/configure b/configure index aa05d49..6fef02c 100755 --- a/configure +++ b/configure @@ -659,6 +659,7 @@ multi_arch base_machine add_on_subdirs add_ons +build_tunables build_pt_chown build_nscd link_obsolete_rpc @@ -773,6 +774,7 @@ enable_systemtap enable_build_nscd enable_nscd enable_pt_chown +enable_tunables enable_mathvec with_cpu ' @@ -1440,6 +1442,7 @@ Optional Features: --disable-build-nscd disable building and installing the nscd daemon --disable-nscd library functions will not contact the nscd daemon --enable-pt_chown Enable building and installing pt_chown + --enable-tunables Enable tunables support --enable-mathvec Enable building and installing mathvec [default depends on architecture] @@ -3642,6 +3645,19 @@ if test "$build_pt_chown" = yes; then fi +# Check whether --enable-tunables was given. +if test "${enable_tunables+set}" = set; then : + enableval=$enable_tunables; build_tunables=$enableval +else + build_tunables=no +fi + + +if test "$build_tunables" = yes; then + $as_echo "#define BUILD_TUNABLES 1" >>confdefs.h + +fi + # The abi-tags file uses a fairly simplistic model for name recognition that # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell. diff --git a/configure.ac b/configure.ac index ee7a3f1..41507ac 100644 --- a/configure.ac +++ b/configure.ac @@ -392,6 +392,16 @@ if test "$build_pt_chown" = yes; then AC_DEFINE(HAVE_PT_CHOWN) fi +AC_ARG_ENABLE([tunables], + [AS_HELP_STRING([--enable-tunables], + [Enable tunables support])], + [build_tunables=$enableval], + [build_tunables=no]) +AC_SUBST(build_tunables) +if test "$build_tunables" = yes; then + AC_DEFINE(BUILD_TUNABLES) +fi + # The abi-tags file uses a fairly simplistic model for name recognition that # can't distinguish i486-pc-linux-gnu fully from i486-pc-gnu. So we mutate a # $host_os of `gnu*' here to be `gnu-gnu*' just so that it can tell. diff --git a/csu/init-first.c b/csu/init-first.c index 77c6e1c..a4e0c54 100644 --- a/csu/init-first.c +++ b/csu/init-first.c @@ -26,6 +26,9 @@ #include <sys/param.h> #include <sys/types.h> #include <libc-internal.h> +#if BUILD_TUNABLES +# include <tunables/tunables.h> +#endif #include <ldsodefs.h> @@ -96,6 +99,10 @@ _init (int argc, char **argv, char **envp) #if defined SHARED && !defined NO_CTORS_DTORS_SECTIONS __libc_global_ctors (); #endif + +#if BUILD_TUNABLES + __tunables_init (__environ); +#endif } /* This function is defined here so that if this file ever gets into diff --git a/csu/libc-start.c b/csu/libc-start.c index f4aa01a..7629fec 100644 --- a/csu/libc-start.c +++ b/csu/libc-start.c @@ -32,7 +32,7 @@ extern int __libc_multiple_libcs; #include <tls.h> #ifndef SHARED # include <dl-osinfo.h> -extern void __pthread_initialize_minimal (void); +extern void __pthread_initialize_minimal (int, char **, char **); # ifndef THREAD_SET_STACK_GUARD /* Only exported for architectures that don't store the stack guard canary in thread local area. */ @@ -193,7 +193,7 @@ LIBC_START_MAIN (int (*main) (int, char **, char ** MAIN_AUXVEC_DECL), /* Initialize the thread library at least a bit since the libgcc functions are using thread functions if these are available and we need to setup errno. */ - __pthread_initialize_minimal (); + __pthread_initialize_minimal (argc, argv, __environ); /* Set up the stack checker's canary. */ uintptr_t stack_chk_guard = _dl_setup_stack_chk_guard (_dl_random); diff --git a/malloc/arena.c b/malloc/arena.c index 1edb4d4..f773b10 100644 --- a/malloc/arena.c +++ b/malloc/arena.c @@ -19,6 +19,11 @@ #include <stdbool.h> +#if BUILD_TUNABLES +# define TUNABLE_NAMESPACE malloc +# include <tunables/tunables.h> +#endif + /* Compile-time constants. */ #define HEAP_MIN_SIZE (32 * 1024) @@ -332,6 +337,61 @@ ptmalloc_unlock_all2 (void) # endif #endif /* !NO_THREADS */ +#if BUILD_TUNABLES +static void +tunable_set_mallopt_top_pad (const char *val) +{ + __libc_mallopt (M_TOP_PAD, atoi (val)); +} + +static void +tunable_set_mallopt_perturb (const char *val) +{ + __libc_mallopt (M_PERTURB, atoi (val)); +} + +static void +tunable_set_mallopt_mmap_max (const char *val) +{ + __libc_mallopt (M_MMAP_MAX, atoi (val)); +} + +static void +tunable_set_mallopt_arena_max (const char *val) +{ + __libc_mallopt (M_ARENA_MAX, atoi (val)); +} + +static void +tunable_set_mallopt_arena_test (const char *val) +{ + __libc_mallopt (M_ARENA_TEST, atoi (val)); +} + +static void +tunable_set_mallopt_trim_threshold (const char *val) +{ + __libc_mallopt (M_TRIM_THRESHOLD, atoi (val)); +} + +static void +tunable_set_mallopt_mmap_threshold (const char *val) +{ + __libc_mallopt (M_MMAP_THRESHOLD, atoi (val)); +} + +static void +tunable_set_mallopt_check (const char *val) +{ + if (val[0]) + { + __libc_mallopt (M_CHECK_ACTION, (int) (val[0] - '0')); + if (check_action != 0) + __malloc_check_init (); + } +} + +#else /* Initialization routine. */ #include <string.h> extern char **_environ; @@ -366,6 +426,7 @@ next_env_entry (char ***position) return result; } +#endif #ifdef SHARED @@ -401,6 +462,29 @@ ptmalloc_init (void) thread_arena = &main_arena; thread_atfork (ptmalloc_lock_all, ptmalloc_unlock_all, ptmalloc_unlock_all2); + +#if BUILD_TUNABLES + COMPAT_TUNABLES_NAMESPACE_BEGIN (8); + + COMPAT_TUNABLE_REGISTER_SECURE (check, "MALLOC_CHECK_", + tunable_set_mallopt_check); + COMPAT_TUNABLE_REGISTER (top_pad, "MALLOC_TOP_PAD_", + tunable_set_mallopt_top_pad); + COMPAT_TUNABLE_REGISTER (perturb, "MALLOC_PERTURB_", + tunable_set_mallopt_perturb); + COMPAT_TUNABLE_REGISTER (mmap_threshold, "MALLOC_MMAP_THRESHOLD_", + tunable_set_mallopt_mmap_threshold); + COMPAT_TUNABLE_REGISTER (trim_threshold, "MALLOC_TRIM_THRESHOLD_", + tunable_set_mallopt_trim_threshold); + COMPAT_TUNABLE_REGISTER (mmap_max, "MALLOC_MMAP_MAX_", + tunable_set_mallopt_mmap_max); + COMPAT_TUNABLE_REGISTER (arena_max, "MALLOC_ARENA_MAX", + tunable_set_mallopt_arena_max); + COMPAT_TUNABLE_REGISTER (arena_test, "MALLOC_ARENA_TEST", + tunable_set_mallopt_arena_test); + + COMPAT_TUNABLES_NAMESPACE_INIT (); +#else const char *s = NULL; if (__glibc_likely (_environ != NULL)) { @@ -469,6 +553,8 @@ ptmalloc_init (void) if (check_action != 0) __malloc_check_init (); } +#endif + void (*hook) (void) = atomic_forced_read (__malloc_initialize_hook); if (hook != NULL) (*hook)(); diff --git a/manual/install.texi b/manual/install.texi index de9d270..95392ea 100644 --- a/manual/install.texi +++ b/manual/install.texi @@ -195,6 +195,11 @@ configure with @option{--disable-werror}. By default for x86_64, @theglibc{} is built with vector math library. Use this option to disable vector math library. +@item --enable-tunables +Tunables support allows additional library parameters to be customized at +runtime for each application. This is an experimental feature and affects +startup time and is thus disabled by default. + @item --build=@var{build-system} @itemx --host=@var{host-system} These options are for cross-compiling. If you specify both options and diff --git a/nptl/nptl-init.c b/nptl/nptl-init.c index bdbdfed..6667801 100644 --- a/nptl/nptl-init.c +++ b/nptl/nptl-init.c @@ -38,6 +38,9 @@ #include <kernel-features.h> #include <libc-internal.h> #include <pthread-pids.h> +#if BUILD_TUNABLES +# include <tunables/tunables.h> +#endif #ifndef TLS_MULTIPLE_THREADS_IN_TCB /* Pointer to the corresponding variable in libc. */ @@ -297,7 +300,7 @@ extern void **__libc_dl_error_tsd (void) __attribute__ ((const)); static bool __nptl_initial_report_events __attribute_used__; void -__pthread_initialize_minimal_internal (void) +__pthread_initialize_minimal_internal (int argc, char **argv, char **envp) { #ifndef SHARED /* Unlike in the dynamically linked case the dynamic linker has not @@ -311,6 +314,10 @@ __pthread_initialize_minimal_internal (void) __asm __volatile (""); #endif +#if BUILD_TUNABLES + __tunables_init (envp); +#endif + /* Minimal initialization of the thread descriptor. */ struct pthread *pd = THREAD_SELF; __pthread_initialize_pids (pd); diff --git a/scripts/gen-tunables.awk b/scripts/gen-tunables.awk new file mode 100644 index 0000000..ccbf789 --- /dev/null +++ b/scripts/gen-tunables.awk @@ -0,0 +1,84 @@ +# Generate tunable-list.h from tunables.list + +BEGIN { + ns="" + top_ns="" +} + +$2 == "{" { + if (top_ns == "") { + top_ns = $1 + } + else if (ns == "") { + ns = $1 + count = 0 + } + else { + printf ("Unexpected occurrence of '{' inside a namespace: %s:%d\n", + FILENAME, FNR) + exit 1 + } + + next +} + +$1 == "}" { + if (ns != "") { + ns = "" + } + else if (top_ns != "") { + top_ns = "" + } + else { + printf ("syntax error: extra }: %s:%d\n", FILENAME, FNR) + exit 1 + } + next +} + +{ + if (ns == "") { + print "Invalid tunable outside a namespace" + exit 1 + } + val[top_ns][ns][count] = $1 + count = count + 1 +} + +END { + if (ns != "") { + print "Unterminated namespace. Is a closing brace missing?" + exit 1 + } + + print "/* Print a full tunable enum name. */" + print "#include <stddef.h>" + print "#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id)" + print "#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id\n" + + print "/* Full name for a tunable is top_ns.tunable_ns.id. */" + print "#define TUNABLE_NAME_S(top,ns,id) #top \".\" #ns \".\" #id\n" + + print "typedef enum" + print "{" + for (t in val) { + for (n in val[t]) { + for (c in val[t][n]) { + printf (" TUNABLE_ENUM_NAME(%s, %s, %s),\n", t, n, val[t][n][c]); + } + } + } + print "} tunable_id_t;\n" + print "#ifdef TUNABLES_INTERNAL" + print "static tunable_t tunable_list[] = {" + for (t in val) { + for (n in val[t]) { + for (c in val[t][n]) { + printf (" {TUNABLE_NAME_S(%s, %s, %s), NULL, NULL, false},\n", + t, n, val[t][n][c], t, n, val[t][n][c]); + } + } + } + print "};" + print "#endif" +} diff --git a/tunables/Makefile b/tunables/Makefile new file mode 100644 index 0000000..44439f6 --- /dev/null +++ b/tunables/Makefile @@ -0,0 +1,33 @@ +# Copyright (C) 2016 Free Software Foundation, Inc. +# This file is part of the GNU C Library. + +# The GNU C Library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. + +# The GNU C Library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. + +# You should have received a copy of the GNU Lesser General Public +# License along with the GNU C Library; if not, see +# <http://www.gnu.org/licenses/>. + +# +# Makefile for tunables. +# +subdir := tunables + +include ../Makeconfig + +routines = tunables + +$(objpfx)tunables.os: tunable-list.h + +tunable-list.h: $(..)scripts/gen-tunables.awk tunables.list + $(AWK) -f $^ > $@.tmp + mv $@{.tmp,} + +include ../Rules diff --git a/tunables/README b/tunables/README new file mode 100644 index 0000000..02a3f7a --- /dev/null +++ b/tunables/README @@ -0,0 +1,93 @@ + TUNABLE FRAMEWORK + ================= + +The tunable framework allows modules within glibc to register variables that +may be tweaked through an environment variable or an API call. It aims to +enforce a strict namespace rule to bring consistency to naming of these tunable +environment variables across the project. + +ADDING A NEW TUNABLE +-------------------- + +The TOP_NAMESPACE is defined by default as 'glibc' and it may be overridden in +distributions for specific tunables if they want to add their own tunables. +Downstream implementations are discouraged from using the 'glibc' namespace for +tunables they don't already have consensus to push upstream. + +There are two steps to adding a tunable: + +1. Add a tunable ID: + +Modules that wish to use the tunables interface must define the +TUNABLE_NAMESPACE macro. Following this, for each tunable you want to +register, call the TUNABLE_REGISTER macro. + +The TUNABLE_REGISTER macro takes the following arguments: + +- id: The short name of the tunable. It will be concatenated with + the TOP_NAMESPACE and TUNABLE_NAMESPACE to build the full ID of + the tunable. + +- setter: A function that accepts a string input and initializes the + tunable variable. The prototype of the function should be as + follows: + + void setter_func (const char *) + +2. Add to tunables.list: + +This file nests tunable ids inside their tunable namespace within curly braces +and multiple such namespaces are in turn nested inside their top namespaces in +a similar manner. The malloc and resolv snippets for example could look like +this: + +glibc { + malloc { + check + top_pad + } + resolv { + secure_dns + } +} + +Add your tunable in this hierarchy using the proper nesting for your namespace +and top namespace. If the top namespace is not glibc, create a new snippet +with its own hierarchy, separate from the glibc hierarchy. + +ADDING A LEGACY TUNABLE +----------------------- + +One may add tunables for variables that are currently controlled by users with +environment variables. Multiple such variables should be defined together for +better performance by using the COMPAT_TUNABLES_NAMESPACE_BEGIN macro, which +accepts a numeric parameter with the number of tunables you intend to register +in the block. + +Individual tunables are then registered using the COMPAT_TUNABLE_REGISTER and +COMPAT_TUNABLE_REGISTER_SECURE macros. They take the following parameters: + +- id: The tunable ID, similar to the normal way of adding a tunable + +- env: The name of the legacy environment variable so that one may + read its value and use it. + +- setter: The setter function, similar to the normal way of adding a + tunable. + +COMPAT_TUNABLE_REGISTER reads the environment variable only for non-setuid +binaries, whereas COMPAT_TUNABLE_REGISTER_SECURE always attempts to read the +environment variable. Use COMPAT_TUNABLE_REGISTER_SECURE with care. + +You will also have to add the tunable id to tunables/tunables.list. + +DO NOT ADD NEW TUNABLES USING THIS MECHANISM! + +FUTURE WORK +----------- + +The framework currently only allows a one-time initialization of variables +through environment variables and in some cases, modification of variables via +an API call. A future goal for this project is to allow tweaking of some +values in a running process, possibly through some kind of shared memory +mechanism. diff --git a/tunables/Versions b/tunables/Versions new file mode 100644 index 0000000..4c68b0e --- /dev/null +++ b/tunables/Versions @@ -0,0 +1,8 @@ +# Exports from tunables should only be in the GLIBC_PRIVATE namespace. +libc { + GLIBC_PRIVATE { + __compat_tunables_init_envvars; + __tunables_init; + __tunable_register; + } +} diff --git a/tunables/tunable-list.h b/tunables/tunable-list.h new file mode 100644 index 0000000..2b1caa4 --- /dev/null +++ b/tunables/tunable-list.h @@ -0,0 +1,32 @@ +/* Print a full tunable enum name. */ +#include <stddef.h> +#define TUNABLE_ENUM_NAME(top,ns,id) TUNABLE_ENUM_NAME1 (top,ns,id) +#define TUNABLE_ENUM_NAME1(top,ns,id) top ## _ ## ns ## _ ## id + +/* Full name for a tunable is top_ns.tunable_ns.id. */ +#define TUNABLE_NAME_S(top,ns,id) #top "." #ns "." #id + +typedef enum +{ + TUNABLE_ENUM_NAME(glibc, malloc, check), + TUNABLE_ENUM_NAME(glibc, malloc, top_pad), + TUNABLE_ENUM_NAME(glibc, malloc, perturb), + TUNABLE_ENUM_NAME(glibc, malloc, mmap_threshold), + TUNABLE_ENUM_NAME(glibc, malloc, trim_threshold), + TUNABLE_ENUM_NAME(glibc, malloc, mmap_max), + TUNABLE_ENUM_NAME(glibc, malloc, arena_max), + TUNABLE_ENUM_NAME(glibc, malloc, arena_test), +} tunable_id_t; + +#ifdef TUNABLES_INTERNAL +static tunable_t tunable_list[] = { + {TUNABLE_NAME_S(glibc, malloc, check), NULL, NULL, false}, + {TUNABLE_NAME_S(glibc, malloc, top_pad), NULL, NULL, false}, + {TUNABLE_NAME_S(glibc, malloc, perturb), NULL, NULL, false}, + {TUNABLE_NAME_S(glibc, malloc, mmap_threshold), NULL, NULL, false}, + {TUNABLE_NAME_S(glibc, malloc, trim_threshold), NULL, NULL, false}, + {TUNABLE_NAME_S(glibc, malloc, mmap_max), NULL, NULL, false}, + {TUNABLE_NAME_S(glibc, malloc, arena_max), NULL, NULL, false}, + {TUNABLE_NAME_S(glibc, malloc, arena_test), NULL, NULL, false}, +}; +#endif diff --git a/tunables/tunables.c b/tunables/tunables.c new file mode 100644 index 0000000..8834aff --- /dev/null +++ b/tunables/tunables.c @@ -0,0 +1,163 @@ +/* The tunable framework. See the README to know how to use the tunable in + a glibc module. + + Copyright (C) 2016 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + <http://www.gnu.org/licenses/>. */ + +#include <stddef.h> +#include <stdlib.h> +#include <string.h> +#include <stdbool.h> +#include <unistd.h> +#include <sys/param.h> + +extern char **__environ; + +#define TUNABLES_INTERNAL 1 +#include "tunables.h" + +/* We avoid calling into the C library as much as we can, especially functions + that we know could use tunables in future for some reason or the other. For + example, we call mmap directly instead of malloc because malloc uses + tunables. Similarly, the string functions may use tunables in future to + select optimal routines and we want to make sure we stay out of the way and + not invoke their initializers before the tunable list has been + initialized. */ + +static void +t_memcpy (char *dest, const char *src, size_t len) +{ + for (size_t i = 0; i < len; i++) + dest[i] = src[i]; +} + +static int +t_strcmp (const char *a, const char *b) +{ + size_t i = 0; + + for (i = 0; *a != '\0' && *b != '\0'; i++, a++, b++) + if (*a != *b) + return *a - *b; + + return *a - *b; +} + +static bool +get_next_env (char ***envp, char **name, size_t *namelen, char **val) +{ + char **ev = *envp; + + while (ev != NULL && *ev != '\0') + { + char *envline = *ev; + int len = 0; + + while (envline[len] != '\0' && envline[len] != '=') + len++; + + /* Just the name and no value, go to the next one. */ + if (envline[len] == '\0') + continue; + + *name = envline; + *namelen = len; + *val = &envline[len + 1]; + *envp = ++ev; + + return true; + } + + return false; +} + +/* Initialize tunables from the GLIBC_TUNABLES environment variable. The + variable is set as colon separated name=value pairs. This routine may be + called from the constructor of either libpthread.so or lic.so. Constructor + calls for libraries are protected by the dl_load_lock, so this routine won't + be called concurrently from diferent threads. */ +void +__tunables_init (char **envp) +{ + /* Empty for now. */ +} +libc_hidden_def (__tunables_init) + +/* Initialize all tunables using its legacy environment variable values whose + names are passed in ENVVARS. */ +void +__compat_tunables_init_envvars (struct compat_tunable_env *envvars, int count) +{ + /* Traverse through the environment to find environment variables we may need + to set. */ + char **envp = __environ; + char *envname = NULL; + char *envval = NULL; + size_t len = 0; + + while (get_next_env (&envp, &envname, &len, &envval)) + { + int init_count = 0; + for (int i = 0; i < count; i++) + { + tunable_id_t t = envvars[i].id; + tunable_t *cur = &tunable_list[t]; + + /* Skip over tunables that have already been initialized. */ + if (cur->initialized) + { + init_count++; + continue; + } + + const char *name = envvars[i].env; + char *ename = alloca (len + 1); + + t_memcpy (ename, envname, len); + ename[len] = '\0'; + + /* We have a match. Initialize and move on to the next line. */ + if (t_strcmp (ename, name) == 0) + { + cur->val = envval; + cur->set (cur->val); + cur->initialized = true; + break; + } + } + + /* All of the tunable envvars have been initialized. */ + if (count == init_count) + break; + } +} +libc_hidden_def (__compat_tunables_init_envvars) + +/* Initialize a tunable and set its value. */ +void +__tunable_register (tunable_id_t id, tunable_setter_t set_func) +{ + tunable_t *cur = &tunable_list[id]; + + cur->set = set_func; + if (cur->val != NULL) + { + set_func (cur->val); + cur->initialized = true; + } +} +libc_hidden_def (__tunable_register) diff --git a/tunables/tunables.h b/tunables/tunables.h new file mode 100644 index 0000000..6aef30e --- /dev/null +++ b/tunables/tunables.h @@ -0,0 +1,110 @@ +/* The tunable framework. See the README to know how to use the tunable in + a glibc module. + + Copyright (C) 2016 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + <http://www.gnu.org/licenses/>. */ + +#include <stdbool.h> + +typedef void (*tunable_setter_t) (const char *); + +/* A tunable. */ +struct _tunable +{ + const char *name; + const char *val; + tunable_setter_t set; + bool initialized; +}; + +typedef struct _tunable tunable_t; + +#include "tunable-list.h" + +struct compat_tunable_env +{ + tunable_id_t id; + const char *env; +}; + + +extern void __compat_tunables_init_envvars (struct compat_tunable_env *, int); +extern void __tunable_register (tunable_id_t, tunable_setter_t); +extern void __tunables_init (char **); + +/* Avoid a PLT reference when called from within libc.so. */ +libc_hidden_proto (__compat_tunables_init_envvars) +libc_hidden_proto (__tunable_register) +libc_hidden_proto (__tunables_init) + +/* Register a tunable ID in namespace NS inside the glibc top namespace with + SET as the setter function. */ +#define TUNABLE_REGISTER(ns,id,set) \ + TUNABLE_REGISTER_FULL (TOP_NAMESPACE, ns, id, set) + +/* Register a tunable ID in namespace NS inside the TNS top namespace with + SET as the setter function. */ +#define TUNABLE_REGISTER_FULL(tns,ns,id,set) \ + __tunable_register (TUNABLE_ENUM_NAME (tns, ns, id), (set)) + + + +/* COMPATIBILITY INTERFACE. These macros should be used to define tunables for + variables that are currently being controlled using environment variables. + New tunables must not use this interface. */ + +#define _ADD_COMPAT_TUNABLE_ENV(__id,__env) \ +({ \ + envvars[envvars_cnt].id = TUNABLE_ENUM_NAME (TOP_NAMESPACE, \ + TUNABLE_NAMESPACE, __id); \ + envvars[envvars_cnt++].env = (__env); \ +}) + +/* Start registering tunables in the current namespace. */ +#define COMPAT_TUNABLES_NAMESPACE_BEGIN(size) \ + { \ + struct compat_tunable_env envvars[size]; \ + int envvars_cnt = 0; + +/* Register a tunable. This macro validates that the call is OK and then calls + tunable_init to do the real work of adding the tunable and setting its value + based on its environment variable(s). */ +#define COMPAT_TUNABLE_REGISTER(id,env,set) \ +({ \ + assert (envvars_cnt < (sizeof (envvars) \ + / sizeof (struct compat_tunable_env))); \ + if (!__libc_enable_secure) \ + _ADD_COMPAT_TUNABLE_ENV (id, env); \ + TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \ +}) + +/* Does exactly the same thing as TUNABLE_REGISTER, except that it allows the + tunable to look for environment variable values even for setuid binaries. + This is a separate macro and not just another parameter in TUNABLE_REGISTER + to avoid accidentally setting a secure flag where it is not required. */ +#define COMPAT_TUNABLE_REGISTER_SECURE(id,env,set) \ +({ \ + assert (envvars_cnt < (sizeof (envvars) \ + / sizeof (struct compat_tunable_env))); \ + _ADD_COMPAT_TUNABLE_ENV(id, env); \ + TUNABLE_REGISTER (TUNABLE_NAMESPACE, id, set); \ +}) + +/* Initialize tunables in the namespace. */ +#define COMPAT_TUNABLES_NAMESPACE_INIT() \ + __compat_tunables_init_envvars (envvars, envvars_cnt); \ + } diff --git a/tunables/tunables.list b/tunables/tunables.list new file mode 100644 index 0000000..f1335cc --- /dev/null +++ b/tunables/tunables.list @@ -0,0 +1,12 @@ +glibc { + malloc { + check + top_pad + perturb + mmap_threshold + trim_threshold + mmap_max + arena_max + arena_test + } +}