Message ID | 20170615131042.GA28885@gmail.com |
---|---|
State | New |
Headers | show |
On Thu, Jun 15, 2017 at 6:10 AM, H.J. Lu <hjl.tools@gmail.com> wrote: > The current IFUNC selection is based on microbenchmarks in glibc. It > should give the best performance for most workloads. But other choices > may have better performance for a particular workload or on the hardware > which wasn't available at the selection was made. The environment > variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used > to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz, > where the feature name is case-sensitive and has to match the ones in > cpu-features.h. It can be used by glibc developers to override the > IFUNC selection to tune for a new processor or improve performance for > a particular workload. It isn't intended for normal end users. > > NOTE: the IFUNC selection may change over time. Please check all > multiarch implementations when experimenting. > > Also, GLIBC_TUNABLES=glibc.tune.non_temporal_threshold=NUMBER is > provided to set threshold to use non temporal store to NUMBER, > GLIBC_TUNABLES=glibc.tune.data_cache_size=NUMBER to set data cache size, > GLIBC_TUNABLES=glibc.tune.shared_cache_size=NUMBER to set shared cache > size. > > Any comments? I will check it in this week if there is no objection. H.J. > H.J. > --- > 2017-06-15 H.J. Lu <hongjiu.lu@intel.com> > Erich Elsen <eriche@google.com> > > * elf/dl-tunables.list (tune): Add ifunc, non_temporal_threshold, > data_cache_size and shared_cache_size. > * manual/tunables.texi: Document glibc.tune.ifunc, > glibc.tune.data_cache_size, glibc.tune.shared_cache_size and > glibc.tune.non_temporal_threshold. > * sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file. > * sysdeps/x86/cpu-tunables.c: Likewise. > * sysdeps/x86/cacheinfo.c > (init_cacheinfo): Check and get data cache size, shared cache > size and non temporal threshold from cpu_features. > * sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE): > New. > [HAVE_TUNABLES] Include <unistd.h>. > [HAVE_TUNABLES] Include <elf/dl-tunables.h>. > [HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise. > [HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set > IFUNC selection, data cache size, shared cache size and non > temporal threshold. > * sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size, > shared_cache_size and non_temporal_threshold. > --- > elf/dl-tunables.list | 16 ++ > manual/tunables.texi | 36 ++++ > sysdeps/unix/sysv/linux/x86/dl-sysdep.c | 21 ++ > sysdeps/x86/cacheinfo.c | 10 +- > sysdeps/x86/cpu-features.c | 19 ++ > sysdeps/x86/cpu-features.h | 8 + > sysdeps/x86/cpu-tunables.c | 330 ++++++++++++++++++++++++++++++++ > 7 files changed, 439 insertions(+), 1 deletion(-) > create mode 100644 sysdeps/unix/sysv/linux/x86/dl-sysdep.c > create mode 100644 sysdeps/x86/cpu-tunables.c > > diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list > index 41ce9af..78354fb 100644 > --- a/elf/dl-tunables.list > +++ b/elf/dl-tunables.list > @@ -82,6 +82,22 @@ glibc { > type: UINT_64 > env_alias: LD_HWCAP_MASK > default: HWCAP_IMPORTANT > + } > + ifunc { > + type: STRING > + security_level: SXID_IGNORE > + } > + non_temporal_threshold { > + type: SIZE_T > + security_level: SXID_IGNORE > + } > + data_cache_size { > + type: SIZE_T > + security_level: SXID_IGNORE > + } > + shared_cache_size { > + type: SIZE_T > + security_level: SXID_IGNORE > } > } > } > diff --git a/manual/tunables.texi b/manual/tunables.texi > index c9a4cb7..709f58a 100644 > --- a/manual/tunables.texi > +++ b/manual/tunables.texi > @@ -198,6 +198,14 @@ is 8 times the number of cores online. > @cindex hardware capability tunables > @cindex hwcap tunables > @cindex tunables, hwcap > +@cindex ifunc tunables > +@cindex tunables, ifunc > +@cindex data_cache_size tunables > +@cindex tunables, data_cache_size > +@cindex shared_cache_size tunables > +@cindex tunables, shared_cache_size > +@cindex non_temporal_threshold tunables > +@cindex tunables, non_temporal_threshold > > @deftp {Tunable namespace} glibc.tune > Behavior of @theglibc{} can be tuned to assume specific hardware capabilities > @@ -213,3 +221,31 @@ extensions available in the processor at runtime for some architectures. The > @code{glibc.tune.hwcap_mask} tunable allows the user to mask out those > capabilities at runtime, thus disabling use of those extensions. > @end deftp > + > +@deftp Tunable glibc.tune.ifunc > +The @code{glibc.tune.ifunc=-xxx,yyy,-zzz...} tunable allows the user to > +enable CPU/ARCH feature @code{yyy}, disable CPU/ARCH feature @code{xxx} > +and @code{zzz} where the feature name is case-sensitive and has to match > +the ones in @code{sysdeps/x86/cpu-features.h}. > + > +This tunable is specific to i386 and x86-64. > +@end deftp > + > +@deftp Tunable glibc.tune.data_cache_size > +The @code{glibc.tune.data_cache_size} tunable allows the user to set > +data cache size in bytes for use in memory and string routines. > + > +This tunable is specific to i386 and x86-64. > +@end deftp > + > +@deftp Tunable glibc.tune.shared_cache_size > +The @code{glibc.tune.shared_cache_size} tunable allows the user to set > +shared cache size in bytes for use in memory and string routines. > +@end deftp > + > +@deftp Tunable glibc.tune.non_temporal_threshold > +The @code{glibc.tune.non_temporal_threshold} tunable allows the user to > +set threshold in bytes for non temporal store. > + > +This tunable is specific to i386 and x86-64. > +@end deftp > diff --git a/sysdeps/unix/sysv/linux/x86/dl-sysdep.c b/sysdeps/unix/sysv/linux/x86/dl-sysdep.c > new file mode 100644 > index 0000000..64eb0d7 > --- /dev/null > +++ b/sysdeps/unix/sysv/linux/x86/dl-sysdep.c > @@ -0,0 +1,21 @@ > +/* Operating system support for run-time dynamic linker. X86 version. > + Copyright (C) 2017 Free Software Foundation, Inc. > + This file is part of the GNU C Library. > + > + The GNU C Library is free software; you can redistribute it and/or > + modify it under the terms of the GNU Lesser General Public > + License as published by the Free Software Foundation; either > + version 2.1 of the License, or (at your option) any later version. > + > + The GNU C Library is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + Lesser General Public License for more details. > + > + You should have received a copy of the GNU Lesser General Public > + License along with the GNU C Library; if not, see > + <http://www.gnu.org/licenses/>. */ > + > +#include <config.h> > +#include <sysdeps/x86/cpu-tunables.c> > +#include <sysdeps/unix/sysv/linux/dl-sysdep.c> > diff --git a/sysdeps/x86/cacheinfo.c b/sysdeps/x86/cacheinfo.c > index 8000fd1..9542dfb 100644 > --- a/sysdeps/x86/cacheinfo.c > +++ b/sysdeps/x86/cacheinfo.c > @@ -752,6 +752,9 @@ intel_bug_no_cache_info: > #endif > } > > + if (cpu_features->data_cache_size != 0) > + data = cpu_features->data_cache_size; > + > if (data > 0) > { > __x86_raw_data_cache_size_half = data / 2; > @@ -762,6 +765,9 @@ intel_bug_no_cache_info: > __x86_data_cache_size = data; > } > > + if (cpu_features->shared_cache_size != 0) > + shared = cpu_features->shared_cache_size; > + > if (shared > 0) > { > __x86_raw_shared_cache_size_half = shared / 2; > @@ -777,7 +783,9 @@ intel_bug_no_cache_info: > store becomes faster on a 8-core processor. This is the 3/4 of the > total shared cache size. */ > __x86_shared_non_temporal_threshold > - = __x86_shared_cache_size * threads * 3 / 4; > + = (cpu_features->non_temporal_threshold != 0 > + ? cpu_features->non_temporal_threshold > + : __x86_shared_cache_size * threads * 3 / 4); > } > > #endif > diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c > index 4288001..1ca0310 100644 > --- a/sysdeps/x86/cpu-features.c > +++ b/sysdeps/x86/cpu-features.c > @@ -20,6 +20,15 @@ > #include <cpu-features.h> > #include <dl-hwcap.h> > > +#if HAVE_TUNABLES > +# define TUNABLE_NAMESPACE tune > +# include <unistd.h> /* Get STDOUT_FILENO for _dl_printf. */ > +# include <elf/dl-tunables.h> > + > +extern void TUNABLE_CALLBACK (set_ifunc) (tunable_val_t *) > + attribute_hidden; > +#endif > + > static void > get_common_indeces (struct cpu_features *cpu_features, > unsigned int *family, unsigned int *model, > @@ -312,6 +321,16 @@ no_cpuid: > cpu_features->model = model; > cpu_features->kind = kind; > > +#if HAVE_TUNABLES > + TUNABLE_GET (ifunc, tunable_val_t *, TUNABLE_CALLBACK (set_ifunc)); > + cpu_features->non_temporal_threshold > + = TUNABLE_GET (non_temporal_threshold, long int, NULL); > + cpu_features->data_cache_size > + = TUNABLE_GET (data_cache_size, long int, NULL); > + cpu_features->shared_cache_size > + = TUNABLE_GET (shared_cache_size, long int, NULL); > +#endif > + > /* Reuse dl_platform, dl_hwcap and dl_hwcap_mask for x86. */ > GLRO(dl_platform) = NULL; > GLRO(dl_hwcap) = 0; > diff --git a/sysdeps/x86/cpu-features.h b/sysdeps/x86/cpu-features.h > index f2329ea..fef5e18 100644 > --- a/sysdeps/x86/cpu-features.h > +++ b/sysdeps/x86/cpu-features.h > @@ -215,6 +215,14 @@ struct cpu_features > unsigned int family; > unsigned int model; > unsigned int feature[FEATURE_INDEX_MAX]; > + /* Data cache size for use in memory and string routines, typically > + L1 size. */ > + long int data_cache_size; > + /* Shared cache size for use in memory and string routines, typically > + L2 or L3 size. */ > + long int shared_cache_size; > + /* Threshold to use non temporal store. */ > + long int non_temporal_threshold; > }; > > /* Used from outside of glibc to get access to the CPU features > diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c > new file mode 100644 > index 0000000..dfa3647 > --- /dev/null > +++ b/sysdeps/x86/cpu-tunables.c > @@ -0,0 +1,330 @@ > +/* x86 CPU feature tuning. > + This file is part of the GNU C Library. > + Copyright (C) 2017 Free Software Foundation, Inc. > + > + The GNU C Library is free software; you can redistribute it and/or > + modify it under the terms of the GNU Lesser General Public > + License as published by the Free Software Foundation; either > + version 2.1 of the License, or (at your option) any later version. > + > + The GNU C Library is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + Lesser General Public License for more details. > + > + You should have received a copy of the GNU Lesser General Public > + License along with the GNU C Library; if not, see > + <http://www.gnu.org/licenses/>. */ > + > +#if HAVE_TUNABLES > +# define TUNABLE_NAMESPACE tune > +# include <stdbool.h> > +# include <stdint.h> > +# include <unistd.h> /* Get STDOUT_FILENO for _dl_printf. */ > +# include <elf/dl-tunables.h> > +# include <string.h> > +# include <cpu-features.h> > +# include <ldsodefs.h> > + > +/* We can't use IFUNC memcmp nor strlen in init_cpu_features from libc.a > + since IFUNC must be set up by init_cpu_features. */ > +# if defined USE_MULTIARCH && !defined SHARED > +# ifdef __x86_64__ > +# define DEFAULT_MEMCMP __memcmp_sse2 > +# define DEFAULT_STRLEN __strlen_sse2 > +extern __typeof (strlen) DEFAULT_STRLEN; > +# else > +# define DEFAULT_MEMCMP __memcmp_ia32 > +# define DEFAULT_STRLEN strlen > +# endif > +extern __typeof (memcmp) DEFAULT_MEMCMP; > +# else > +# define DEFAULT_MEMCMP memcmp > +# define DEFAULT_STRLEN strlen > +# endif > + > +# define CHECK_GLIBC_IFUNC_CPU_OFF(f, cpu_features, name, len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + cpu_features->cpuid[index_cpu_##name].reg_##name \ > + &= ~bit_cpu_##name; \ > + break; \ > + } > + > +/* Disable an ARCH feature NAME. We don't enable an ARCH feature which > + isn't available. */ > +# define CHECK_GLIBC_IFUNC_ARCH_OFF(f, cpu_features, name, len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + cpu_features->feature[index_arch_##name] \ > + &= ~bit_arch_##name; \ > + break; \ > + } > + > +/* Enable/disable an ARCH feature NAME. */ > +# define CHECK_GLIBC_IFUNC_ARCH_BOTH(f, cpu_features, name, disable, \ > + len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + if (disable) \ > + cpu_features->feature[index_arch_##name] \ > + &= ~bit_arch_##name; \ > + else \ > + cpu_features->feature[index_arch_##name] \ > + |= bit_arch_##name; \ > + break; \ > + } > + > +/* Enable/disable an ARCH feature NAME. Enable an ARCH feature only > + if the ARCH feature NEED is also enabled. */ > +# define CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH(f, cpu_features, name, \ > + need, disable, len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + if (disable) \ > + cpu_features->feature[index_arch_##name] \ > + &= ~bit_arch_##name; \ > + else if (CPU_FEATURES_ARCH_P (cpu_features, need)) \ > + cpu_features->feature[index_arch_##name] \ > + |= bit_arch_##name; \ > + break; \ > + } > + > +/* Enable/disable an ARCH feature NAME. Enable an ARCH feature only > + if the CPU feature NEED is also enabled. */ > +# define CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH(f, cpu_features, name, \ > + need, disable, len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + if (disable) \ > + cpu_features->feature[index_arch_##name] \ > + &= ~bit_arch_##name; \ > + else if (CPU_FEATURES_CPU_P (cpu_features, need)) \ > + cpu_features->feature[index_arch_##name] \ > + |= bit_arch_##name; \ > + break; \ > + } > + > +attribute_hidden > +void > +TUNABLE_CALLBACK (set_ifunc) (tunable_val_t *valp) > +{ > + /* The current IFUNC selection is based on microbenchmarks in glibc. > + It should give the best performance for most workloads. But other > + choices may have better performance for a particular workload or on > + the hardware which wasn't available when the selection was made. > + The environment variable, GLIBC_IFUNC=-xxx,yyy,-zzz...., can be > + used to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy > + and zzz, where the feature name is case-sensitive and has to match > + the ones in cpu-features.h. It can be used by glibc developers to > + tune for a new processor or override the IFUNC selection to improve > + performance for a particular workload. > + > + Since all CPU/ARCH features are hardware optimizations without > + security implication, except for Prefer_MAP_32BIT_EXEC, which can > + only be disabled, we check GLIBC_IFUNC for programs, including > + set*id ones. > + > + NOTE: the IFUNC selection may change over time. Please check all > + multiarch implementations when experimenting. */ > + > + const char *p = valp->strval; > + struct cpu_features *cpu_features = &GLRO(dl_x86_cpu_features); > + const char *end = p + DEFAULT_STRLEN (p); > + size_t len; > + > + do > + { > + const char *c, *n; > + bool disable; > + size_t nl; > + > + for (c = p; *c != ','; c++) > + if (c >= end) > + break; > + > + len = c - p; > + disable = *p == '-'; > + if (disable) > + { > + n = p + 1; > + nl = len - 1; > + } > + else > + { > + n = p; > + nl = len; > + } > + switch (nl) > + { > + default: > + break; > + case 3: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX, 3); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, CX8, 3); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, FMA, 3); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, HTT, 3); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, RTM, 3); > + } > + break; > + case 4: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX2, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, BMI1, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, BMI2, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, CMOV, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, ERMS, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, FMA4, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE2, 4); > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, I586, 4); > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, I686, 4); > + } > + break; > + case 5: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, LZCNT, 5); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, MOVBE, 5); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSSE3, 5); > + } > + break; > + case 6: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, POPCNT, 6); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_1, 6); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_2, 6); > + } > + break; > + case 7: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512F, 7); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, OSXSAVE, 7); > + } > + break; > + case 8: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512CD, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512BW, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512DQ, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512ER, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512PF, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512VL, 8); > + } > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Slow_BSF, > + disable, 8); > + break; > + case 10: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, AVX_Usable, > + 10); > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, FMA_Usable, > + 10); > + } > + break; > + case 11: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, AVX2_Usable, > + 11); > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, FMA4_Usable, > + 11); > + } > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Prefer_ERMS, > + disable, 11); > + CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH (n, cpu_features, > + Slow_SSE4_2, SSE4_2, > + disable, 11); > + break; > + case 14: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, > + AVX512F_Usable, 14); > + } > + break; > + case 15: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, > + AVX512DQ_Usable, 15); > + } > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Fast_Rep_String, > + disable, 15); > + break; > + case 16: > + { > + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH > + (n, cpu_features, Prefer_No_AVX512, AVX512F_Usable, > + disable, 16); > + } > + break; > + case 18: > + { > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Fast_Copy_Backward, disable, > + 18); > + } > + break; > + case 19: > + { > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Fast_Unaligned_Load, disable, > + 19); > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Fast_Unaligned_Copy, disable, > + 19); > + } > + break; > + case 20: > + { > + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH > + (n, cpu_features, Prefer_No_VZEROUPPER, AVX_Usable, > + disable, 20); > + } > + break; > + case 21: > + { > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Prefer_MAP_32BIT_EXEC, disable, > + 21); > + } > + break; > + case 23: > + { > + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH > + (n, cpu_features, AVX_Fast_Unaligned_Load, AVX_Usable, > + disable, 23); > + } > + break; > + case 26: > + { > + CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH > + (n, cpu_features, Prefer_PMINUB_for_stringop, SSE2, > + disable, 26); > + } > + break; > + case 27: > + { > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Use_dl_runtime_resolve_slow, > + disable, 27); > + } > + break; > + } > + p += len + 1; > + } > + while (p < end); > +} > +#endif > -- > 2.9.4 >
On 15/06/2017 10:10, H.J. Lu wrote: > The current IFUNC selection is based on microbenchmarks in glibc. It > should give the best performance for most workloads. But other choices > may have better performance for a particular workload or on the hardware > which wasn't available at the selection was made. The environment > variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used > to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz, > where the feature name is case-sensitive and has to match the ones in > cpu-features.h. It can be used by glibc developers to override the > IFUNC selection to tune for a new processor or improve performance for > a particular workload. It isn't intended for normal end users. > > NOTE: the IFUNC selection may change over time. Please check all > multiarch implementations when experimenting. > > Also, GLIBC_TUNABLES=glibc.tune.non_temporal_threshold=NUMBER is > provided to set threshold to use non temporal store to NUMBER, > GLIBC_TUNABLES=glibc.tune.data_cache_size=NUMBER to set data cache size, > GLIBC_TUNABLES=glibc.tune.shared_cache_size=NUMBER to set shared cache > size. > > Any comments? > > H.J. > --- > 2017-06-15 H.J. Lu <hongjiu.lu@intel.com> > Erich Elsen <eriche@google.com> > > * elf/dl-tunables.list (tune): Add ifunc, non_temporal_threshold, > data_cache_size and shared_cache_size. > * manual/tunables.texi: Document glibc.tune.ifunc, > glibc.tune.data_cache_size, glibc.tune.shared_cache_size and > glibc.tune.non_temporal_threshold. > * sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file. > * sysdeps/x86/cpu-tunables.c: Likewise. > * sysdeps/x86/cacheinfo.c > (init_cacheinfo): Check and get data cache size, shared cache > size and non temporal threshold from cpu_features. > * sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE): > New. > [HAVE_TUNABLES] Include <unistd.h>. > [HAVE_TUNABLES] Include <elf/dl-tunables.h>. > [HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise. > [HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set > IFUNC selection, data cache size, shared cache size and non > temporal threshold. > * sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size, > shared_cache_size and non_temporal_threshold. > --- > elf/dl-tunables.list | 16 ++ > manual/tunables.texi | 36 ++++ > sysdeps/unix/sysv/linux/x86/dl-sysdep.c | 21 ++ > sysdeps/x86/cacheinfo.c | 10 +- > sysdeps/x86/cpu-features.c | 19 ++ > sysdeps/x86/cpu-features.h | 8 + > sysdeps/x86/cpu-tunables.c | 330 ++++++++++++++++++++++++++++++++ > 7 files changed, 439 insertions(+), 1 deletion(-) > create mode 100644 sysdeps/unix/sysv/linux/x86/dl-sysdep.c > create mode 100644 sysdeps/x86/cpu-tunables.c > > diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list > index 41ce9af..78354fb 100644 > --- a/elf/dl-tunables.list > +++ b/elf/dl-tunables.list > @@ -82,6 +82,22 @@ glibc { > type: UINT_64 > env_alias: LD_HWCAP_MASK > default: HWCAP_IMPORTANT > + } > + ifunc { > + type: STRING > + security_level: SXID_IGNORE > + } > + non_temporal_threshold { > + type: SIZE_T > + security_level: SXID_IGNORE > + } > + data_cache_size { > + type: SIZE_T > + security_level: SXID_IGNORE > + } > + shared_cache_size { > + type: SIZE_T > + security_level: SXID_IGNORE > } Is it possible with current tunables approach to make it arch-specific? The 'ifunc' switch seems a generic one, but the 'non_temporal_threshold', 'data_cache_size', and 'shared_cache_size' are very x86 specific and I see it confusing exposing them to non-x86 architectures. > } > } > diff --git a/manual/tunables.texi b/manual/tunables.texi > index c9a4cb7..709f58a 100644 > --- a/manual/tunables.texi > +++ b/manual/tunables.texi > @@ -198,6 +198,14 @@ is 8 times the number of cores online. > @cindex hardware capability tunables > @cindex hwcap tunables > @cindex tunables, hwcap > +@cindex ifunc tunables > +@cindex tunables, ifunc > +@cindex data_cache_size tunables > +@cindex tunables, data_cache_size > +@cindex shared_cache_size tunables > +@cindex tunables, shared_cache_size > +@cindex non_temporal_threshold tunables > +@cindex tunables, non_temporal_threshold > > @deftp {Tunable namespace} glibc.tune > Behavior of @theglibc{} can be tuned to assume specific hardware capabilities > @@ -213,3 +221,31 @@ extensions available in the processor at runtime for some architectures. The > @code{glibc.tune.hwcap_mask} tunable allows the user to mask out those > capabilities at runtime, thus disabling use of those extensions. > @end deftp > + > +@deftp Tunable glibc.tune.ifunc > +The @code{glibc.tune.ifunc=-xxx,yyy,-zzz...} tunable allows the user to > +enable CPU/ARCH feature @code{yyy}, disable CPU/ARCH feature @code{xxx} > +and @code{zzz} where the feature name is case-sensitive and has to match > +the ones in @code{sysdeps/x86/cpu-features.h}. > + > +This tunable is specific to i386 and x86-64. > +@end deftp > + > +@deftp Tunable glibc.tune.data_cache_size > +The @code{glibc.tune.data_cache_size} tunable allows the user to set > +data cache size in bytes for use in memory and string routines. > + > +This tunable is specific to i386 and x86-64. > +@end deftp > + > +@deftp Tunable glibc.tune.shared_cache_size > +The @code{glibc.tune.shared_cache_size} tunable allows the user to set > +shared cache size in bytes for use in memory and string routines. > +@end deftp > + > +@deftp Tunable glibc.tune.non_temporal_threshold > +The @code{glibc.tune.non_temporal_threshold} tunable allows the user to > +set threshold in bytes for non temporal store. > + > +This tunable is specific to i386 and x86-64. > +@end deftp > diff --git a/sysdeps/unix/sysv/linux/x86/dl-sysdep.c b/sysdeps/unix/sysv/linux/x86/dl-sysdep.c > new file mode 100644 > index 0000000..64eb0d7 > --- /dev/null > +++ b/sysdeps/unix/sysv/linux/x86/dl-sysdep.c > @@ -0,0 +1,21 @@ > +/* Operating system support for run-time dynamic linker. X86 version. > + Copyright (C) 2017 Free Software Foundation, Inc. > + This file is part of the GNU C Library. > + > + The GNU C Library is free software; you can redistribute it and/or > + modify it under the terms of the GNU Lesser General Public > + License as published by the Free Software Foundation; either > + version 2.1 of the License, or (at your option) any later version. > + > + The GNU C Library is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + Lesser General Public License for more details. > + > + You should have received a copy of the GNU Lesser General Public > + License along with the GNU C Library; if not, see > + <http://www.gnu.org/licenses/>. */ > + > +#include <config.h> > +#include <sysdeps/x86/cpu-tunables.c> > +#include <sysdeps/unix/sysv/linux/dl-sysdep.c> > diff --git a/sysdeps/x86/cacheinfo.c b/sysdeps/x86/cacheinfo.c > index 8000fd1..9542dfb 100644 > --- a/sysdeps/x86/cacheinfo.c > +++ b/sysdeps/x86/cacheinfo.c > @@ -752,6 +752,9 @@ intel_bug_no_cache_info: > #endif > } > > + if (cpu_features->data_cache_size != 0) > + data = cpu_features->data_cache_size; > + > if (data > 0) > { > __x86_raw_data_cache_size_half = data / 2; > @@ -762,6 +765,9 @@ intel_bug_no_cache_info: > __x86_data_cache_size = data; > } > > + if (cpu_features->shared_cache_size != 0) > + shared = cpu_features->shared_cache_size; > + > if (shared > 0) > { > __x86_raw_shared_cache_size_half = shared / 2; > @@ -777,7 +783,9 @@ intel_bug_no_cache_info: > store becomes faster on a 8-core processor. This is the 3/4 of the > total shared cache size. */ > __x86_shared_non_temporal_threshold > - = __x86_shared_cache_size * threads * 3 / 4; > + = (cpu_features->non_temporal_threshold != 0 > + ? cpu_features->non_temporal_threshold > + : __x86_shared_cache_size * threads * 3 / 4); > } > > #endif > diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c > index 4288001..1ca0310 100644 > --- a/sysdeps/x86/cpu-features.c > +++ b/sysdeps/x86/cpu-features.c > @@ -20,6 +20,15 @@ > #include <cpu-features.h> > #include <dl-hwcap.h> > > +#if HAVE_TUNABLES > +# define TUNABLE_NAMESPACE tune > +# include <unistd.h> /* Get STDOUT_FILENO for _dl_printf. */ > +# include <elf/dl-tunables.h> > + > +extern void TUNABLE_CALLBACK (set_ifunc) (tunable_val_t *) > + attribute_hidden; > +#endif > + > static void > get_common_indeces (struct cpu_features *cpu_features, > unsigned int *family, unsigned int *model, > @@ -312,6 +321,16 @@ no_cpuid: > cpu_features->model = model; > cpu_features->kind = kind; > > +#if HAVE_TUNABLES > + TUNABLE_GET (ifunc, tunable_val_t *, TUNABLE_CALLBACK (set_ifunc)); > + cpu_features->non_temporal_threshold > + = TUNABLE_GET (non_temporal_threshold, long int, NULL); > + cpu_features->data_cache_size > + = TUNABLE_GET (data_cache_size, long int, NULL); > + cpu_features->shared_cache_size > + = TUNABLE_GET (shared_cache_size, long int, NULL); > +#endif > + > /* Reuse dl_platform, dl_hwcap and dl_hwcap_mask for x86. */ > GLRO(dl_platform) = NULL; > GLRO(dl_hwcap) = 0; > diff --git a/sysdeps/x86/cpu-features.h b/sysdeps/x86/cpu-features.h > index f2329ea..fef5e18 100644 > --- a/sysdeps/x86/cpu-features.h > +++ b/sysdeps/x86/cpu-features.h > @@ -215,6 +215,14 @@ struct cpu_features > unsigned int family; > unsigned int model; > unsigned int feature[FEATURE_INDEX_MAX]; > + /* Data cache size for use in memory and string routines, typically > + L1 size. */ > + long int data_cache_size; > + /* Shared cache size for use in memory and string routines, typically > + L2 or L3 size. */ > + long int shared_cache_size; > + /* Threshold to use non temporal store. */ > + long int non_temporal_threshold; > }; > > /* Used from outside of glibc to get access to the CPU features > diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c > new file mode 100644 > index 0000000..dfa3647 > --- /dev/null > +++ b/sysdeps/x86/cpu-tunables.c > @@ -0,0 +1,330 @@ > +/* x86 CPU feature tuning. > + This file is part of the GNU C Library. > + Copyright (C) 2017 Free Software Foundation, Inc. > + > + The GNU C Library is free software; you can redistribute it and/or > + modify it under the terms of the GNU Lesser General Public > + License as published by the Free Software Foundation; either > + version 2.1 of the License, or (at your option) any later version. > + > + The GNU C Library is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > + Lesser General Public License for more details. > + > + You should have received a copy of the GNU Lesser General Public > + License along with the GNU C Library; if not, see > + <http://www.gnu.org/licenses/>. */ > + > +#if HAVE_TUNABLES > +# define TUNABLE_NAMESPACE tune > +# include <stdbool.h> > +# include <stdint.h> > +# include <unistd.h> /* Get STDOUT_FILENO for _dl_printf. */ > +# include <elf/dl-tunables.h> > +# include <string.h> > +# include <cpu-features.h> > +# include <ldsodefs.h> > + > +/* We can't use IFUNC memcmp nor strlen in init_cpu_features from libc.a > + since IFUNC must be set up by init_cpu_features. */ > +# if defined USE_MULTIARCH && !defined SHARED > +# ifdef __x86_64__ > +# define DEFAULT_MEMCMP __memcmp_sse2 > +# define DEFAULT_STRLEN __strlen_sse2 > +extern __typeof (strlen) DEFAULT_STRLEN; > +# else > +# define DEFAULT_MEMCMP __memcmp_ia32 > +# define DEFAULT_STRLEN strlen > +# endif > +extern __typeof (memcmp) DEFAULT_MEMCMP; > +# else > +# define DEFAULT_MEMCMP memcmp > +# define DEFAULT_STRLEN strlen > +# endif > + > +# define CHECK_GLIBC_IFUNC_CPU_OFF(f, cpu_features, name, len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + cpu_features->cpuid[index_cpu_##name].reg_##name \ > + &= ~bit_cpu_##name; \ > + break; \ > + } > + > +/* Disable an ARCH feature NAME. We don't enable an ARCH feature which > + isn't available. */ > +# define CHECK_GLIBC_IFUNC_ARCH_OFF(f, cpu_features, name, len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + cpu_features->feature[index_arch_##name] \ > + &= ~bit_arch_##name; \ > + break; \ > + } > + > +/* Enable/disable an ARCH feature NAME. */ > +# define CHECK_GLIBC_IFUNC_ARCH_BOTH(f, cpu_features, name, disable, \ > + len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + if (disable) \ > + cpu_features->feature[index_arch_##name] \ > + &= ~bit_arch_##name; \ > + else \ > + cpu_features->feature[index_arch_##name] \ > + |= bit_arch_##name; \ > + break; \ > + } > + > +/* Enable/disable an ARCH feature NAME. Enable an ARCH feature only > + if the ARCH feature NEED is also enabled. */ > +# define CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH(f, cpu_features, name, \ > + need, disable, len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + if (disable) \ > + cpu_features->feature[index_arch_##name] \ > + &= ~bit_arch_##name; \ > + else if (CPU_FEATURES_ARCH_P (cpu_features, need)) \ > + cpu_features->feature[index_arch_##name] \ > + |= bit_arch_##name; \ > + break; \ > + } > + > +/* Enable/disable an ARCH feature NAME. Enable an ARCH feature only > + if the CPU feature NEED is also enabled. */ > +# define CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH(f, cpu_features, name, \ > + need, disable, len) \ > + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ > + if (!DEFAULT_MEMCMP (f, #name, len)) \ > + { \ > + if (disable) \ > + cpu_features->feature[index_arch_##name] \ > + &= ~bit_arch_##name; \ > + else if (CPU_FEATURES_CPU_P (cpu_features, need)) \ > + cpu_features->feature[index_arch_##name] \ > + |= bit_arch_##name; \ > + break; \ > + } > + > +attribute_hidden > +void > +TUNABLE_CALLBACK (set_ifunc) (tunable_val_t *valp) > +{ > + /* The current IFUNC selection is based on microbenchmarks in glibc. > + It should give the best performance for most workloads. But other > + choices may have better performance for a particular workload or on > + the hardware which wasn't available when the selection was made. > + The environment variable, GLIBC_IFUNC=-xxx,yyy,-zzz...., can be > + used to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy > + and zzz, where the feature name is case-sensitive and has to match > + the ones in cpu-features.h. It can be used by glibc developers to > + tune for a new processor or override the IFUNC selection to improve > + performance for a particular workload. > + > + Since all CPU/ARCH features are hardware optimizations without > + security implication, except for Prefer_MAP_32BIT_EXEC, which can > + only be disabled, we check GLIBC_IFUNC for programs, including > + set*id ones. > + > + NOTE: the IFUNC selection may change over time. Please check all > + multiarch implementations when experimenting. */ > + > + const char *p = valp->strval; > + struct cpu_features *cpu_features = &GLRO(dl_x86_cpu_features); > + const char *end = p + DEFAULT_STRLEN (p); > + size_t len; > + > + do > + { > + const char *c, *n; > + bool disable; > + size_t nl; > + > + for (c = p; *c != ','; c++) > + if (c >= end) > + break; > + > + len = c - p; > + disable = *p == '-'; > + if (disable) > + { > + n = p + 1; > + nl = len - 1; > + } > + else > + { > + n = p; > + nl = len; > + } > + switch (nl) > + { > + default: > + break; > + case 3: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX, 3); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, CX8, 3); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, FMA, 3); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, HTT, 3); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, RTM, 3); > + } > + break; > + case 4: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX2, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, BMI1, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, BMI2, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, CMOV, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, ERMS, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, FMA4, 4); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE2, 4); > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, I586, 4); > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, I686, 4); > + } > + break; > + case 5: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, LZCNT, 5); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, MOVBE, 5); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSSE3, 5); > + } > + break; > + case 6: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, POPCNT, 6); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_1, 6); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_2, 6); > + } > + break; > + case 7: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512F, 7); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, OSXSAVE, 7); > + } > + break; > + case 8: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512CD, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512BW, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512DQ, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512ER, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512PF, 8); > + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512VL, 8); > + } > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Slow_BSF, > + disable, 8); > + break; > + case 10: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, AVX_Usable, > + 10); > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, FMA_Usable, > + 10); > + } > + break; > + case 11: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, AVX2_Usable, > + 11); > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, FMA4_Usable, > + 11); > + } > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Prefer_ERMS, > + disable, 11); > + CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH (n, cpu_features, > + Slow_SSE4_2, SSE4_2, > + disable, 11); > + break; > + case 14: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, > + AVX512F_Usable, 14); > + } > + break; > + case 15: > + if (disable) > + { > + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, > + AVX512DQ_Usable, 15); > + } > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Fast_Rep_String, > + disable, 15); > + break; > + case 16: > + { > + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH > + (n, cpu_features, Prefer_No_AVX512, AVX512F_Usable, > + disable, 16); > + } > + break; > + case 18: > + { > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Fast_Copy_Backward, disable, > + 18); > + } > + break; > + case 19: > + { > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Fast_Unaligned_Load, disable, > + 19); > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Fast_Unaligned_Copy, disable, > + 19); > + } > + break; > + case 20: > + { > + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH > + (n, cpu_features, Prefer_No_VZEROUPPER, AVX_Usable, > + disable, 20); > + } > + break; > + case 21: > + { > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Prefer_MAP_32BIT_EXEC, disable, > + 21); > + } > + break; > + case 23: > + { > + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH > + (n, cpu_features, AVX_Fast_Unaligned_Load, AVX_Usable, > + disable, 23); > + } > + break; > + case 26: > + { > + CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH > + (n, cpu_features, Prefer_PMINUB_for_stringop, SSE2, > + disable, 26); > + } > + break; > + case 27: > + { > + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, > + Use_dl_runtime_resolve_slow, > + disable, 27); > + } > + break; > + } > + p += len + 1; > + } > + while (p < end); > +} > +#endif >
On Tue, Jun 20, 2017 at 6:23 AM, Adhemerval Zanella <adhemerval.zanella@linaro.org> wrote: > > > On 15/06/2017 10:10, H.J. Lu wrote: >> The current IFUNC selection is based on microbenchmarks in glibc. It >> should give the best performance for most workloads. But other choices >> may have better performance for a particular workload or on the hardware >> which wasn't available at the selection was made. The environment >> variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used >> to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz, >> where the feature name is case-sensitive and has to match the ones in >> cpu-features.h. It can be used by glibc developers to override the >> IFUNC selection to tune for a new processor or improve performance for >> a particular workload. It isn't intended for normal end users. >> >> NOTE: the IFUNC selection may change over time. Please check all >> multiarch implementations when experimenting. >> >> Also, GLIBC_TUNABLES=glibc.tune.non_temporal_threshold=NUMBER is >> provided to set threshold to use non temporal store to NUMBER, >> GLIBC_TUNABLES=glibc.tune.data_cache_size=NUMBER to set data cache size, >> GLIBC_TUNABLES=glibc.tune.shared_cache_size=NUMBER to set shared cache >> size. >> >> Any comments? >> >> H.J. >> --- >> 2017-06-15 H.J. Lu <hongjiu.lu@intel.com> >> Erich Elsen <eriche@google.com> >> >> * elf/dl-tunables.list (tune): Add ifunc, non_temporal_threshold, >> data_cache_size and shared_cache_size. >> * manual/tunables.texi: Document glibc.tune.ifunc, >> glibc.tune.data_cache_size, glibc.tune.shared_cache_size and >> glibc.tune.non_temporal_threshold. >> * sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file. >> * sysdeps/x86/cpu-tunables.c: Likewise. >> * sysdeps/x86/cacheinfo.c >> (init_cacheinfo): Check and get data cache size, shared cache >> size and non temporal threshold from cpu_features. >> * sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE): >> New. >> [HAVE_TUNABLES] Include <unistd.h>. >> [HAVE_TUNABLES] Include <elf/dl-tunables.h>. >> [HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise. >> [HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set >> IFUNC selection, data cache size, shared cache size and non >> temporal threshold. >> * sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size, >> shared_cache_size and non_temporal_threshold. >> --- >> elf/dl-tunables.list | 16 ++ >> manual/tunables.texi | 36 ++++ >> sysdeps/unix/sysv/linux/x86/dl-sysdep.c | 21 ++ >> sysdeps/x86/cacheinfo.c | 10 +- >> sysdeps/x86/cpu-features.c | 19 ++ >> sysdeps/x86/cpu-features.h | 8 + >> sysdeps/x86/cpu-tunables.c | 330 ++++++++++++++++++++++++++++++++ >> 7 files changed, 439 insertions(+), 1 deletion(-) >> create mode 100644 sysdeps/unix/sysv/linux/x86/dl-sysdep.c >> create mode 100644 sysdeps/x86/cpu-tunables.c >> >> diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list >> index 41ce9af..78354fb 100644 >> --- a/elf/dl-tunables.list >> +++ b/elf/dl-tunables.list >> @@ -82,6 +82,22 @@ glibc { >> type: UINT_64 >> env_alias: LD_HWCAP_MASK >> default: HWCAP_IMPORTANT >> + } >> + ifunc { >> + type: STRING >> + security_level: SXID_IGNORE >> + } >> + non_temporal_threshold { >> + type: SIZE_T >> + security_level: SXID_IGNORE >> + } >> + data_cache_size { >> + type: SIZE_T >> + security_level: SXID_IGNORE >> + } >> + shared_cache_size { >> + type: SIZE_T >> + security_level: SXID_IGNORE >> } > > Is it possible with current tunables approach to make it arch-specific? The > 'ifunc' switch seems a generic one, but the 'non_temporal_threshold', > 'data_cache_size', and 'shared_cache_size' are very x86 specific and I see > it confusing exposing them to non-x86 architectures. Yes, it can be made x86 specific. I will update my patch shortly. Thanks. H.J.
On Tue, Jun 20, 2017 at 6:28 AM, H.J. Lu <hjl.tools@gmail.com> wrote: > On Tue, Jun 20, 2017 at 6:23 AM, Adhemerval Zanella > <adhemerval.zanella@linaro.org> wrote: >> >> >> On 15/06/2017 10:10, H.J. Lu wrote: >>> The current IFUNC selection is based on microbenchmarks in glibc. It >>> should give the best performance for most workloads. But other choices >>> may have better performance for a particular workload or on the hardware >>> which wasn't available at the selection was made. The environment >>> variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used >>> to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz, >>> where the feature name is case-sensitive and has to match the ones in >>> cpu-features.h. It can be used by glibc developers to override the >>> IFUNC selection to tune for a new processor or improve performance for >>> a particular workload. It isn't intended for normal end users. >>> >>> NOTE: the IFUNC selection may change over time. Please check all >>> multiarch implementations when experimenting. >>> >>> Also, GLIBC_TUNABLES=glibc.tune.non_temporal_threshold=NUMBER is >>> provided to set threshold to use non temporal store to NUMBER, >>> GLIBC_TUNABLES=glibc.tune.data_cache_size=NUMBER to set data cache size, >>> GLIBC_TUNABLES=glibc.tune.shared_cache_size=NUMBER to set shared cache >>> size. >>> >>> Any comments? >>> >>> H.J. >>> --- >>> 2017-06-15 H.J. Lu <hongjiu.lu@intel.com> >>> Erich Elsen <eriche@google.com> >>> >>> * elf/dl-tunables.list (tune): Add ifunc, non_temporal_threshold, >>> data_cache_size and shared_cache_size. >>> * manual/tunables.texi: Document glibc.tune.ifunc, >>> glibc.tune.data_cache_size, glibc.tune.shared_cache_size and >>> glibc.tune.non_temporal_threshold. >>> * sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file. >>> * sysdeps/x86/cpu-tunables.c: Likewise. >>> * sysdeps/x86/cacheinfo.c >>> (init_cacheinfo): Check and get data cache size, shared cache >>> size and non temporal threshold from cpu_features. >>> * sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE): >>> New. >>> [HAVE_TUNABLES] Include <unistd.h>. >>> [HAVE_TUNABLES] Include <elf/dl-tunables.h>. >>> [HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise. >>> [HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set >>> IFUNC selection, data cache size, shared cache size and non >>> temporal threshold. >>> * sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size, >>> shared_cache_size and non_temporal_threshold. >>> --- >>> elf/dl-tunables.list | 16 ++ >>> manual/tunables.texi | 36 ++++ >>> sysdeps/unix/sysv/linux/x86/dl-sysdep.c | 21 ++ >>> sysdeps/x86/cacheinfo.c | 10 +- >>> sysdeps/x86/cpu-features.c | 19 ++ >>> sysdeps/x86/cpu-features.h | 8 + >>> sysdeps/x86/cpu-tunables.c | 330 ++++++++++++++++++++++++++++++++ >>> 7 files changed, 439 insertions(+), 1 deletion(-) >>> create mode 100644 sysdeps/unix/sysv/linux/x86/dl-sysdep.c >>> create mode 100644 sysdeps/x86/cpu-tunables.c >>> >>> diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list >>> index 41ce9af..78354fb 100644 >>> --- a/elf/dl-tunables.list >>> +++ b/elf/dl-tunables.list >>> @@ -82,6 +82,22 @@ glibc { >>> type: UINT_64 >>> env_alias: LD_HWCAP_MASK >>> default: HWCAP_IMPORTANT >>> + } >>> + ifunc { >>> + type: STRING >>> + security_level: SXID_IGNORE >>> + } >>> + non_temporal_threshold { >>> + type: SIZE_T >>> + security_level: SXID_IGNORE >>> + } >>> + data_cache_size { >>> + type: SIZE_T >>> + security_level: SXID_IGNORE >>> + } >>> + shared_cache_size { >>> + type: SIZE_T >>> + security_level: SXID_IGNORE >>> } >> >> Is it possible with current tunables approach to make it arch-specific? The >> 'ifunc' switch seems a generic one, but the 'non_temporal_threshold', >> 'data_cache_size', and 'shared_cache_size' are very x86 specific and I see >> it confusing exposing them to non-x86 architectures. > > Yes, it can be made x86 specific. I will update my patch shortly. > Since ifunc and others are handled in the same name space, I have to make non_temporal_threshold, .. as well as ifunc x86 specific. Will it be OK?
On Tuesday 20 June 2017 07:11 PM, H.J. Lu wrote: > Since ifunc and others are handled in the same name space, I have to > make non_temporal_threshold, .. as well as ifunc x86 specific. Will it be > OK? I'm going to add ifunc selection for aarch64 soon, so it makes sense to have that generic. You could name the arch-specific tunables as x86_non_temporal_threshold and so on. Siddhesh
-----Original Message----- From: <libc-alpha-owner@sourceware.org> on behalf of Siddhesh Poyarekar <siddhesh@gotplt.org> Date: Tuesday, June 20, 2017 at 9:36 AM To: "H.J. Lu" <hjl.tools@gmail.com>, Adhemerval Zanella <adhemerval.zanella@linaro.org> Cc: GNU C Library <libc-alpha@sourceware.org> Subject: Re: [PATCH] tunables: Add IFUNC selection and cache sizes >On Tuesday 20 June 2017 07:11 PM, H.J. Lu wrote: >> Since ifunc and others are handled in the same name space, I have to >> make non_temporal_threshold, .. as well as ifunc x86 specific. Will it >>be >> OK? > >I'm going to add ifunc selection for aarch64 soon, so it makes sense to >have that generic. You could name the arch-specific tunables as >x86_non_temporal_threshold and so on. What does it mean arch-specific? ( haswell / skyline / ... ) Victor > >Siddhesh
On Tue, Jun 20, 2017 at 11:36 AM, Rodriguez Bahena, Victor <victor.rodriguez.bahena@intel.com> wrote: > -----Original Message----- > > From: <libc-alpha-owner@sourceware.org> on behalf of Siddhesh Poyarekar > <siddhesh@gotplt.org> > Date: Tuesday, June 20, 2017 at 9:36 AM > To: "H.J. Lu" <hjl.tools@gmail.com>, Adhemerval Zanella > <adhemerval.zanella@linaro.org> > Cc: GNU C Library <libc-alpha@sourceware.org> > Subject: Re: [PATCH] tunables: Add IFUNC selection and cache sizes > >>On Tuesday 20 June 2017 07:11 PM, H.J. Lu wrote: >>> Since ifunc and others are handled in the same name space, I have to >>> make non_temporal_threshold, .. as well as ifunc x86 specific. Will it >>>be >>> OK? >> >>I'm going to add ifunc selection for aarch64 soon, so it makes sense to >>have that generic. You could name the arch-specific tunables as >>x86_non_temporal_threshold and so on. > > What does it mean arch-specific? ( haswell / skyline / ... ) > It is x86 vs aarch64.
diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list index 41ce9af..78354fb 100644 --- a/elf/dl-tunables.list +++ b/elf/dl-tunables.list @@ -82,6 +82,22 @@ glibc { type: UINT_64 env_alias: LD_HWCAP_MASK default: HWCAP_IMPORTANT + } + ifunc { + type: STRING + security_level: SXID_IGNORE + } + non_temporal_threshold { + type: SIZE_T + security_level: SXID_IGNORE + } + data_cache_size { + type: SIZE_T + security_level: SXID_IGNORE + } + shared_cache_size { + type: SIZE_T + security_level: SXID_IGNORE } } } diff --git a/manual/tunables.texi b/manual/tunables.texi index c9a4cb7..709f58a 100644 --- a/manual/tunables.texi +++ b/manual/tunables.texi @@ -198,6 +198,14 @@ is 8 times the number of cores online. @cindex hardware capability tunables @cindex hwcap tunables @cindex tunables, hwcap +@cindex ifunc tunables +@cindex tunables, ifunc +@cindex data_cache_size tunables +@cindex tunables, data_cache_size +@cindex shared_cache_size tunables +@cindex tunables, shared_cache_size +@cindex non_temporal_threshold tunables +@cindex tunables, non_temporal_threshold @deftp {Tunable namespace} glibc.tune Behavior of @theglibc{} can be tuned to assume specific hardware capabilities @@ -213,3 +221,31 @@ extensions available in the processor at runtime for some architectures. The @code{glibc.tune.hwcap_mask} tunable allows the user to mask out those capabilities at runtime, thus disabling use of those extensions. @end deftp + +@deftp Tunable glibc.tune.ifunc +The @code{glibc.tune.ifunc=-xxx,yyy,-zzz...} tunable allows the user to +enable CPU/ARCH feature @code{yyy}, disable CPU/ARCH feature @code{xxx} +and @code{zzz} where the feature name is case-sensitive and has to match +the ones in @code{sysdeps/x86/cpu-features.h}. + +This tunable is specific to i386 and x86-64. +@end deftp + +@deftp Tunable glibc.tune.data_cache_size +The @code{glibc.tune.data_cache_size} tunable allows the user to set +data cache size in bytes for use in memory and string routines. + +This tunable is specific to i386 and x86-64. +@end deftp + +@deftp Tunable glibc.tune.shared_cache_size +The @code{glibc.tune.shared_cache_size} tunable allows the user to set +shared cache size in bytes for use in memory and string routines. +@end deftp + +@deftp Tunable glibc.tune.non_temporal_threshold +The @code{glibc.tune.non_temporal_threshold} tunable allows the user to +set threshold in bytes for non temporal store. + +This tunable is specific to i386 and x86-64. +@end deftp diff --git a/sysdeps/unix/sysv/linux/x86/dl-sysdep.c b/sysdeps/unix/sysv/linux/x86/dl-sysdep.c new file mode 100644 index 0000000..64eb0d7 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86/dl-sysdep.c @@ -0,0 +1,21 @@ +/* Operating system support for run-time dynamic linker. X86 version. + Copyright (C) 2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + <http://www.gnu.org/licenses/>. */ + +#include <config.h> +#include <sysdeps/x86/cpu-tunables.c> +#include <sysdeps/unix/sysv/linux/dl-sysdep.c> diff --git a/sysdeps/x86/cacheinfo.c b/sysdeps/x86/cacheinfo.c index 8000fd1..9542dfb 100644 --- a/sysdeps/x86/cacheinfo.c +++ b/sysdeps/x86/cacheinfo.c @@ -752,6 +752,9 @@ intel_bug_no_cache_info: #endif } + if (cpu_features->data_cache_size != 0) + data = cpu_features->data_cache_size; + if (data > 0) { __x86_raw_data_cache_size_half = data / 2; @@ -762,6 +765,9 @@ intel_bug_no_cache_info: __x86_data_cache_size = data; } + if (cpu_features->shared_cache_size != 0) + shared = cpu_features->shared_cache_size; + if (shared > 0) { __x86_raw_shared_cache_size_half = shared / 2; @@ -777,7 +783,9 @@ intel_bug_no_cache_info: store becomes faster on a 8-core processor. This is the 3/4 of the total shared cache size. */ __x86_shared_non_temporal_threshold - = __x86_shared_cache_size * threads * 3 / 4; + = (cpu_features->non_temporal_threshold != 0 + ? cpu_features->non_temporal_threshold + : __x86_shared_cache_size * threads * 3 / 4); } #endif diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c index 4288001..1ca0310 100644 --- a/sysdeps/x86/cpu-features.c +++ b/sysdeps/x86/cpu-features.c @@ -20,6 +20,15 @@ #include <cpu-features.h> #include <dl-hwcap.h> +#if HAVE_TUNABLES +# define TUNABLE_NAMESPACE tune +# include <unistd.h> /* Get STDOUT_FILENO for _dl_printf. */ +# include <elf/dl-tunables.h> + +extern void TUNABLE_CALLBACK (set_ifunc) (tunable_val_t *) + attribute_hidden; +#endif + static void get_common_indeces (struct cpu_features *cpu_features, unsigned int *family, unsigned int *model, @@ -312,6 +321,16 @@ no_cpuid: cpu_features->model = model; cpu_features->kind = kind; +#if HAVE_TUNABLES + TUNABLE_GET (ifunc, tunable_val_t *, TUNABLE_CALLBACK (set_ifunc)); + cpu_features->non_temporal_threshold + = TUNABLE_GET (non_temporal_threshold, long int, NULL); + cpu_features->data_cache_size + = TUNABLE_GET (data_cache_size, long int, NULL); + cpu_features->shared_cache_size + = TUNABLE_GET (shared_cache_size, long int, NULL); +#endif + /* Reuse dl_platform, dl_hwcap and dl_hwcap_mask for x86. */ GLRO(dl_platform) = NULL; GLRO(dl_hwcap) = 0; diff --git a/sysdeps/x86/cpu-features.h b/sysdeps/x86/cpu-features.h index f2329ea..fef5e18 100644 --- a/sysdeps/x86/cpu-features.h +++ b/sysdeps/x86/cpu-features.h @@ -215,6 +215,14 @@ struct cpu_features unsigned int family; unsigned int model; unsigned int feature[FEATURE_INDEX_MAX]; + /* Data cache size for use in memory and string routines, typically + L1 size. */ + long int data_cache_size; + /* Shared cache size for use in memory and string routines, typically + L2 or L3 size. */ + long int shared_cache_size; + /* Threshold to use non temporal store. */ + long int non_temporal_threshold; }; /* Used from outside of glibc to get access to the CPU features diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c new file mode 100644 index 0000000..dfa3647 --- /dev/null +++ b/sysdeps/x86/cpu-tunables.c @@ -0,0 +1,330 @@ +/* x86 CPU feature tuning. + This file is part of the GNU C Library. + Copyright (C) 2017 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + <http://www.gnu.org/licenses/>. */ + +#if HAVE_TUNABLES +# define TUNABLE_NAMESPACE tune +# include <stdbool.h> +# include <stdint.h> +# include <unistd.h> /* Get STDOUT_FILENO for _dl_printf. */ +# include <elf/dl-tunables.h> +# include <string.h> +# include <cpu-features.h> +# include <ldsodefs.h> + +/* We can't use IFUNC memcmp nor strlen in init_cpu_features from libc.a + since IFUNC must be set up by init_cpu_features. */ +# if defined USE_MULTIARCH && !defined SHARED +# ifdef __x86_64__ +# define DEFAULT_MEMCMP __memcmp_sse2 +# define DEFAULT_STRLEN __strlen_sse2 +extern __typeof (strlen) DEFAULT_STRLEN; +# else +# define DEFAULT_MEMCMP __memcmp_ia32 +# define DEFAULT_STRLEN strlen +# endif +extern __typeof (memcmp) DEFAULT_MEMCMP; +# else +# define DEFAULT_MEMCMP memcmp +# define DEFAULT_STRLEN strlen +# endif + +# define CHECK_GLIBC_IFUNC_CPU_OFF(f, cpu_features, name, len) \ + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ + if (!DEFAULT_MEMCMP (f, #name, len)) \ + { \ + cpu_features->cpuid[index_cpu_##name].reg_##name \ + &= ~bit_cpu_##name; \ + break; \ + } + +/* Disable an ARCH feature NAME. We don't enable an ARCH feature which + isn't available. */ +# define CHECK_GLIBC_IFUNC_ARCH_OFF(f, cpu_features, name, len) \ + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ + if (!DEFAULT_MEMCMP (f, #name, len)) \ + { \ + cpu_features->feature[index_arch_##name] \ + &= ~bit_arch_##name; \ + break; \ + } + +/* Enable/disable an ARCH feature NAME. */ +# define CHECK_GLIBC_IFUNC_ARCH_BOTH(f, cpu_features, name, disable, \ + len) \ + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ + if (!DEFAULT_MEMCMP (f, #name, len)) \ + { \ + if (disable) \ + cpu_features->feature[index_arch_##name] \ + &= ~bit_arch_##name; \ + else \ + cpu_features->feature[index_arch_##name] \ + |= bit_arch_##name; \ + break; \ + } + +/* Enable/disable an ARCH feature NAME. Enable an ARCH feature only + if the ARCH feature NEED is also enabled. */ +# define CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH(f, cpu_features, name, \ + need, disable, len) \ + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ + if (!DEFAULT_MEMCMP (f, #name, len)) \ + { \ + if (disable) \ + cpu_features->feature[index_arch_##name] \ + &= ~bit_arch_##name; \ + else if (CPU_FEATURES_ARCH_P (cpu_features, need)) \ + cpu_features->feature[index_arch_##name] \ + |= bit_arch_##name; \ + break; \ + } + +/* Enable/disable an ARCH feature NAME. Enable an ARCH feature only + if the CPU feature NEED is also enabled. */ +# define CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH(f, cpu_features, name, \ + need, disable, len) \ + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ + if (!DEFAULT_MEMCMP (f, #name, len)) \ + { \ + if (disable) \ + cpu_features->feature[index_arch_##name] \ + &= ~bit_arch_##name; \ + else if (CPU_FEATURES_CPU_P (cpu_features, need)) \ + cpu_features->feature[index_arch_##name] \ + |= bit_arch_##name; \ + break; \ + } + +attribute_hidden +void +TUNABLE_CALLBACK (set_ifunc) (tunable_val_t *valp) +{ + /* The current IFUNC selection is based on microbenchmarks in glibc. + It should give the best performance for most workloads. But other + choices may have better performance for a particular workload or on + the hardware which wasn't available when the selection was made. + The environment variable, GLIBC_IFUNC=-xxx,yyy,-zzz...., can be + used to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy + and zzz, where the feature name is case-sensitive and has to match + the ones in cpu-features.h. It can be used by glibc developers to + tune for a new processor or override the IFUNC selection to improve + performance for a particular workload. + + Since all CPU/ARCH features are hardware optimizations without + security implication, except for Prefer_MAP_32BIT_EXEC, which can + only be disabled, we check GLIBC_IFUNC for programs, including + set*id ones. + + NOTE: the IFUNC selection may change over time. Please check all + multiarch implementations when experimenting. */ + + const char *p = valp->strval; + struct cpu_features *cpu_features = &GLRO(dl_x86_cpu_features); + const char *end = p + DEFAULT_STRLEN (p); + size_t len; + + do + { + const char *c, *n; + bool disable; + size_t nl; + + for (c = p; *c != ','; c++) + if (c >= end) + break; + + len = c - p; + disable = *p == '-'; + if (disable) + { + n = p + 1; + nl = len - 1; + } + else + { + n = p; + nl = len; + } + switch (nl) + { + default: + break; + case 3: + if (disable) + { + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX, 3); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, CX8, 3); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, FMA, 3); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, HTT, 3); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, RTM, 3); + } + break; + case 4: + if (disable) + { + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX2, 4); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, BMI1, 4); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, BMI2, 4); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, CMOV, 4); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, ERMS, 4); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, FMA4, 4); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE2, 4); + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, I586, 4); + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, I686, 4); + } + break; + case 5: + if (disable) + { + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, LZCNT, 5); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, MOVBE, 5); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSSE3, 5); + } + break; + case 6: + if (disable) + { + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, POPCNT, 6); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_1, 6); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_2, 6); + } + break; + case 7: + if (disable) + { + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512F, 7); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, OSXSAVE, 7); + } + break; + case 8: + if (disable) + { + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512CD, 8); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512BW, 8); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512DQ, 8); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512ER, 8); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512PF, 8); + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512VL, 8); + } + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Slow_BSF, + disable, 8); + break; + case 10: + if (disable) + { + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, AVX_Usable, + 10); + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, FMA_Usable, + 10); + } + break; + case 11: + if (disable) + { + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, AVX2_Usable, + 11); + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, FMA4_Usable, + 11); + } + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Prefer_ERMS, + disable, 11); + CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH (n, cpu_features, + Slow_SSE4_2, SSE4_2, + disable, 11); + break; + case 14: + if (disable) + { + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, + AVX512F_Usable, 14); + } + break; + case 15: + if (disable) + { + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, + AVX512DQ_Usable, 15); + } + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Fast_Rep_String, + disable, 15); + break; + case 16: + { + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH + (n, cpu_features, Prefer_No_AVX512, AVX512F_Usable, + disable, 16); + } + break; + case 18: + { + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, + Fast_Copy_Backward, disable, + 18); + } + break; + case 19: + { + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, + Fast_Unaligned_Load, disable, + 19); + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, + Fast_Unaligned_Copy, disable, + 19); + } + break; + case 20: + { + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH + (n, cpu_features, Prefer_No_VZEROUPPER, AVX_Usable, + disable, 20); + } + break; + case 21: + { + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, + Prefer_MAP_32BIT_EXEC, disable, + 21); + } + break; + case 23: + { + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH + (n, cpu_features, AVX_Fast_Unaligned_Load, AVX_Usable, + disable, 23); + } + break; + case 26: + { + CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH + (n, cpu_features, Prefer_PMINUB_for_stringop, SSE2, + disable, 26); + } + break; + case 27: + { + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, + Use_dl_runtime_resolve_slow, + disable, 27); + } + break; + } + p += len + 1; + } + while (p < end); +} +#endif