| Message ID | CAH6eHdQ=EPO3cVqrKdH_rOLtRN2KJMJk5_Kw=sATqrVFSRJpPA@mail.gmail.com |
|---|---|
| State | New |
| Headers | show |
On 7 Nov 2011, at 13:45, Jonathan Wakely wrote: > This provides a working thread::hardware_concurrency on platforms that > support pthread_num_processors_np or the "hw.ncpu" sysctl, but by > testing for the features in configure rather than hardcoding OS macro > tests in thread.cc if the system supports run-time User-controllable core-count then this will not account for that. > > * acinclude.m4 (GLIBCXX_CHECK_SC_NPROC_ONLN): Define. > (GLIBCXX_CHECK_PTHREADS_NUM_PROCESSORS_NP): Define. > (GLIBCXX_CHECK_SYSCTL_HW_NCPU): Define. > * configure.ac: Use new checks. > * configure: Regenerate. > * config.h.in: Regenerate. > * src/thread.cc: Check new config macros. > * testsuite/lib/libstdc++.exp: Likewise. > > Tested x86_64-linux and x86_64-netbsd5.1, committed to trunk > <nprocs.patch>
On 7 November 2011 14:10, Iain Sandoe wrote: > > On 7 Nov 2011, at 13:45, Jonathan Wakely wrote: > >> This provides a working thread::hardware_concurrency on platforms that >> support pthread_num_processors_np or the "hw.ncpu" sysctl, but by >> testing for the features in configure rather than hardcoding OS macro >> tests in thread.cc > > if the system supports run-time User-controllable core-count then this will > not account for that. Yep. Do you know how to detect the right value in that case? If not the alternative is to revert to the previous behaviour of returning zero. (My understanding was that at least some of the options would work for that case, NPROCESSORS_ONLN is distinct from _SC_NPROCESSORS_CONF, for example)
On 7 November 2011 14:16, Jonathan Wakely wrote: > On 7 November 2011 14:10, Iain Sandoe wrote: >> >> On 7 Nov 2011, at 13:45, Jonathan Wakely wrote: >> >>> This provides a working thread::hardware_concurrency on platforms that >>> support pthread_num_processors_np or the "hw.ncpu" sysctl, but by >>> testing for the features in configure rather than hardcoding OS macro >>> tests in thread.cc >> >> if the system supports run-time User-controllable core-count then this will >> not account for that. > > Yep. Do you know how to detect the right value in that case? If not > the alternative is to revert to the previous behaviour of returning > zero. > > (My understanding was that at least some of the options would work for > that case, NPROCESSORS_ONLN is distinct from _SC_NPROCESSORS_CONF, for > example) ... which might be a reason to prefer sysconf to sysctl if the latter doesn't handle disabled cores (NetBSD supports both, but I found the sysctl to be slightly faster, so ordered the checks to prefer sysctl) Pretty much the only docs I have to go on for this are glibc's http://www.gnu.org/s/hello/manual/libc/Processor-Resources.html so I know disabled cores are accounted for correctly on GNU systems. If people care about other platforms they need to provide working code, or at least point me at decent documentation.
On 7 Nov 2011, at 14:16, Jonathan Wakely wrote: > On 7 November 2011 14:10, Iain Sandoe wrote: >> >> On 7 Nov 2011, at 13:45, Jonathan Wakely wrote: >> >>> This provides a working thread::hardware_concurrency on platforms >>> that >>> support pthread_num_processors_np or the "hw.ncpu" sysctl, but by >>> testing for the features in configure rather than hardcoding OS >>> macro >>> tests in thread.cc >> >> if the system supports run-time User-controllable core-count then >> this will >> not account for that. > > Yep. Do you know how to detect the right value in that case? If not > the alternative is to revert to the previous behaviour of returning > zero. > > (My understanding was that at least some of the options would work for > that case, NPROCESSORS_ONLN is distinct from _SC_NPROCESSORS_CONF, for > example) Hm. I don't know about the general case - on Darwin it's certainly possible to get at the information at runtime; hw.logicalcpu_max: 4 hw.logicalcpu: 4 hw.physicalcpu_max: 4 hw.physicalcpu: 4 hw.ncpu = 4 and ... switching the control panel to dual CPU .. hw.logicalcpu_max: 4 hw.logicalcpu: 2 hw.physicalcpu_max: 4 hw.physicalcpu: 2 hw.ncpu = 4 but, as you say, that would be the implication of "ONLN" vs "CONF" .. cheers Iain
On 7 November 2011 14:23, Iain Sandoe wrote: > > On 7 Nov 2011, at 14:16, Jonathan Wakely wrote: > >> On 7 November 2011 14:10, Iain Sandoe wrote: >>> >>> On 7 Nov 2011, at 13:45, Jonathan Wakely wrote: >>> >>>> This provides a working thread::hardware_concurrency on platforms that >>>> support pthread_num_processors_np or the "hw.ncpu" sysctl, but by >>>> testing for the features in configure rather than hardcoding OS macro >>>> tests in thread.cc >>> >>> if the system supports run-time User-controllable core-count then this >>> will >>> not account for that. >> >> Yep. Do you know how to detect the right value in that case? If not >> the alternative is to revert to the previous behaviour of returning >> zero. >> >> (My understanding was that at least some of the options would work for >> that case, NPROCESSORS_ONLN is distinct from _SC_NPROCESSORS_CONF, for >> example) > > Hm. I don't know about the general case - on Darwin it's certainly possible > to get at the information at runtime; > > hw.logicalcpu_max: 4 > hw.logicalcpu: 4 > hw.physicalcpu_max: 4 > hw.physicalcpu: 4 > hw.ncpu = 4 > > and ... switching the control panel to dual CPU .. > > hw.logicalcpu_max: 4 > hw.logicalcpu: 2 > hw.physicalcpu_max: 4 > hw.physicalcpu: 2 > hw.ncpu = 4 > > but, as you say, that would be the implication of "ONLN" vs "CONF" .. Does Darwin support those sysconf values? AFAICT it doesn't. If not, we could find out the equivalent of HW_NCPU for "hw.logicalcpu" or just use sysctlbyname, but as Apple haven't documented it in the sysctl manpage I don't know if those sysctls are stable or reliable. If someone wants to improve the checks be my guest, as long as it's done via autoconf macros, not by making the macro mess in src/thread.cc any worse.
On 7 Nov 2011, at 14:23, Iain Sandoe wrote: > On 7 Nov 2011, at 14:16, Jonathan Wakely wrote: > >> On 7 November 2011 14:10, Iain Sandoe wrote: >>> >>> On 7 Nov 2011, at 13:45, Jonathan Wakely wrote: >>> >>>> This provides a working thread::hardware_concurrency on platforms >>>> that >>>> support pthread_num_processors_np or the "hw.ncpu" sysctl, but by >>>> testing for the features in configure rather than hardcoding OS >>>> macro >>>> tests in thread.cc >>> >>> if the system supports run-time User-controllable core-count then >>> this will >>> not account for that. >> >> Yep. Do you know how to detect the right value in that case? If not >> the alternative is to revert to the previous behaviour of returning >> zero. >> >> (My understanding was that at least some of the options would work >> for >> that case, NPROCESSORS_ONLN is distinct from _SC_NPROCESSORS_CONF, >> for >> example) > > Hm. I don't know about the general case - on Darwin it's certainly > possible to get at the information at runtime; > > hw.logicalcpu_max: 4 > hw.logicalcpu: 4 > hw.physicalcpu_max: 4 > hw.physicalcpu: 4 > hw.ncpu = 4 > > and ... switching the control panel to dual CPU .. > > hw.logicalcpu_max: 4 > hw.logicalcpu: 2 > hw.physicalcpu_max: 4 > hw.physicalcpu: 2 > hw.ncpu = 4 > > but, as you say, that would be the implication of "ONLN" vs "CONF" .. however, apparently not - ... #include <unistd.h> #include <stdio.h> int main (int ac, char *avv[]) { printf ("n-procs: %d\n", sysconf (_SC_NPROCESSORS_ONLN)); } $ gcc-4.2 ../tests/nprocs.c -o tt ./tt n-procs: 4 .. switched to dual cpu ... $ sysctl hw.logicalcpu hw.logicalcpu: 2 $ ./tt n-procs: 4 :-( so there's a reason to use the systlbyname (and use hw.logicalcpu or similar, maybe). [unless that's just a buggy sysconf] Iain
On 7 Nov 2011, at 14:38, Jonathan Wakely wrote: > On 7 November 2011 14:23, Iain Sandoe wrote: >> >> On 7 Nov 2011, at 14:16, Jonathan Wakely wrote: >> >>> On 7 November 2011 14:10, Iain Sandoe wrote: >>>> >>>> On 7 Nov 2011, at 13:45, Jonathan Wakely wrote: >>>> >>>>> This provides a working thread::hardware_concurrency on >>>>> platforms that >>>>> support pthread_num_processors_np or the "hw.ncpu" sysctl, but by >>>>> testing for the features in configure rather than hardcoding OS >>>>> macro >>>>> tests in thread.cc >>>> >>>> if the system supports run-time User-controllable core-count then >>>> this >>>> will >>>> not account for that. >>> >>> Yep. Do you know how to detect the right value in that case? If >>> not >>> the alternative is to revert to the previous behaviour of returning >>> zero. >>> >>> (My understanding was that at least some of the options would work >>> for >>> that case, NPROCESSORS_ONLN is distinct from _SC_NPROCESSORS_CONF, >>> for >>> example) >> >> Hm. I don't know about the general case - on Darwin it's certainly >> possible >> to get at the information at runtime; >> >> hw.logicalcpu_max: 4 >> hw.logicalcpu: 4 >> hw.physicalcpu_max: 4 >> hw.physicalcpu: 4 >> hw.ncpu = 4 >> >> and ... switching the control panel to dual CPU .. >> >> hw.logicalcpu_max: 4 >> hw.logicalcpu: 2 >> hw.physicalcpu_max: 4 >> hw.physicalcpu: 2 >> hw.ncpu = 4 >> >> but, as you say, that would be the implication of "ONLN" vs "CONF" .. > > Does Darwin support those sysconf values? AFAICT it doesn't. from Darwin 9 onwards I believe so .. (don't actually have a copy of Darwin11 yet). > > If not, we could find out the equivalent of HW_NCPU for > "hw.logicalcpu" or just use sysctlbyname, but as Apple haven't > documented it in the sysctl manpage I don't know if those sysctls are > stable or reliable. > > If someone wants to improve the checks be my guest, as long as it's > done via autoconf macros, not by making the macro mess in > src/thread.cc any worse.
On 7 November 2011 14:40, Iain Sandoe wrote: > > On 7 Nov 2011, at 14:23, Iain Sandoe wrote: > >> On 7 Nov 2011, at 14:16, Jonathan Wakely wrote: >> >>> On 7 November 2011 14:10, Iain Sandoe wrote: >>>> >>>> On 7 Nov 2011, at 13:45, Jonathan Wakely wrote: >>>> >>>>> This provides a working thread::hardware_concurrency on platforms that >>>>> support pthread_num_processors_np or the "hw.ncpu" sysctl, but by >>>>> testing for the features in configure rather than hardcoding OS macro >>>>> tests in thread.cc >>>> >>>> if the system supports run-time User-controllable core-count then this >>>> will >>>> not account for that. >>> >>> Yep. Do you know how to detect the right value in that case? If not >>> the alternative is to revert to the previous behaviour of returning >>> zero. >>> >>> (My understanding was that at least some of the options would work for >>> that case, NPROCESSORS_ONLN is distinct from _SC_NPROCESSORS_CONF, for >>> example) >> >> Hm. I don't know about the general case - on Darwin it's certainly >> possible to get at the information at runtime; >> >> hw.logicalcpu_max: 4 >> hw.logicalcpu: 4 >> hw.physicalcpu_max: 4 >> hw.physicalcpu: 4 >> hw.ncpu = 4 >> >> and ... switching the control panel to dual CPU .. >> >> hw.logicalcpu_max: 4 >> hw.logicalcpu: 2 >> hw.physicalcpu_max: 4 >> hw.physicalcpu: 2 >> hw.ncpu = 4 >> >> but, as you say, that would be the implication of "ONLN" vs "CONF" .. > > however, apparently not - ... > > #include <unistd.h> > #include <stdio.h> > > int main (int ac, char *avv[]) > { > printf ("n-procs: %d\n", sysconf (_SC_NPROCESSORS_ONLN)); > } > > $ gcc-4.2 ../tests/nprocs.c -o tt > > ./tt > n-procs: 4 > > .. switched to dual cpu ... > > $ sysctl hw.logicalcpu > hw.logicalcpu: 2 > > $ ./tt > n-procs: 4 > > :-( > > so there's a reason to use the systlbyname (and use hw.logicalcpu or > similar, maybe). > [unless that's just a buggy sysconf] Well if that's how they want to play it then I'm not even going to think about changing that code without being able to test it on a real system! What does "switch to dual cpu" actually mean? disable hyperthreading? disable one core on each die to save power? disable one die to save power? lie about number of cores so buggy software doesn't get confused? I think someone with access to a darwin box and motivation to improve it will have to make any improvements. I would suggest using sysctlnametomib and caching the result, then using sysctl, to avoid the overhead of systlbyname parsing the name on every call. At some point in the near future (no pun intended) I want to enhance std::async so it checks the system load and maybe the hardware_concurrency when deciding whether to run an asynchronous task in a new thread or not, so I want thread:hardware_concurrency() to be as fast as possible.
On 7 Nov 2011, at 14:52, Jonathan Wakely wrote: > On 7 November 2011 14:40, Iain Sandoe wrote: >> so there's a reason to use the systlbyname (and use hw.logicalcpu or >> similar, maybe). >> [unless that's just a buggy sysconf] > > Well if that's how they want to play it then I'm not even going to > think about changing that code without being able to test it on a real > system! fair ;-) > What does "switch to dual cpu" actually mean? disable hyperthreading? > disable one core on each die to save power? disable one die to save > power? lie about number of cores so buggy software doesn't get > confused? it's done via a control panel which is intended to be a developer's tool to allow testing their code on machines with different capabilities from their development box. I don't think that the average Joe User would need/want to do this.... > I think someone with access to a darwin box and motivation to improve > it will have to make any improvements. I would suggest using > sysctlnametomib and caching the result, then using sysctl, to avoid > the overhead of systlbyname parsing the name on every call. yeah ... if only one could ulimit -hoursinday unlimited ... > At some > point in the near future (no pun intended) I want to enhance > std::async so it checks the system load and maybe the > hardware_concurrency when deciding whether to run an asynchronous task > in a new thread or not, so I want thread:hardware_concurrency() to be > as fast as possible. For the 99.99% case, I suspect that assuming that the number of cpus online == number in the box is safe on Darwin (for normal end users). An interested dev. can do what you suggest .. Iain
Index: acinclude.m4 =================================================================== --- acinclude.m4 (revision 181081) +++ acinclude.m4 (working copy) @@ -3459,7 +3459,95 @@ AC_LANG_RESTORE ]) +dnl +dnl Check whether sysconf(_SC_NPROC_ONLN) is available in <unistd.h>, and define _GLIBCXX_USE_SC_NPROC_ONLN. +dnl +AC_DEFUN([GLIBCXX_CHECK_SC_NPROC_ONLN], [ + AC_LANG_SAVE + AC_LANG_CPLUSPLUS + ac_save_CXXFLAGS="$CXXFLAGS" + CXXFLAGS="$CXXFLAGS -fno-exceptions" + + AC_MSG_CHECKING([for _SC_NPROC_ONLN]) + AC_CACHE_VAL(glibcxx_cv_SC_NPROC_ONLN, [ + GCC_TRY_COMPILE_OR_LINK( + [#include <unistd.h>], + [int n = sysconf(_SC_NPROC_ONLN);], + [glibcxx_cv_SC_NPROC_ONLN=yes], + [glibcxx_cv_SC_NPROC_ONLN=no]) + ]) + if test $glibcxx_cv_SC_NPROC_ONLN = yes; then + AC_DEFINE(_GLIBCXX_USE_SC_NPROC_ONLN, 1, [Define if _SC_NPROC_ONLN is available in <unistd.h>.]) + fi + AC_MSG_RESULT($glibcxx_cv_SC_NPROC_ONLN) + + CXXFLAGS="$ac_save_CXXFLAGS" + AC_LANG_RESTORE +]) + +dnl +dnl Check whether pthread_num_processors_np is available in <pthread.h>, and define _GLIBCXX_USE_PTHREADS_NUM_PROCESSORS_NP. +dnl +AC_DEFUN([GLIBCXX_CHECK_PTHREADS_NUM_PROCESSORS_NP], [ + + AC_LANG_SAVE + AC_LANG_CPLUSPLUS + ac_save_CXXFLAGS="$CXXFLAGS" + CXXFLAGS="$CXXFLAGS -fno-exceptions" + + AC_MSG_CHECKING([for pthreads_num_processors_np]) + AC_CACHE_VAL(glibcxx_cv_PTHREADS_NUM_PROCESSORS_NP, [ + GCC_TRY_COMPILE_OR_LINK( + [#include <pthread.h>], + [int n = pthread_num_processors_np();], + [glibcxx_cv_PTHREADS_NUM_PROCESSORS_NP=yes], + [glibcxx_cv_PTHREADS_NUM_PROCESSORS_NP=no]) + ]) + if test $glibcxx_cv_PTHREADS_NUM_PROCESSORS_NP = yes; then + AC_DEFINE(_GLIBCXX_USE_PTHREADS_NUM_PROCESSORS_NP, 1, [Define if pthreads_num_processors_np is available in <pthread.h>.]) + fi + AC_MSG_RESULT($glibcxx_cv_PTHREADS_NUM_PROCESSORS_NP) + + CXXFLAGS="$ac_save_CXXFLAGS" + AC_LANG_RESTORE +]) + +dnl +dnl Check whether sysctl is available in <pthread.h>, and define _GLIBCXX_USE_SYSCTL_HW_NCPU. +dnl +AC_DEFUN([GLIBCXX_CHECK_SYSCTL_HW_NCPU], [ + + AC_LANG_SAVE + AC_LANG_CPLUSPLUS + ac_save_CXXFLAGS="$CXXFLAGS" + CXXFLAGS="$CXXFLAGS -fno-exceptions" + + AC_MSG_CHECKING([for hw.ncpu sysctl]) + AC_CACHE_VAL(glibcxx_cv_SYSCTL_HW_NCPU, [ + GCC_TRY_COMPILE_OR_LINK( + [ + #include <stddef.h> + #include <sys/sysctl.h> + ], + [ + int count; + size_t size = sizeof(count); + int mib[] = { CTL_HW, HW_NCPU }; + sysctl(mib, 2, &count, &size, NULL, 0); + ], + [glibcxx_cv_SYSCTL_HW_NCPU=yes], + [glibcxx_cv_SYSCTL_HW_NCPU=no]) + ]) + if test $glibcxx_cv_SYSCTL_HW_NCPU = yes; then + AC_DEFINE(_GLIBCXX_USE_SYSCTL_HW_NCPU, 1, [Define if sysctl(), CTL_HW and HW_NCPU are available in <sys/sysctl.h>.]) + fi + AC_MSG_RESULT($glibcxx_cv_SYSCTL_HW_NCPU) + + CXXFLAGS="$ac_save_CXXFLAGS" + AC_LANG_RESTORE +]) + # Macros from the top-level gcc directory. m4_include([../config/gc++filt.m4]) m4_include([../config/tls.m4]) Index: configure.ac =================================================================== --- configure.ac (revision 181081) +++ configure.ac (working copy) @@ -174,6 +174,9 @@ GLIBCXX_CHECK_GET_NPROCS AC_CHECK_HEADERS(unistd.h) GLIBCXX_CHECK_SC_NPROCESSORS_ONLN +GLIBCXX_CHECK_SC_NPROC_ONLN +GLIBCXX_CHECK_PTHREADS_NUM_PROCESSORS_NP +GLIBCXX_CHECK_SYSCTL_HW_NCPU # Check for available headers. AC_CHECK_HEADERS([endian.h execinfo.h float.h fp.h ieeefp.h inttypes.h \ Index: src/thread.cc =================================================================== --- src/thread.cc (revision 181081) +++ src/thread.cc (working copy) @@ -30,9 +30,27 @@ #if defined(_GLIBCXX_USE_GET_NPROCS) # include <sys/sysinfo.h> # define _GLIBCXX_NPROCS get_nprocs() +#elif defined(_GLIBCXX_USE_PTHREADS_NUM_PROCESSORS_NP) +# define _GLIBCXX_NPROCS pthread_num_processors_np() +#elif defined(_GLIBCXX_USE_SYSCTL_HW_NCPU) +# include <stddef.h> +# include <sys/sysctl.h> +static inline int get_nprocs() +{ + int count; + size_t size = sizeof(count); + int mib[] = { CTL_HW, HW_NCPU }; + if (!sysctl(mib, 2, &count, &size, NULL, 0)) + return count; + return 0; +} +# define _GLIBCXX_NPROCS get_nprocs() #elif defined(_GLIBCXX_USE_SC_NPROCESSORS_ONLN) # include <unistd.h> # define _GLIBCXX_NPROCS sysconf(_SC_NPROCESSORS_ONLN) +#elif defined(_GLIBCXX_USE_SC_NPROC_ONLN) +# include <unistd.h> +# define _GLIBCXX_NPROCS sysconf(_SC_NPROC_ONLN) #else # define _GLIBCXX_NPROCS 0 #endif Index: testsuite/lib/libstdc++.exp =================================================================== --- testsuite/lib/libstdc++.exp (revision 181081) +++ testsuite/lib/libstdc++.exp (working copy) @@ -1673,13 +1673,12 @@ set f [open $src "w"] puts $f "#include <bits/c++config.h>" puts $f "#if defined(_GLIBCXX_USE_GET_NPROCS)" - puts $f "#elif defined(_GLIBCXX_USE_SYSCONF)" - puts $f "# include <unistd.h>" - puts $f "# if !defined(_SC_NPROCESSORS_ONLN)" - puts $f "# error No sysconf option" - puts $f "# endif" + puts $f "#elif defined(_GLIBCXX_USE_PTHREADS_NUM_PROCESSORS_NP)" + puts $f "#elif defined(_GLIBCXX_USE_SYSCTL_HW_NCPU)" + puts $f "#elif defined(_GLIBCXX_USE_SC_NPROCESSORS_ONLN)" + puts $f "#elif defined(_GLIBCXX_USE_SC_NPROC_ONLN)" puts $f "#else" - puts $f "# error No get_nprocs or sysconf" + puts $f "# error hardware_concurrency not implemented" puts $f "#endif" close $f
This provides a working thread::hardware_concurrency on platforms that support pthread_num_processors_np or the "hw.ncpu" sysctl, but by testing for the features in configure rather than hardcoding OS macro tests in thread.cc * acinclude.m4 (GLIBCXX_CHECK_SC_NPROC_ONLN): Define. (GLIBCXX_CHECK_PTHREADS_NUM_PROCESSORS_NP): Define. (GLIBCXX_CHECK_SYSCTL_HW_NCPU): Define. * configure.ac: Use new checks. * configure: Regenerate. * config.h.in: Regenerate. * src/thread.cc: Check new config macros. * testsuite/lib/libstdc++.exp: Likewise. Tested x86_64-linux and x86_64-netbsd5.1, committed to trunk