Patchwork [7/7] automatically set PARALLEL_JOBS when BR2_JLEVEL is 0

login
register
mail settings
Submitter Nathan Lynch
Date June 12, 2012, 2:09 a.m.
Message ID <1339466983-19786-8-git-send-email-ntl@pobox.com>
Download mbox | patch
Permalink /patch/164305/
State Superseded
Headers show

Comments

Nathan Lynch - June 12, 2012, 2:09 a.m.
When BR2_JLEVEL is 0, set PARALLEL_JOBS to the number of CPUs
detected.  This allows one to more or less fully utilize the host
system without manually tuning the configuration.

Also make 0 the default value for BR2_JLEVEL.

Signed-off-by: Nathan Lynch <ntl@pobox.com>
---
 Config.in           |    8 +++++---
 package/Makefile.in |    4 ++++
 2 files changed, 9 insertions(+), 3 deletions(-)
Arnout Vandecappelle - June 13, 2012, 11:27 p.m.
On 06/12/12 04:09, Nathan Lynch wrote:
> When BR2_JLEVEL is 0, set PARALLEL_JOBS to the number of CPUs
> detected.  This allows one to more or less fully utilize the host
> system without manually tuning the configuration.

  In my experience, you need 2 times the number of CPUs to fully utilize
the processor because most compilations have long iowait latencies.

  Otherwise, looks good.

  Regards,
  Arnout
Nathan Lynch - June 13, 2012, 11:44 p.m.
On Thu, 2012-06-14 at 01:27 +0200, Arnout Vandecappelle wrote:
> On 06/12/12 04:09, Nathan Lynch wrote:
> > When BR2_JLEVEL is 0, set PARALLEL_JOBS to the number of CPUs
> > detected.  This allows one to more or less fully utilize the host
> > system without manually tuning the configuration.
> 
>   In my experience, you need 2 times the number of CPUs to fully utilize
> the processor because most compilations have long iowait latencies.

Yeah.  I'm happy to make this change, assuming others agree?

Thanks for having a look!
Peter Korsgaard - June 14, 2012, 6:07 a.m.
>>>>> "Nathan" == Nathan Lynch <ntl@pobox.com> writes:

 >> In my experience, you need 2 times the number of CPUs to fully utilize
 >> the processor because most compilations have long iowait latencies.

 Nathan> Yeah.  I'm happy to make this change, assuming others agree?

I was going to say something like that as well, so yes please.
Nathan Lynch - June 14, 2012, 6:16 a.m.
On Thu, 2012-06-14 at 08:07 +0200, Peter Korsgaard wrote:
> >>>>> "Nathan" == Nathan Lynch <ntl@pobox.com> writes:
> 
>  >> In my experience, you need 2 times the number of CPUs to fully utilize
>  >> the processor because most compilations have long iowait latencies.
> 
>  Nathan> Yeah.  I'm happy to make this change, assuming others agree?
> 
> I was going to say something like that as well, so yes please.

Certainly.  Is an update of patch #7 acceptable or shall I re-send the
series?
Richard Braun - June 14, 2012, 8:33 a.m.
On Thu, Jun 14, 2012 at 01:27:05AM +0200, Arnout Vandecappelle wrote:
>  In my experience, you need 2 times the number of CPUs to fully utilize
> the processor because most compilations have long iowait latencies.

Unless you have little memory, this is no longer true, and as some
compilations can eat a lot of memory, doubling their number reduces the
amount of available page cache, potentially increasing the amount of
I/O. Using the number of available processors should be just fine.
Peter Korsgaard - June 14, 2012, 8:56 a.m.
>>>>> "Nathan" == Nathan Lynch <ntl@pobox.com> writes:

 Nathan> On Thu, 2012-06-14 at 08:07 +0200, Peter Korsgaard wrote:
 >> >>>>> "Nathan" == Nathan Lynch <ntl@pobox.com> writes:
 >> 
 >> >> In my experience, you need 2 times the number of CPUs to fully utilize
 >> >> the processor because most compilations have long iowait latencies.
 >> 
 Nathan> Yeah.  I'm happy to make this change, assuming others agree?
 >> 
 >> I was going to say something like that as well, so yes please.

 Nathan> Certainly.  Is an update of patch #7 acceptable or shall I
 Nathan> re-send the series?

Just #7 (with a clearly marked [PATCHv2] in subject) is fine if that's
the only change since last time.

Patch

diff --git a/Config.in b/Config.in
index 95c2e8c..3745255 100644
--- a/Config.in
+++ b/Config.in
@@ -178,10 +178,12 @@  config BR2_DEBIAN_MIRROR
 endmenu
 
 config BR2_JLEVEL
-	int "Number of jobs to run simultaneously"
-	default "2"
+	int "Number of jobs to run simultaneously (0 for auto)"
+	default "0"
 	help
-	  Number of jobs to run simultaneously
+	  Number of jobs to run simultaneously.  If 0, determine
+	  automatically according to number of CPUs on the host
+	  system.
 
 config BR2_CCACHE
 	bool "Enable compiler cache"
diff --git a/package/Makefile.in b/package/Makefile.in
index 9177a1b..a9d2771 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -6,7 +6,11 @@  HOSTMAKE=$(MAKE)
 endif
 HOSTMAKE :=$(shell which $(HOSTMAKE) || type -p $(HOSTMAKE) || echo make)
 
+ifeq ($(BR2_JLEVEL),0)
+PARALLEL_JOBS:=$(shell getconf _NPROCESSORS_ONLN 2>/dev/null || echo 1)
+else
 PARALLEL_JOBS:=$(BR2_JLEVEL)
+endif
 
 MAKE1:=$(HOSTMAKE) -j1
 MAKE:=$(HOSTMAKE) -j$(PARALLEL_JOBS)