diff mbox

RFC: PATCH to avoid linking multiple front ends at once with parallel make

Message ID 518B1F2E.4080009@redhat.com
State New
Headers show

Commit Message

Jason Merrill May 9, 2013, 3:59 a.m. UTC 
On 05/01/2013 07:40 PM, Mike Stump wrote:
> $ bash -c "trap 'echo remove lock' 0; true"; echo $?

Thanks for the suggestion.  Here's a revised patch:

Comments

Mike Stump May 9, 2013, 5:20 p.m. UTC | #1 
On May 8, 2013, at 8:59 PM, Jason Merrill <jason@redhat.com> wrote:
> Thanks for the suggestion.  Here's a revised patch:

Nice.  I too see a whole machine style lock up on linux (openSUSE 11.4) and my machine has 12G and 8*2 cores.  I just ignore it and wait patiently for the storm to pass, but, it is annoying.  I use -g3.  It isn't a laptop.  You aren't the only one.

A note to future people that want to do mods with this.  Not unreasonable to sense -g3 and have one limit with -g3 and a different one with -g.
Alexandre Oliva May 14, 2013, 4:49 a.m. UTC | #2 
On May  9, 2013, Jason Merrill <jason@redhat.com> wrote:

> On 05/01/2013 07:40 PM, Mike Stump wrote:
>> $ bash -c "trap 'echo remove lock' 0; true"; echo $?

> Thanks for the suggestion.  Here's a revised patch:

I like the idea of the patch, for I feel your pain too ;-)

However, rather than implementing the locking in Makefiles, I'm thinking
it might be wiser to do so in a script that takes the lock name and the
command to run while holding the lock.

ifeq (@DO_LINK_MUTEX@,true)
LLINKER = $(SHELL) $(srcdir)/lock-and-run linkfe.lck $(LINKER)
else
LLINKER = $(LINKER)
endif

lock-and-run:
#! /bin/sh
lockdir=$1 prog=$2; shift 2 || exit 1
status=1
trap 'rmdir "$lockdir"; exit $status' 0 1 2 15
until mkdir "$lockdir"; do sleep 1; done
$prog "$@"
status=$?
rmdir "$lockdir"
exit $status
Tom Tromey May 14, 2013, 5:24 p.m. UTC | #3 
>>>>> "Alexandre" == Alexandre Oliva <aoliva@redhat.com> writes:

Alexandre> However, rather than implementing the locking in Makefiles,
Alexandre> I'm thinking it might be wiser to do so in a script that
Alexandre> takes the lock name and the command to run while holding the
Alexandre> lock.

It seems to me you can get the desired functionality in the Makefiles
using order-only dependencies -- just force some ordering of the
targets.

Tom
Mike Stump May 14, 2013, 6:04 p.m. UTC | #4 
On May 13, 2013, at 9:49 PM, Alexandre Oliva <aoliva@redhat.com> wrote:
> However, rather than implementing the locking in Makefiles, I'm thinking
> it might be wiser to do so in a script that takes the lock name and the
> command to run while holding the lock.

I worry about quoting.  Anytime you accept and pass on a argument list, typically means that you destroy the argument list, unless you have something to quote it for you.  bash (sh in general) has a fetish for destroying arguments.  :-(  A linker option that has \, or spaces or ' or " in it, is the type of problem I'm thinking of.  Also applies to filenames.
Jason Merrill May 14, 2013, 8:39 p.m. UTC | #5 
On 05/14/2013 01:24 PM, Tom Tromey wrote:
> It seems to me you can get the desired functionality in the Makefiles
> using order-only dependencies -- just force some ordering of the
> targets.

Order dependencies are the wrong solution: for one thing, they don't 
allow for building just a single front end, which I do all the time. 
And rebuilding, say, the Fortran front end shouldn't mean that we then 
need to rebuild the Java front end as well.

Jason
Jason Merrill May 14, 2013, 8:46 p.m. UTC | #6 
On 05/14/2013 02:04 PM, Mike Stump wrote:
> I worry about quoting.

In my experience, "$@" does fine for passing on arguments.

Jason
Mike Stump May 14, 2013, 9:38 p.m. UTC | #7 
On May 14, 2013, at 1:46 PM, Jason Merrill <jason@redhat.com> wrote:
> On 05/14/2013 02:04 PM, Mike Stump wrote:
>> I worry about quoting.
> 
> In my experience, "$@" does fine for passing on arguments.

:-)  It used to require ${0+"$@"}, but later shells fixed the edge case to make "$@" work nicely.  While I'll still worry about quoting, I agree "$@" should work just fine.
diff mbox

Patch

commit 79b52becc6a8649933f7442d46e8054f8b6abd4b
Author: Jason Merrill <jason@redhat.com>
Date:   Thu Apr 25 17:54:17 2013 -0400

    	* Makefile.in (link-mutex, remove-link-mutex): New targets.
    	(native, mostlyclean): Remove link mutex.
    	* configure.ac: Handle --enable-link-mutex.
    	* */Make-lang.in: Use link mutex.

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 903125e..f8968c5 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1737,8 +1737,8 @@  start.encap: native xgcc$(exeext) cpp$(exeext) specs \
 rest.encap: lang.rest.encap
 # This is what is made with the host's compiler
 # whether making a cross compiler or not.
-native: config.status auto-host.h build-@POSUB@ $(LANGUAGES) \
-	$(EXTRA_PROGRAMS) $(COLLECT2) lto-wrapper$(exeext) \
+native: remove-link-mutex config.status auto-host.h build-@POSUB@ \
+	$(LANGUAGES) $(EXTRA_PROGRAMS) $(COLLECT2) lto-wrapper$(exeext) \
 	gcc-ar$(exeext) gcc-nm$(exeext) gcc-ranlib$(exeext)
 
 ifeq ($(enable_plugin),yes)
@@ -4529,6 +4529,8 @@  mostlyclean: lang.mostlyclean
 	-rm -f gtype.state
 # Delete genchecksum outputs
 	-rm -f *-checksum.c
+# Delete front-end linking mutex directory
+	-rm -rf link-mutex
 
 # Delete all files made by compilation
 # that don't exist in the distribution.
@@ -5335,3 +5337,20 @@  po/gcc.pot: force
 	$(MAKE) srcextra
 	AWK=$(AWK) $(SHELL) $(srcdir)/po/exgettext \
 		$(XGETTEXT) gcc $(srcdir)
+
+# Like LINKER, but remove the link mutex.  Used for linking front ends.
+ifeq (@DO_LINK_MUTEX@,true)
+LLINKER = trap '$(MAKE) remove-link-mutex' 0; $(LINKER)
+else
+LLINKER = $(LINKER)
+endif
+
+.PHONY: link-mutex
+link-mutex:
+ifeq (@DO_LINK_MUTEX@,true)
+	while ! mkdir link-mutex 2>/dev/null; do sleep 1; done
+endif
+
+.PHONY: remove-link-mutex
+remove-link-mutex:
+	-rm -rf link-mutex
diff --git a/gcc/ada/gcc-interface/Make-lang.in b/gcc/ada/gcc-interface/Make-lang.in
index ef12b4b..c7eda7e 100644
--- a/gcc/ada/gcc-interface/Make-lang.in
+++ b/gcc/ada/gcc-interface/Make-lang.in
@@ -185,6 +185,7 @@  endif
 GCC_LINKERFLAGS = $(filter-out -Werror, $(ALL_LINKERFLAGS))
 
 GCC_LINK=$(LINKER) $(GCC_LINKERFLAGS) $(LDFLAGS)
+GCC_LLINK=$(LLINKER) $(GCC_LINKERFLAGS) $(LDFLAGS)
 
 # Lists of files for various purposes.
 
@@ -562,7 +563,9 @@  TARGET_ADA_SRCS =
 # Since the RTL should be built with the latest compiler, remove the
 #  stamp target in the parent directory whenever gnat1 is rebuilt
 gnat1$(exeext): $(TARGET_ADA_SRCS) $(GNAT1_OBJS) $(ADA_BACKEND) libcommon-target.a $(LIBDEPS)
-	+$(GCC_LINK) -o $@ $(GNAT1_OBJS) $(ADA_BACKEND) libcommon-target.a $(LIBS) $(SYSLIBS) $(BACKENDLIBS) $(CFLAGS)
+	$(MAKE) link-mutex
+	+$(GCC_LLINK) -o $@ $(GNAT1_OBJS) $(ADA_BACKEND) \
+	  libcommon-target.a $(LIBS) $(SYSLIBS) $(BACKENDLIBS) $(CFLAGS)
 	$(RM) stamp-gnatlib2-rts stamp-tools
 
 gnatbind$(exeext): ada/b_gnatb.o $(CONFIG_H) $(GNATBIND_OBJS) ggc-none.o libcommon-target.a $(LIBDEPS)
diff --git a/gcc/c/Make-lang.in b/gcc/c/Make-lang.in
index 8310e0a..1b6189a 100644
--- a/gcc/c/Make-lang.in
+++ b/gcc/c/Make-lang.in
@@ -75,7 +75,8 @@  cc1-checksum.c : build/genchecksum$(build_exeext) checksum-options \
 cc1-checksum.o : cc1-checksum.c $(CONFIG_H) $(SYSTEM_H)
 
 cc1$(exeext): $(C_OBJS) cc1-checksum.o $(BACKEND) $(LIBDEPS)
-	+$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ $(C_OBJS) \
+	$(MAKE) link-mutex
+	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ $(C_OBJS) \
 	  cc1-checksum.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
 #
 # Build hooks:
diff --git a/gcc/configure b/gcc/configure
index 5161ae4..bd81007 100755
--- a/gcc/configure
+++ b/gcc/configure
@@ -670,6 +670,7 @@  subdirs
 dollar
 gcc_tooldir
 enable_lto
+DO_LINK_MUTEX
 MAINT
 zlibinc
 zlibdir
@@ -916,6 +917,7 @@  with_long_double_128
 with_gc
 with_system_zlib
 enable_maintainer_mode
+enable_link_mutex
 enable_version_specific_runtime_libs
 enable_plugin
 enable_libquadmath_support
@@ -1627,6 +1629,8 @@  Optional Features:
   --enable-maintainer-mode
                           enable make rules and dependencies not useful (and
                           sometimes confusing) to the casual installer
+  --enable-link-mutex     avoid linking multiple front-ends at once to avoid
+                          thrashing on the build machine
   --enable-version-specific-runtime-libs
                           specify that runtime libraries should be installed
                           in a compiler-specific directory
@@ -17830,7 +17834,7 @@  else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 17833 "configure"
+#line 17837 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
@@ -17936,7 +17940,7 @@  else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 17939 "configure"
+#line 17943 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
@@ -27044,6 +27048,26 @@  else
   MAINT='#'
 fi
 
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to avoid linking multiple front-ends at once" >&5
+$as_echo_n "checking whether to avoid linking multiple front-ends at once... " >&6; }
+  # Check whether --enable-link-mutex was given.
+if test "${enable_link_mutex+set}" = set; then :
+  enableval=$enable_link_mutex; do_link_mutex=$enableval
+else
+  do_link_mutex=no
+fi
+
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $do_link_mutex" >&5
+$as_echo "$do_link_mutex" >&6; }
+
+if test "$do_link_mutex" = "yes"; then
+   DO_LINK_MUTEX=true
+else
+   DO_LINK_MUTEX=false
+fi
+
+
 # --------------
 # Language hooks
 # --------------
diff --git a/gcc/configure.ac b/gcc/configure.ac
index b042f00..9ab03ec 100644
--- a/gcc/configure.ac
+++ b/gcc/configure.ac
@@ -4941,6 +4941,24 @@  else
 fi
 AC_SUBST(MAINT)dnl
 
+dnl Whether to prevent multiple front-ends from linking at the same time
+
+AC_MSG_CHECKING([whether to avoid linking multiple front-ends at once])
+  AC_ARG_ENABLE(link-mutex,
+[AS_HELP_STRING([--enable-link-mutex],
+		[avoid linking multiple front-ends at once to avoid thrashing
+		 on the build machine])],
+      do_link_mutex=$enableval,
+      do_link_mutex=no)
+AC_MSG_RESULT($do_link_mutex)
+
+if test "$do_link_mutex" = "yes"; then
+   DO_LINK_MUTEX=true
+else
+   DO_LINK_MUTEX=false
+fi
+AC_SUBST(DO_LINK_MUTEX)
+
 # --------------
 # Language hooks
 # --------------
diff --git a/gcc/cp/Make-lang.in b/gcc/cp/Make-lang.in
index cda4897..f8b3583 100644
--- a/gcc/cp/Make-lang.in
+++ b/gcc/cp/Make-lang.in
@@ -100,7 +100,8 @@  cc1plus-checksum.c : build/genchecksum$(build_exeext) checksum-options \
 cc1plus-checksum.o : cc1plus-checksum.c $(CONFIG_H) $(SYSTEM_H)
 
 cc1plus$(exeext): $(CXX_OBJS) cc1plus-checksum.o $(BACKEND) $(LIBDEPS)
-	+$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
+	$(MAKE) link-mutex
+	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 	      $(CXX_OBJS) cc1plus-checksum.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
 
 ifeq ($(ENABLE_MAINTAINER_RULES), true)
diff --git a/gcc/doc/install.texi b/gcc/doc/install.texi
index 1265129..7d7a5c7 100644
--- a/gcc/doc/install.texi
+++ b/gcc/doc/install.texi
@@ -1293,6 +1293,13 @@  opposite effect.  If neither option is specified, the configure script
 will try to guess whether the @code{.init_array} and
 @code{.fini_array} sections are supported and, if they are, use them.
 
+@item --enable-link-mutex
+When building GCC, use a mutex to avoid linking the compilers for
+multiple languages at the same time, to avoid thrashing on build
+systems with limited free memory.  This functionality requires that
+the default shell be bash, as it uses the 'trap' built-in command.
+The default is not to use such a mutex.
+
 @item --enable-maintainer-mode
 The build rules that regenerate the Autoconf and Automake output files as
 well as the GCC master message catalog @file{gcc.pot} are normally
diff --git a/gcc/fortran/Make-lang.in b/gcc/fortran/Make-lang.in
index c3f826c..4e414e2 100644
--- a/gcc/fortran/Make-lang.in
+++ b/gcc/fortran/Make-lang.in
@@ -98,7 +98,8 @@  gfortran-cross$(exeext): gfortran$(exeext)
 # The compiler itself is called f951.
 f951$(exeext): $(F95_OBJS) \
 		$(BACKEND) $(LIBDEPS) attribs.o
-	+$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
+	$(MAKE) link-mutex
+	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 		$(F95_OBJS) $(BACKEND) $(ZLIB) $(LIBS) attribs.o \
 		$(BACKENDLIBS)
 
diff --git a/gcc/go/Make-lang.in b/gcc/go/Make-lang.in
index dd98080..93ed920 100644
--- a/gcc/go/Make-lang.in
+++ b/gcc/go/Make-lang.in
@@ -76,7 +76,8 @@  GO_OBJS = \
 	go/unsafe.o
 
 go1$(exeext): $(GO_OBJS) attribs.o $(BACKEND) $(LIBDEPS)
-	+$(CXX) $(ALL_CXXFLAGS) $(LDFLAGS) -o $@ \
+	$(MAKE) link-mutex
+	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 	      $(GO_OBJS) attribs.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
 
 # Documentation.
diff --git a/gcc/java/Make-lang.in b/gcc/java/Make-lang.in
index 7ed3220..4d2ef8a 100644
--- a/gcc/java/Make-lang.in
+++ b/gcc/java/Make-lang.in
@@ -99,7 +99,8 @@  jvspec.o-warn = -Wno-error
 
 jc1$(exeext): $(JAVA_OBJS) $(BACKEND) $(LIBDEPS) attribs.o
 	rm -f $@
-	+$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
+	$(MAKE) link-mutex
+	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 		$(JAVA_OBJS) $(BACKEND) $(ZLIB) $(LIBICONV) $(LIBS) attribs.o $(BACKENDLIBS)
 
 jcf-dump$(exeext): $(JCFDUMP_OBJS) $(LIBDEPS)
diff --git a/gcc/lto/Make-lang.in b/gcc/lto/Make-lang.in
index a0bdc26..e0db780 100644
--- a/gcc/lto/Make-lang.in
+++ b/gcc/lto/Make-lang.in
@@ -71,7 +71,8 @@  lto.stagefeedback:
 lto-warn = $(STRICT_WARN)
 
 $(LTO_EXE): $(LTO_OBJS) $(BACKEND) $(LIBDEPS)
-	+$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
+	$(MAKE) link-mutex
+	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 		$(LTO_OBJS) $(BACKEND) $(BACKENDLIBS) $(LIBS)
 
 # Dependencies
diff --git a/gcc/objc/Make-lang.in b/gcc/objc/Make-lang.in
index 9e005c1..5faf54c 100644
--- a/gcc/objc/Make-lang.in
+++ b/gcc/objc/Make-lang.in
@@ -68,7 +68,8 @@  cc1obj-checksum.c : build/genchecksum$(build_exeext) checksum-options \
 cc1obj-checksum.o : cc1obj-checksum.c $(CONFIG_H) $(SYSTEM_H)
 
 cc1obj$(exeext): $(OBJC_OBJS) $(C_AND_OBJC_OBJS) cc1obj-checksum.o $(BACKEND) $(LIBDEPS)
-	+$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
+	$(MAKE) link-mutex
+	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 	      $(OBJC_OBJS) $(C_AND_OBJC_OBJS) cc1obj-checksum.o \
 	      $(BACKEND) $(LIBS) $(BACKENDLIBS)
 
diff --git a/gcc/objcp/Make-lang.in b/gcc/objcp/Make-lang.in
index 731ca9e..665e91e 100644
--- a/gcc/objcp/Make-lang.in
+++ b/gcc/objcp/Make-lang.in
@@ -72,7 +72,8 @@  cc1objplus-checksum.c : build/genchecksum$(build_exeext) checksum-options \
 cc1objplus-checksum.o : cc1objplus-checksum.c $(CONFIG_H) $(SYSTEM_H)
 
 cc1objplus$(exeext): $(OBJCXX_OBJS) cc1objplus-checksum.o $(BACKEND) $(LIBDEPS)
-	+$(LINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
+	$(MAKE) link-mutex
+	+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ \
 		$(OBJCXX_OBJS) cc1objplus-checksum.o $(BACKEND) $(LIBS) $(BACKENDLIBS)
 
 # Objective C++ language specific files.